Friday, 2018-01-05

clarkblaunch node is running, doing the yum update now (which appears to have found the right kernel but I'll double check pti is enabled before updating dns records00:04
*** esberglu has quit IRC00:04
*** xarses_ has quit IRC00:11
*** jistr has quit IRC00:11
*** caphrim007_ has joined #openstack-infra00:12
clarkbok think I've noticed is that our install_puppet that install the rdo stuff that we want to remove from the dib builds is also there for when we install puppet on real centos servers00:13
*** caphrim007_ has quit IRC00:13
clarkbI'm just going to roll with that for now, BUT we should consider rebuilding these servers again in the future if we can excise that00:14
*** jaypipes has quit IRC00:14
*** caphrim007 has quit IRC00:15
*** zhurong has joined #openstack-infra00:16
clarkb(it was present on the old servers too)00:16
*** jistr has joined #openstack-infra00:16
mordredclarkb: ++00:18
clarkbpti is enabled and test clones work00:19
clarkbI will update dns now please keep eyeballs open for any weirdness00:19
*** xarses_ has joined #openstack-infra00:20
*** slaweq has joined #openstack-infra00:20
*** sdague has quit IRC00:22
clarkbok DNS is updated 2001:4800:7819:103:be76:4eff:fe04:77e6 and 104.130.246.85 is the new server00:22
clarkbI'll wait about an hour then check connections to old server before deleting old server00:23
clarkbthen will bump the dns ttl back up to an hour00:23
*** esberglu has joined #openstack-infra00:28
*** esberglu has quit IRC00:29
*** EmilienM has quit IRC00:30
*** markmcclain has quit IRC00:30
*** markmcclain has joined #openstack-infra00:32
clarkbmwhahaha: hey if our centos7 image builds haven't updated yet with pti enabled they should be updating soonish. Will be curious to know if you notice significantly longer test runs as a result. (you are biggest consumer of centos7 images I think)00:33
mwhahahaclarkb: ok. We'll keep an eye out00:34
*** xarses_ has quit IRC00:36
*** edmondsw has joined #openstack-infra00:36
*** edmondsw has quit IRC00:41
dmsimardclarkb: TTL is largely a suggestion, you might break people by killing off the old machine right away00:46
*** dave-mccowan has joined #openstack-infra00:48
fungidmsimard: easy enough to figure out by watching logs or tcpdump though00:48
clarkbdmsimard: yaI know I am tailing logs too00:48
clarkbthere has definitely been a transition from one to the other00:48
dmsimard Indeed00:48
clarkbin the past we have had to just break some people though because things like jvm resolve a record once and never reresolve it00:49
*** gyee has quit IRC00:49
clarkbthere is about another half hour on the actual ttl at this point00:50
dmsimardSure00:50
*** slaweq has quit IRC00:53
openstackgerritZara proposed openstack-infra/storyboard master: Use correct tox command for upgrading the database  https://review.openstack.org/53127200:54
*** bobh has joined #openstack-infra00:55
*** slaweq has joined #openstack-infra00:56
*** cuongnv has joined #openstack-infra00:58
*** slaweq has quit IRC01:00
*** markmcclain has quit IRC01:05
*** markmcclain has joined #openstack-infra01:13
*** hoangcx has joined #openstack-infra01:15
*** nicolasbock has joined #openstack-infra01:17
*** kiennt26 has joined #openstack-infra01:18
hoangcxHi Team, I have a small question regarding to this patch: https://review.openstack.org/#/c/522695/. It has just been merged recently. Then I tried to open the docs link: https://docs.openstack.org/neutron-vpnaas/latest/ with unlucky01:20
hoangcxIs the the link the access is correct? Or do i need to do more work to have it? Thanks a lot01:21
clarkbhoangcx: have you merged any docs changes to neutron-vpnaas? those builds run in response to docs changes merging01:22
*** caphrim007 has joined #openstack-infra01:26
clarkbdmsimard: there is a redhat ip still hitting the old git.o.o host. Is your concern thta you know red hat breaks dns? :)01:27
dmsimard(╯°□°)╯︵ ┻━┻01:28
dmsimardMy plan is foiled01:28
clarkbthere are a couple other IPs that I can't identify as well01:28
clarkband some web crawlers01:29
*** caphrim007 has quit IRC01:31
*** slaweq has joined #openstack-infra01:33
*** Apoorva_ has joined #openstack-infra01:33
clarkbat this rate I may just leave the old server up until morning. Production is shifted over to new server but there are a few stragglers and I am running out of daylight. I also expect tomorrow to be a long day assuming we get kernels so don't want to work late tonight01:36
*** Apoorva has quit IRC01:37
clarkb#status log git0*.openstack.org patched and kernels running with PTI enabled. git.openstack.org has been replaced with a new server running with PTI enabled. The old server is still in places for straggler clients. Will need to be deleted and DNS record TTLs set back to one hour01:37
openstackstatusclarkb: finished logging01:37
*** slaweq has quit IRC01:37
*** Apoorva_ has quit IRC01:38
*** hemna_ has quit IRC01:39
fungithanks clarkb!01:45
*** jroll has quit IRC01:46
*** kgiusti has left #openstack-infra01:49
*** gagehugo has joined #openstack-infra01:50
*** jroll has joined #openstack-infra01:51
*** hoangcx has quit IRC01:55
*** hoangcx has joined #openstack-infra01:55
hoangcxclarkb: Ah I see. I will try to merged one doc patch to see the reflection. Thanks for your info/guidance.01:56
*** cuongnv has quit IRC01:57
*** cuongnv has joined #openstack-infra01:57
*** caphrim007 has joined #openstack-infra01:58
*** liujiong has joined #openstack-infra02:02
*** caphrim007 has quit IRC02:03
*** slaweq has joined #openstack-infra02:05
hoangcxclarkb: BTW, Could you please help to check and land this one? https://review.openstack.org/#/c/527872/ Thank you02:12
*** threestrands_ has joined #openstack-infra02:14
*** eharney has quit IRC02:16
*** threestrands has quit IRC02:16
*** zzzeek has quit IRC02:19
*** zzzeek has joined #openstack-infra02:20
*** namnh has joined #openstack-infra02:20
*** zzzeek has quit IRC02:20
*** zzzeek has joined #openstack-infra02:21
*** smatzek has joined #openstack-infra02:24
*** edmondsw has joined #openstack-infra02:24
*** edmondsw has quit IRC02:28
*** markmcclain has quit IRC02:29
*** eharney has joined #openstack-infra02:30
*** mriedem has quit IRC02:31
*** markmcclain has joined #openstack-infra02:34
*** harlowja has quit IRC02:36
*** smatzek has quit IRC02:38
*** slaweq has quit IRC02:39
*** markmcclain has quit IRC02:43
*** markmcclain has joined #openstack-infra02:45
*** slaweq has joined #openstack-infra02:46
*** spzala has quit IRC02:47
*** EmilienM has joined #openstack-infra02:51
*** slaweq has quit IRC02:51
*** xinliang has quit IRC02:52
*** markmcclain has quit IRC02:53
*** markmcclain has joined #openstack-infra02:55
*** caphrim007 has joined #openstack-infra02:59
*** caphrim007 has quit IRC02:59
*** ijw has quit IRC03:00
*** caphrim007 has joined #openstack-infra03:01
*** xinliang has joined #openstack-infra03:04
*** xinliang has quit IRC03:04
*** xinliang has joined #openstack-infra03:04
*** swest has quit IRC03:05
*** swest has joined #openstack-infra03:05
*** spzala has joined #openstack-infra03:06
*** spzala has quit IRC03:12
*** ramishra has joined #openstack-infra03:14
*** nicolasbock has quit IRC03:14
*** slaweq has joined #openstack-infra03:21
*** greghaynes has quit IRC03:23
*** slaweq has quit IRC03:25
*** greghaynes has joined #openstack-infra03:27
*** yamamoto has joined #openstack-infra03:28
*** links has joined #openstack-infra03:29
*** Apoorva has joined #openstack-infra03:30
*** links has quit IRC03:34
*** zhurong has quit IRC03:34
*** eharney has quit IRC03:38
*** eharney has joined #openstack-infra03:39
*** s-shiono has joined #openstack-infra03:42
*** markmcclain has quit IRC03:45
dmsimardJust stumbled on this: "Add a welcome bot to Gerrit for first time contributors" https://phabricator.wikimedia.org/T7335703:45
dmsimardSounds like some people do appreciate our attempts at being welcoming to new contributors :D03:46
*** markmcclain has joined #openstack-infra03:47
*** liujiong has quit IRC03:49
*** liujiong has joined #openstack-infra03:49
*** spzala has joined #openstack-infra03:51
*** spzala has quit IRC03:55
*** annp has joined #openstack-infra03:56
*** links has joined #openstack-infra03:56
*** bobh has quit IRC03:57
*** slaweq has joined #openstack-infra04:00
*** s-shiono_ has joined #openstack-infra04:06
*** udesale has joined #openstack-infra04:08
*** s-shiono has quit IRC04:09
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules  https://review.openstack.org/52294504:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul  https://review.openstack.org/50971504:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager  https://review.openstack.org/52477304:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers  https://review.openstack.org/52462004:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: builder: do not cleanup image for driver not managing image  https://review.openstack.org/51692004:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool  https://review.openstack.org/46862404:13
*** edmondsw has joined #openstack-infra04:13
*** edmondsw has quit IRC04:17
*** sree has joined #openstack-infra04:27
*** psachin has joined #openstack-infra04:33
*** eharney has quit IRC04:34
*** slaweq has quit IRC04:34
*** slaweq has joined #openstack-infra04:36
*** slaweq has quit IRC04:41
*** ijw has joined #openstack-infra04:41
*** ijw has quit IRC04:42
*** ijw has joined #openstack-infra04:42
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Do pep8 housekeeping according to zuul rules  https://review.openstack.org/52294504:50
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Use same flake8 config as in zuul  https://review.openstack.org/50971504:50
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Disambiguate with Netflix and Javascript zuul  https://review.openstack.org/53129204:58
*** sree_ has joined #openstack-infra04:59
*** sree_ is now known as Guest4579004:59
*** sree has quit IRC05:02
*** dave-mccowan has quit IRC05:06
*** hoangcx has left #openstack-infra05:06
*** slaweq has joined #openstack-infra05:08
*** dmsimard has quit IRC05:09
*** markmcclain has quit IRC05:12
*** slaweq has quit IRC05:13
*** markmcclain has joined #openstack-infra05:15
*** iyamahat has joined #openstack-infra05:20
*** janki has joined #openstack-infra05:24
*** dmsimard has joined #openstack-infra05:26
*** dmsimard has quit IRC05:26
*** dmsimard has joined #openstack-infra05:26
*** dmsimard has quit IRC05:32
*** dmsimard has joined #openstack-infra05:34
*** dmsimard has quit IRC05:34
*** dmsimard has joined #openstack-infra05:34
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test job  https://review.openstack.org/51265305:36
*** hongbin has joined #openstack-infra05:37
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs  https://review.openstack.org/51265305:44
AJaegerstephenfin, mordred, here's the updated tox-with-sudo jobs including a functional one ^05:44
*** iyamahat has quit IRC05:45
AJaegerstephenfin: I'll test it with openvif05:45
AJaegerI mean: os-vif05:45
*** slaweq has joined #openstack-infra05:46
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs  https://review.openstack.org/51265305:46
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs  https://review.openstack.org/51265305:49
*** ianychoi has quit IRC05:52
*** caphrim007 has quit IRC05:53
*** Apoorva has quit IRC05:55
*** caphrim007 has joined #openstack-infra05:56
*** iyamahat has joined #openstack-infra05:59
*** aeng has quit IRC05:59
*** jtomasek has joined #openstack-infra06:11
*** threestrands_ has quit IRC06:13
*** slaweq has quit IRC06:19
openstackgerritmegan guiney proposed openstack-infra/openstack-zuul-jobs master: [WIP] docs sitemap generation automation  https://review.openstack.org/52486206:19
openstackgerritArmando Migliaccio proposed openstack-infra/openstack-zuul-jobs master: Restore branch conditional on tempest-full  https://review.openstack.org/53105806:27
*** slaweq has joined #openstack-infra06:28
*** bhavik1 has joined #openstack-infra06:29
openstackgerritChandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template  https://review.openstack.org/53130706:31
chandankumarAJaeger: ^^06:31
chandankumarAJaeger: i am not sure how to use template jobs with in a job template need help on that06:32
*** armaan_ has joined #openstack-infra06:54
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs  https://review.openstack.org/51265306:56
AJaegerchandankumar: commented - you cannot. Just remove them06:58
*** jtomasek has quit IRC07:04
*** pgadiya has joined #openstack-infra07:06
*** slaweq has quit IRC07:07
openstackgerritChandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template  https://review.openstack.org/53130707:11
chandankumarAJaeger: we need one more change in project config to update the jobs for all tempest plugins na?07:12
*** claudiub has joined #openstack-infra07:14
*** armax has quit IRC07:29
*** iyamahat has quit IRC07:35
*** slaweq has joined #openstack-infra07:35
chandankumarAJaeger: https://review.openstack.org/#/c/530024/ please have a look gate jobs are not working07:38
*** slaweq has quit IRC07:40
*** slaweq has joined #openstack-infra07:41
chandankumarAJaeger: regarding this review https://review.openstack.org/#/c/530023 you have given this example https://review.openstack.org/#/c/530099/ so we donot need to move any playbooks here?07:45
*** zhurong has joined #openstack-infra07:48
*** ijw has quit IRC07:48
*** edmondsw has joined #openstack-infra07:49
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: handler: fix support for handler without launch_manager  https://review.openstack.org/52477307:52
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers  https://review.openstack.org/52462007:52
*** edmondsw has quit IRC07:53
*** shardy has joined #openstack-infra07:57
*** shardy has quit IRC08:01
*** shardy has joined #openstack-infra08:03
*** spzala has joined #openstack-infra08:07
*** markvoelker has joined #openstack-infra08:09
*** migi has quit IRC08:11
*** spzala has quit IRC08:11
*** eumel8 has joined #openstack-infra08:12
*** tesseract has joined #openstack-infra08:12
*** gouthamr has quit IRC08:13
AJaegerchandankumar: yes, you need to update project-config as well08:14
AJaegerchandankumar: what do you mean: Gates not working on 530024?08:15
*** florianf has joined #openstack-infra08:15
*** armaan_ has quit IRC08:15
*** armaan has joined #openstack-infra08:16
AJaegerchandankumar: regarding 530023: you don't need to move the playbooks for the two *tox* jobs. just follow that example and use native tox jobs.08:16
*** zhurong has quit IRC08:16
*** armaan has quit IRC08:16
AJaegerchandankumar: checked 530024 - all looks good to me.08:18
*** aviau has quit IRC08:19
*** aviau has joined #openstack-infra08:20
*** aviau has joined #openstack-infra08:20
AJaegerstephenfin: so, testing with 530961 looks good from my side for the tox-with-sudo job. Problem is that the helper is not found but sudo works - so, might need some more work from your side on that one... If you think the job is wrong, please tell us.08:24
*** zhurong has joined #openstack-infra08:27
*** rcernin has quit IRC08:28
*** armaan has joined #openstack-infra08:31
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs project template for all tempest plugins  https://review.openstack.org/53133908:42
*** markvoelker has quit IRC08:44
*** ijw has joined #openstack-infra08:49
*** migi has joined #openstack-infra08:51
*** armax has joined #openstack-infra08:55
*** ijw has quit IRC08:55
*** s-shiono_ has quit IRC08:59
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Add a plugin interface for drivers  https://review.openstack.org/52462009:00
*** jpich has joined #openstack-infra09:02
*** liujiong has quit IRC09:03
*** stakeda has quit IRC09:09
*** armax has quit IRC09:14
*** johnthetubaguy has quit IRC09:17
*** johnthetubaguy has joined #openstack-infra09:18
stephenfinAJaeger: Quite possibly. Going to look into it today and report back what I find09:18
stephenfinAJaeger: And thank you for following up on that. I appreciate it :)09:19
AJaegerstephenfin: you're welcome.09:21
AJaegerconfig-core, please review https://review.openstack.org/#/c/512653/ to add tox-with-sudo jobs and https://review.openstack.org/#/c/530104/ for bindep v3 native testing09:22
*** derekh has joined #openstack-infra09:23
AJaegerstephenfin: you might also need a full devstack setup for the functional tests. Might be that testing with tox is not possible. so, check first whether you get it working on your system.09:23
*** lucas-afk is now known as lucasagomes09:33
*** udesale__ has joined #openstack-infra09:33
*** mnencia has quit IRC09:34
*** udesale has quit IRC09:35
*** mnencia has joined #openstack-infra09:36
*** edmondsw has joined #openstack-infra09:37
*** mnencia has quit IRC09:40
*** markvoelker has joined #openstack-infra09:41
*** edmondsw has quit IRC09:42
*** mnencia has joined #openstack-infra09:43
*** jascott1 has quit IRC09:43
*** jascott1 has joined #openstack-infra09:43
*** bhavik1 has quit IRC09:44
*** jascott1 has quit IRC09:48
*** claudiub|2 has joined #openstack-infra09:51
*** ijw has joined #openstack-infra09:51
*** hongbin has quit IRC09:52
*** claudiub has quit IRC09:54
*** tosky has joined #openstack-infra09:54
*** ijw has quit IRC09:56
*** jtomasek has joined #openstack-infra09:58
aspiersI'd like to register a storyboard task to oslo-specs, but storyboard seems to be missing a whole bunch of projects including that one. Any ideas?10:01
aspiersDo I have to wait for the Oslo team to migrate to storyboard?10:02
*** Guest45790 has quit IRC10:06
*** sree has joined #openstack-infra10:07
*** sree has quit IRC10:09
*** sree has joined #openstack-infra10:09
*** pbourke has joined #openstack-infra10:11
*** e0ne has joined #openstack-infra10:12
*** cuongnv has quit IRC10:13
*** markvoelker has quit IRC10:15
frickleraspiers: seems oslo issues are still being tracked at https://bugs.launchpad.net/oslo10:21
*** kiennt26 has quit IRC10:22
aspiersfrickler: yeah, I figured10:23
frickleraspiers: is there a specific reason you want to use storyboard? most projects are still on launchpad10:25
aspiersfrickler: yes there is: http://lists.openstack.org/pipermail/openstack-sigs/2017-November/000176.html10:27
*** udesale has joined #openstack-infra10:28
aspiersfrickler: but also because it seems to be the Right Thing to Do; IIUC, long-term the community is planning a wholesale migration from launchpad to SB, hence presentations like https://www.youtube.com/watch?v=n_PaKuN4Skk10:29
*** gcb has quit IRC10:29
*** udesale__ has quit IRC10:29
*** namnh_ has joined #openstack-infra10:33
*** namnh_ has quit IRC10:34
*** shardy has quit IRC10:35
eumel8Storyboards are almost pushed by the Foundation. You can contact Kendall for any questions about that.10:35
aspierseumel8: thanks for the info10:35
*** sree has quit IRC10:35
*** sree has joined #openstack-infra10:36
*** namnh has quit IRC10:37
*** sambetts|afk is now known as sambetts10:37
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: [WIP] Refactor status functions, add more status endpoints  https://review.openstack.org/53135410:39
chandankumarAJaeger: https://review.openstack.org/531339 i have added the template here10:39
aspiershrm diablo_rojo is not in this channel10:39
aspiersI guess I'll ask her in #storyboard10:40
frickleraspiers: eumel8: well, IMHO storyboard isn't ready for general use yet, ymmv10:41
aspiersfrickler: sure10:42
*** dtantsur|afk is now known as dtantsur10:42
*** shardy has joined #openstack-infra10:42
*** sree has quit IRC10:42
*** sdague has joined #openstack-infra10:43
AJaegerchandankumar: seen it - but we first need to figure out some problems with pypi, I'm not approving any changes with new publish-to-pypi templates.10:43
*** sree has joined #openstack-infra10:43
chandankumarAJaeger: ack!10:43
*** sree has quit IRC10:48
openstackgerritAndreas Jaeger proposed openstack-infra/storyboard master: Create a StoryBoard gui manual  https://review.openstack.org/32547410:50
AJaegeraspiers: ^10:51
*** sree has joined #openstack-infra10:52
*** ijw has joined #openstack-infra10:52
*** szaher has joined #openstack-infra10:54
frickleraspiers: ah, found your story about storyboard ;) added to the list of issues I consider blockers, but I'm about to discussing that, too https://storyboard.openstack.org/#!/search?tags=blocking-storyboard-migration10:54
*** electrofelix has joined #openstack-infra10:55
*** zhurong has quit IRC10:57
openstackgerritMatthieu Huin proposed openstack-infra/nodepool master: [WIP] Refactor status functions, add more status endpoints  https://review.openstack.org/53135411:01
*** pcaruana has joined #openstack-infra11:01
*** udesale__ has joined #openstack-infra11:02
*** udesale has quit IRC11:05
AJaegerfrickler, could you  review https://review.openstack.org/#/c/512653/ to add tox-with-sudo jobs and https://review.openstack.org/#/c/530104/ for bindep v3 native testing , please?11:06
*** Qiming has quit IRC11:10
*** sree has quit IRC11:10
*** markvoelker has joined #openstack-infra11:11
*** Qiming has joined #openstack-infra11:13
*** sree has joined #openstack-infra11:14
*** sree has quit IRC11:18
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Run pep8 jobs for cinder tempest plugin  https://review.openstack.org/53136111:19
*** sree has joined #openstack-infra11:22
openstackgerritAndreas Jaeger proposed openstack-infra/storyboard master: Create a StoryBoard gui manual  https://review.openstack.org/32547411:25
AJaegerfrickler: check https://review.openstack.org/#/c/529662/ for why 512653 should work...11:26
*** udesale has joined #openstack-infra11:27
openstackgerritMerged openstack-infra/project-config master: Add native Zuul v3 bindep fallback tests  https://review.openstack.org/53010411:28
fricklerAJaeger: oops, doing a git pull, thanks11:28
AJaegerfrickler: we discussed that yesterday in chat and that's why patchset 1 and 2 are completely different ;)11:29
*** udesale__ has quit IRC11:29
*** khappone has quit IRC11:29
AJaegerfrickler: see also the needed-by, it seems to work just fine11:29
AJaeger(the error there is unrelated, sudo works)11:30
*** iyamahat has joined #openstack-infra11:32
AJaegerfrickler: could you review https://review.openstack.org/531307 as well, please? https://review.openstack.org/531339 and https://review.openstack.org/531361 show why I want it ;)11:32
AJaegerthanks!11:32
*** iyamahat_ has joined #openstack-infra11:32
*** Qiming has quit IRC11:33
*** Qiming has joined #openstack-infra11:34
*** nicolasbock has joined #openstack-infra11:35
fricklerAJaeger: ack11:35
*** sree has quit IRC11:36
*** iyamahat has quit IRC11:36
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin  https://review.openstack.org/53136111:36
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Added openstack-tempest-plugins job template  https://review.openstack.org/53130711:40
*** sree has joined #openstack-infra11:40
*** udesale has quit IRC11:44
*** iyamahat_ has quit IRC11:45
*** markvoelker has quit IRC11:45
*** katkapilatova1 has joined #openstack-infra11:53
*** sree has quit IRC11:53
*** katkapilatova1 has left #openstack-infra11:54
*** sree has joined #openstack-infra11:56
*** Qiming has quit IRC11:57
*** Qiming has joined #openstack-infra12:00
*** annp has quit IRC12:00
*** jcoufal has joined #openstack-infra12:04
*** spzala has joined #openstack-infra12:07
*** spzala has quit IRC12:12
*** sree has quit IRC12:16
*** katkapilatova1 has joined #openstack-infra12:20
*** sree has joined #openstack-infra12:22
*** kong has quit IRC12:23
openstackgerritChandan Kumar proposed openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin  https://review.openstack.org/53136112:23
*** smatzek has joined #openstack-infra12:23
*** numans has quit IRC12:24
*** numans has joined #openstack-infra12:24
*** spzala has joined #openstack-infra12:25
*** jamesdenton has quit IRC12:25
*** rhallisey has joined #openstack-infra12:25
*** numans has quit IRC12:25
*** jcoufal_ has joined #openstack-infra12:27
*** spzala has quit IRC12:29
*** jcoufal has quit IRC12:29
*** numans has joined #openstack-infra12:30
*** sree has quit IRC12:30
*** sree has joined #openstack-infra12:34
*** sree has quit IRC12:39
*** markvoelker has joined #openstack-infra12:42
*** sree has joined #openstack-infra12:42
openstackgerritMerged openstack-infra/project-config master: Add #openstack-kayobe to accessbot channel list  https://review.openstack.org/53077812:44
*** lucasagomes is now known as lucas-hungry12:46
*** sree has quit IRC12:47
*** rkukura has quit IRC12:47
*** rkukura has joined #openstack-infra12:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Fix bindep_file for bindep fallback  https://review.openstack.org/53136912:48
*** janki has quit IRC12:50
AJaegerfrickler: the check experimental on the bindep showed that it did not work, see http://logs.openstack.org/25/531225/1/experimental/project-config-bindep-fallback-centos-7/bb3a0a0/job-output.txt.gz . What do you think of fix above?  ^12:51
*** sree has joined #openstack-infra12:56
openstackgerritJens Harbott (frickler) proposed openstack-infra/project-config master: Fix bindep_file for bindep fallback  https://review.openstack.org/53136912:57
AJaegerfrickler: oops - thanks for fixing12:58
*** sree has quit IRC13:05
openstackgerritMerged openstack-infra/project-config master: Fix bindep_file for bindep fallback  https://review.openstack.org/53136913:06
*** psachin has quit IRC13:09
*** jkilpatr has joined #openstack-infra13:09
*** erlon has joined #openstack-infra13:11
*** sree has joined #openstack-infra13:11
openstackgerritJens Harbott (frickler) proposed openstack-infra/devstack-gate master: Add neutron-tempest-plugin-api job  https://review.openstack.org/52900013:12
fricklerAJaeger: as you may have noticed, I started hijacking your etherpad, do you mind if I add ^^ too?13:13
*** edmondsw has joined #openstack-infra13:13
*** markvoelker has quit IRC13:15
AJaegerfrickler: go ahead!13:17
*** edmondsw has quit IRC13:17
AJaegerfrickler: I was planning to retire it but we can keep using it ...13:17
*** kiennt26 has joined #openstack-infra13:18
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: DNM: Test new bindep tests  https://review.openstack.org/53122513:19
AJaegerfrickler: bindep check experimental worked on 53125, now let's do negative test ;)13:20
*** sree has quit IRC13:20
AJaegermordred: could you review https://etherpad.openstack.org/p/zbUQLqDR6H , please?13:21
*** dave-mccowan has joined #openstack-infra13:22
*** sree has joined #openstack-infra13:23
AJaegermordred: eumel8 and myself wonder why the variable is set at all - it breaks openstack-manuals publishing13:23
*** eharney has joined #openstack-infra13:24
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Add /node-list to the webapp  https://review.openstack.org/49996913:26
*** ijw has quit IRC13:26
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: Add /label-list to the webapp  https://review.openstack.org/50025413:26
*** sree has quit IRC13:27
*** jcoufal_ has quit IRC13:28
*** sree has joined #openstack-infra13:28
openstackgerritVasyl Saienko proposed openstack-infra/project-config master: Add relase notes to n-g-s  https://review.openstack.org/53137513:29
*** dbecker has joined #openstack-infra13:30
openstackgerritStibbons proposed openstack-dev/pbr master: Support v<semver> version  https://review.openstack.org/53116113:30
fungidmsimard: we should keep an eye on mw's efforts to reimplement the welcome message feature as a gerrit plugin, since it sounds like we'll need it for our next gerrit upgrade anyway13:31
fungigreat find--thanks for the link!13:32
*** pgadiya has quit IRC13:36
*** markvoelker has joined #openstack-infra13:36
AJaegerconfig-core, frickler and myself cleaned up the backlog list - and added new ones. Please take a few changes from https://etherpad.openstack.org/p/Nvt3ovbn5x for review.13:36
*** fultonj has quit IRC13:37
openstackgerritMerged openstack-infra/zuul-jobs master: Add tox-nodejs-npm base job for python and javascript  https://review.openstack.org/52966313:40
*** rlandy has joined #openstack-infra13:41
*** fultonj has joined #openstack-infra13:45
*** links has quit IRC13:47
*** caphrim007_ has joined #openstack-infra13:47
*** pcichy_ has joined #openstack-infra13:48
*** hughhalf has quit IRC13:48
*** tonyb has quit IRC13:49
*** caphrim007 has quit IRC13:50
*** tonyb has joined #openstack-infra13:50
*** swest has quit IRC13:50
*** pahuang has quit IRC13:50
*** jistr is now known as jistr|biab13:51
*** pgadiya has joined #openstack-infra13:53
*** edmondsw has joined #openstack-infra13:53
*** pgadiya has quit IRC13:53
*** pahuang has joined #openstack-infra13:54
*** hughhalf has joined #openstack-infra13:54
*** spzala has joined #openstack-infra13:56
*** swest has joined #openstack-infra14:00
*** sree has quit IRC14:00
*** sree has joined #openstack-infra14:01
*** spzala has quit IRC14:03
*** sree has quit IRC14:05
*** claudiub|2 is now known as claudiub14:08
*** mriedem has joined #openstack-infra14:12
*** bobh has joined #openstack-infra14:14
openstackgerritStephen Finucane proposed openstack-dev/pbr master: Support v<semver> version  https://review.openstack.org/53116114:19
openstackgerritStephen Finucane proposed openstack-infra/infra-manual master: zuul v3: Expand section on what *to* convert  https://review.openstack.org/52456614:21
openstackgerritStephen Finucane proposed openstack-infra/infra-manual master: Remove version and release  https://review.openstack.org/52456714:21
openstackgerritStephen Finucane proposed openstack-infra/infra-manual master: trivial: Remove cruft from 'conf.py'  https://review.openstack.org/52456814:21
openstackgerritDmitry Tantsur proposed openstack-infra/project-config master: Adjust gerrit ACLs for networking-generic-switch  https://review.openstack.org/53138214:26
*** ijw has joined #openstack-infra14:27
*** bobh has quit IRC14:29
dmsimardfungi: yeah I didn't read the whole thread but it seemed like they wanted to leverage jeepyb14:29
*** spzala has joined #openstack-infra14:30
*** smatzek has quit IRC14:31
*** smatzek has joined #openstack-infra14:32
*** spzala has quit IRC14:35
*** katkapilatova1 has quit IRC14:35
*** smatzek has quit IRC14:37
openstackgerritDmitry Tantsur proposed openstack-infra/project-config master: Adjust gerrit ACLs for networking-generic-switch  https://review.openstack.org/53138214:37
*** ramishra has quit IRC14:37
*** gridinv has quit IRC14:39
*** esberglu has joined #openstack-infra14:39
*** gridinv has joined #openstack-infra14:40
*** esberglu_ has joined #openstack-infra14:42
*** esberglu has quit IRC14:45
*** esberglu_ is now known as esberglu14:45
*** hongbin has joined #openstack-infra14:48
*** jaypipes has joined #openstack-infra14:48
*** jaypipes is now known as leakypipes14:51
*** nicolasbock has quit IRC14:51
*** lucas-hungry is now known as lucasagomes14:52
fungidmsimard: first they did, but then they didn't like relying on the ssh api (understandable, there was no rest api when we wrote that originally) and then they decided that relying on the gerrit database is fragile (which it is) and noted that the design will break completely with gerrit 2.14, so proposed reimplementing as a gerrit plugin, which i think is a grand idea now that the plugin interface14:57
fungihas been ironed out14:57
fungiif i had the slightest bit of familiarity with java, i'd have offered to help since we'll need it anyway14:58
*** smatzek has joined #openstack-infra15:02
*** jistr|biab is now known as jistr15:04
*** sree has joined #openstack-infra15:07
*** Goneri has joined #openstack-infra15:13
pabelangerwow, git05.o.o spam happy yesterday :)15:16
openstackgerritSam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest  https://review.openstack.org/53072615:17
*** sree has quit IRC15:21
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53139015:21
*** gouthamr has joined #openstack-infra15:22
fungipabelanger: we seem to spur the git servers on to send out their backlog of accumulated crash reports when we reboot them. for some reason those aren't getting sent when they happen (perhaps because their local mta is stopped?)15:22
pabelangerfungi: okay, is anybody looking into that?15:23
funginot that i'm aware of... it's hard to pin down because we don't find out about it until it's fixed15:24
pabelangeralso, does look like git servers are updated to 7.4. Going to see how we can turn on automatic updates now, like ubuntu15:24
pabelangerfungi: ack15:24
fungisomeone could start, i suppose, by trying to generate e-mail from those servers to root and seeing if we receive them or they end up queued somewhere15:25
*** weshay is now known as weshay_interview15:28
*** wolverineav has joined #openstack-infra15:31
*** caphrim007_ has quit IRC15:32
*** caphrim007 has joined #openstack-infra15:33
mriedemclarkb: is there a known issue with getting cinder-specs docs refreshed? https://specs.openstack.org/openstack/cinder-specs/15:37
mriedemhasn't been updated since 12/2115:37
*** caphrim007 has quit IRC15:37
AJaegermriedem: from 21st to 2nd we didn't publish any docs by accident. Should be fixed now.15:38
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53139015:39
AJaegermriedem: https://wiki.openstack.org/wiki/Infrastructure_Status actually from 23rd to 3rd15:39
mriedemAJaeger: so are the docs updated on a cron?15:39
mriedemor does something have to be forced15:39
AJaegermriedem: no, as post jobs. So, merge something and it should publish15:39
mriedemoh15:39
AJaegerit was just broken for that time ;(15:39
mriedemis there a way to do a manual publish?15:40
fungimriedem: a zul admin can manually reenqueue the ref for the branch tip into the post pipeline15:41
fungier, a zuul admin (sorry zul!)15:41
mnasermriedem: time to fix the copyright year in cinder-specs)15:41
*** kiennt26 has quit IRC15:41
mriedemmnaser: smcginnis said he'd push a cleanup patch15:41
*** spzala has joined #openstack-infra15:43
AJaegerconfig-core, the old backlog review list is cleaned up - but frickler and myself added some more items to it. Please take a few changes from https://etherpad.openstack.org/p/Nvt3ovbn5x for review.15:46
smcginnismnaser: In a doc, or is that the generated timestamp?15:48
mnasersmcginnis: i was making a bad joke about making a bunch of patches that just update dates :>15:49
smcginnismnaser: Haha, pretty close to what I did. ;)15:49
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Install yum-crontab on CentOS servers  https://review.openstack.org/53140115:50
*** iyamahat has joined #openstack-infra15:50
*** stephenfin is now known as finucannot15:53
*** slaweq has quit IRC15:56
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Install yum-crontab on CentOS servers  https://review.openstack.org/53140115:56
*** iyamahat has quit IRC15:59
*** iyamahat has joined #openstack-infra16:03
openstackgerritFabien Boucher proposed openstack-infra/zuul-jobs master: Add roles for static web content build and publication  https://review.openstack.org/53140616:05
*** mtreinish has quit IRC16:08
*** mtreinish has joined #openstack-infra16:09
mordredAJaeger: re: zuul_work_dir - responded a little on that etherpad16:09
AJaegermordred: thanks. Reading your comments, did I understand you correctly that you will fix this?16:11
AJaegermordred: btw. 531406 above might be interesting for you to review16:11
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Don't treat finger client disconnect as exception  https://review.openstack.org/53140816:12
*** ijw has quit IRC16:13
openstackgerritSam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest  https://review.openstack.org/53072616:14
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53139016:18
mordredAJaeger: yah - on it16:23
*** yamamoto has quit IRC16:26
AJaegermordred: great, thanks16:27
*** weshay_interview is now known as weshay16:30
*** pcichy_ has quit IRC16:30
fungiinfra-root: a useful data point on meltdown patching (while waiting for ubuntu to get their packages built)... my debian/stable systems in rackspace rebooted with the purported patched kernel do have CONFIG_PAGE_TABLE_ISOLATION=y in /boot/config-`uname -r` and "Kernel/User page tables isolation: enabled" shows in dmesg output, but no cpu_insecure in /proc/cpuinfo (even though it's GenuineIntel) and16:34
fungino /sys/kernel/debug/x86/pti_enabled or /proc/config.gz to check16:34
*** e0ne has quit IRC16:34
dmsimardugh16:34
fungii have a feeling ubuntu trusty and xenial will be similar16:34
*** benj_ has joined #openstack-infra16:35
corvuscan we just build our own kernels?  /me ducks16:38
mnaseror you can use centos /me ducks16:38
openstackgerritMerged openstack-infra/statusbot master: Fix continuation character under Python 2  https://review.openstack.org/52746116:39
mnaseralso side note does anyone know how can one get an openstack cloak on freenode :>16:39
*** caphrim007 has joined #openstack-infra16:39
*** armax has joined #openstack-infra16:39
corvusmnaser: we rebooted centos yesterday and none of the things that are supposed to tell us kpti is enabled were present.16:39
*** caphrim007 has quit IRC16:39
mnasercorvus: CONFIG_KAISER=y16:39
corvusmnaser: it's the most frakenkernel of them all16:39
mnaserin /boot/config-$(uname -r)16:40
mnaser(i only found this because i was looking through the git repos)16:40
*** openstackstatus has quit IRC16:40
corvusmnaser: wow, didn't they stop calling it kaiser in like october?16:40
*** caphrim007 has joined #openstack-infra16:40
dmsimardcorvus: we found a reliable way to tell after you left, actually16:40
mnaserhttps://git.centos.org/commitdiff/rpms!kernel.git/6202b1a49906b21f0cdce08b4ff853a65c65875016:40
mnaserthats how i found out it was that anyways16:41
dmsimardcorvus: /sys/kernel/debug/x86/pti_enabled will be present and 1 if enabled16:41
*** caphrim007 has quit IRC16:41
*** openstackstatus has joined #openstack-infra16:41
*** ChanServ sets mode: +v openstackstatus16:41
*** caphrim007 has joined #openstack-infra16:41
dmsimardhowever it turns out that debian's flavor doesn't have that so..16:41
openstackgerritDarragh Bailey proposed openstack/python-jenkins master: Pass form data as body instead of in the url  https://review.openstack.org/47495216:41
mnaseralso, i think it might have to do with the qemu-kvm which runs the VMs16:41
openstackgerritDarragh Bailey proposed openstack/python-jenkins master: Migration to using requests  https://review.openstack.org/20563716:41
*** caphrim007 has quit IRC16:41
mnaserbecause the qemu-kvm patch for libvirt exposes microcode related stuff to the guests16:42
* mnaser looks for patch again16:42
mnaserhttps://git.centos.org/blob/rpms!libvirt.git/4cd28d3f2984894ad10b69b4cd5880a9c8d0e156/SOURCES!libvirt-cpu-add-CPU-features-and-model-for-indirect-branch-prediction-protection.patch16:42
dmsimardyeah that's if you expose host-model16:43
dmsimardit depends, you could expose a generic qemu processor for example16:43
mnaserbut i think by default it picks one of those based on your capabilities16:43
*** caphrim007 has joined #openstack-infra16:43
*** mikal has quit IRC16:43
dmsimardwhat default? the one from nova?16:43
dmsimardI think the default is host-model16:43
dmsimardwhich is a subset of all the host processor flags which should have the most compatibility (for things like live migration etc)16:44
mnaseryep16:44
mnaserso with that patch now that processor flag includes the microcode related stuff16:44
*** mikal has joined #openstack-infra16:45
*** jaosorior has quit IRC16:45
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53139016:46
fungiremember we're mostly talking about hosts in rackspace at the moment, so these are on xen anyway16:46
*** katkapilatova1 has joined #openstack-infra16:47
fungithere are some similar xen patches out as of yesterday, but its hypervisor is also purportedly immune to meltdown getting attacked by guests if you're using hvm for all guests (which all the flavors we've booted are) and they provided a hypervisor shim you can live migrate your pv guests on top of to mitigate attacks from them as well16:48
*** yamamoto has joined #openstack-infra16:50
*** Apoorva has joined #openstack-infra16:51
eharneyi'm seeing some odd behavior with irrelevant-files filtering here:  https://review.openstack.org/#/c/531197/16:52
eharneyAFAICT tempest-full shouldn't run here, because of https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml?id=0a93ac#n3892   ?16:52
*** ijw has joined #openstack-infra16:55
fungieharney: i want to say irrelevant-files doesn't work in a job variant unless it's a subset of irrelevant-files in the original job definition due to zuul not knowing how to deselect a job via the variant once it's been selected to run16:55
fungisomeone should fact-check me on that one ^16:56
corvuseharney: what fungi said plus: it's been selected to run because tempest-full is in the integrated-gate template16:56
AJaegerfungi: I run 4.14.11 on openSUSE have cpu_insecure, and the dmesg output of enabled - but no pti_enabled16:56
smcginnisOh right, it can't be in a child definition, it needs to be at the source, right?16:56
eharneyok, should we just clean out the irrelevant-file filters in project-config there, then?16:56
*** ijw_ has joined #openstack-infra16:57
smcginnisAnd speaking of tempest-full, any update on that job with stable branches?16:57
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul feature/zuulv3: Delete IncludeRole object from result object for include_role tasks  https://review.openstack.org/50423816:57
*** armaan has quit IRC16:57
fungiAJaeger: is that on bare metal or a virtual machine?16:57
AJaegerfungi: bare metal16:58
*** armaan has joined #openstack-infra16:58
fungii'm still waiting for updated kernel packages in debian/unstable, which is all i have on my bare metal systems so far16:58
corvuseharney, smcginnis: yeah, if you want it not to run there, you can add those irrelevant-files to the job itself, or you could drop the use of the integrated-gate template and keep the project stanza you have.16:58
AJaegersmcginnis: there's a patch for tempest that needs to merge16:58
dmsimardmordred: btw mnaser asked earlier if anyone knew how to get an openstack freenode cloak ?16:58
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Remove explicit bindep parameters  https://review.openstack.org/53143016:58
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add zuul_work_dir defaults to base-test  https://review.openstack.org/53143116:58
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Set defaults for zuul_work_dir and bindep_dir  https://review.openstack.org/53143216:58
mnaserdmsimard: you somehow have one16:59
mnaser:P16:59
corvuseharney, smcginnis: (or add those irrelevant-files to the integrated-gate template)16:59
mordreddmsimard: corvus is the right person to ask about knowing how - I think we have the ability but currently have no defined process for handing them out?16:59
dmsimardmnaser: it's a red hat cloak16:59
dmsimardmnaser: redhat/openstack16:59
mnaserohhh i see16:59
AJaegersmcginnis: https://review.openstack.org/#/c/53104616:59
AJaegersorry, smcginnis, check https://review.openstack.org/#/c/531305/17:00
dmsimardmordred: it needs to be handed out by ircops regardless but I believe someone in specific needs to do the request17:00
smcginnisAJaeger: Thank you!17:00
corvusmordred: oh, did i inherit that too?  i've never worked on getting a cloak.  :)17:00
mordredcorvus: I just figure you have all the mega freenode powers17:00
*** ijw has quit IRC17:00
corvusmordred: according to the docs, yes...17:00
AJaegersmcginnis: 531046 merged 3h ago and should fix it, rest is cleanup. Sorrry, was a bit confused...17:00
mordreddmsimard, corvus: I got my openstack cloak from rick clark 7 years ago17:01
dirkAJaeger: doesn't loading ucode-intel remove that cpu_insecure marker?17:01
dmsimardthat name doesn't ring me a bell, I'm a newbie from grizzly :(17:01
mordreddmsimard: :)17:01
AJaegerdirk: don't think so17:01
dirkAJaeger: fungi: iirc the official kernel updates for opensuse are meanwhile released17:01
openstackgerritMatthieu Huin proposed openstack-infra/nodepool feature/zuulv3: [WIP] Refactor status functions, add web endpoints, allow params  https://review.openstack.org/53139017:02
corvusmnaser, mordred, dmsimard: if someone wants to document the criteria for obtaining a cloak, i'm happy to execute the process as the openstack group contact.  :)17:02
mordreddmsimard: he was the rackspace engineering manager over the team who initially kicked off the effort that became openstack and was on thefirst several instaces of the PPB17:02
mnaserhttps://meta.wikimedia.org/wiki/IRC/Cloaks -- i think this is a nice base document17:02
fungidirk: awesome. and yeah, my debian/stable systems are already patched, i'm waiting for equivalent patches for debian/unstable at this point (which tends to lag since it doesn't get explicit security team support)17:03
mnasermy question is does corvus have access to doing them directly or does he have to reach out to staff, because that might be a lot of work to take on17:03
mordredcorvus: I kind of think the criteria should be "any active ATC who requests one"17:03
fungidirk: but ubuntu unfortunately still seems to have no updated kernel packages17:03
dmsimardmnaser: only ircops can hand out cloaks, but someone authorized to do so must do the request17:03
* Shrews wants to trade his invisibility cloak for the elder wand17:03
mordredcorvus: but given I think the mechanism is "corvus requests a cloak" - we might want to make the request one from corvus have a larger barrier to entry17:03
fungiShrews: i have a +3 lart of smiting if you need to borrow one17:04
dmsimardyeah "criteria to obtain a cloak" is certainly a good question17:04
corvusmnaser: i believe i will have to request access.  but i'm happy to do it.  if there are a lot of requests (there have been zero over the past 4 years), i will probably batch them.  :)17:04
mordredcorvus: unless we wanted to make the openstack bot account the openstack group contact :)17:04
Shrewsfungi: lol17:04
mugsieso they never finished the cloak / group management tool?17:04
dmsimardthere is a group management thing now17:04
*** mriedem is now known as mriedem_luncharo17:04
*** Apoorva has quit IRC17:04
dmsimardthey reserve a namespace automatically when you register a "primary namespace" channel17:04
dmsimardas well as a cloak17:05
dmsimardbut you still need to ask for the cloak17:05
dmsimardlike I have the #ara namespace reserved and I could have an ara cloak17:05
openstackgerritEric Harney proposed openstack-infra/project-config master: Clean up irrelevant-files for Cinder tempest-full  https://review.openstack.org/53143417:05
mnaseryeah the asking for cloak happens manually17:05
dmsimardbut someone could make ##ara in the silly secondary namespace thing17:05
dmsimardbut no one could register #ara-is-awesome without my consent17:06
mordredAJaeger: I pushed up three patches - https://review.openstack.org/531430 should fix your original issue17:08
corvusmnaser: yeah, that wiki page looks like a good basis.  if something like that lands in infra-manual, i'll make the requests.17:08
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config master: Switch to pep8 jobs for manila-specs  https://review.openstack.org/52115917:08
*** jpich has quit IRC17:08
mordredAJaeger, dmsimard, corvus, pabelanger: https://review.openstack.org/531432 is a followup idea that I'd like opinions on before I do any of the work to verify it works17:08
clarkbgood morning. Looks like no USNs yet17:09
dmsimardmordred: should it just be included in zuul proper ?17:09
clarkbas far as rax and our control plane goes apparently xen is only vulnerable in a limited set of use cases (64bit PV guests can attack xen but no others and all our instances should be hvm so if instances are segregated by xen mode or if rax stopped doing pv instances we are good)17:09
dmsimardat least the work dir17:09
clarkbI'm going to take a look at old git.o.o now and prep for deleting it17:10
dmsimardmordred: wait, I'm thinking of something else -- nevermind what I said17:10
AJaegermordred: thanks, reviewing17:10
chandankumardmsimard: please have a look on this https://review.openstack.org/#/c/531361/ thanks :-)17:11
mordreddmsimard: yah - must be in base if we do it anywhere - otherwise variable precedence will do the wrong thing17:11
dmsimardchandankumar: you're in luck, I'm actually doing a batch of review right now17:11
*** dtroyer has quit IRC17:11
corvusmordred: is that so that we can drop zuul_work_dir from defaults/main.yaml in all roles?  anything else?17:11
*** jtomasek has quit IRC17:12
openstackgerritMerged openstack-infra/project-config master: neutron-vpnaas: Move permission to neutron-release/neutron-stable-maint  https://review.openstack.org/52954217:12
dmsimardAJaeger, frickler, chandankumar: I added a comment in https://review.openstack.org/#/c/531307/ -- we should be able to prevent those, indentation mistakes in yaml should not land17:12
AJaegermordred: readd the bindep_profile: doc to 531430, please17:12
dmsimardchandankumar: can you please submit a follow up fix to address the indentation mistake17:13
chandankumardmsimard: yup17:13
dmsimardchandankumar: there's actually two indentation mistakes17:13
dmsimardadded a second comment17:13
AJaegerdmsimard: that template is not used currently ;( We should test it - but don't ;(17:13
AJaegerdmsimard: good catch, thanks!17:14
dmsimardcorvus: does zuul do some amount of yaml linting-ish checks when it validates the configuration or should we add a job to detect things like bad indentation ?17:14
dmsimardcorvus: i.e, https://review.openstack.org/#/c/531307/2/zuul.d/project-templates.yaml17:15
AJaegerdmsimard: if you want to review, you could also go over the entries in https://etherpad.openstack.org/p/Nvt3ovbn5x - but whatever works for you... I put in that list a backlog over some of our repos and frickler enhanced17:15
*** ijw_ has quit IRC17:15
dmsimardAJaeger: going through my two dashboards right now, I'll hit the pad after17:15
corvusdmsimard: it does not lint for style.17:16
openstackgerritChandan Kumar proposed openstack-infra/openstack-zuul-jobs master: Fixed indentation typo in tempest plugin job template  https://review.openstack.org/53143617:16
dmsimardchandankumar: thanks17:17
dmsimardcorvus: style in yaml can arguably lead to syntax errors (or outright unexpected behavior) though is mostly what I'm worried about17:17
AJaegerdmsimard: then you might have found everything on that one ;) Thanks17:17
openstackgerritMerged openstack-infra/project-config master: Switch to pep8 jobs for manila-specs  https://review.openstack.org/52115917:18
clarkbok git.openstack.org is being hit by one web search indexer and a handful of IPs owned by advancedhosters.com17:19
clarkber old git.o.o17:20
AJaegerchandankumar: thanks!17:20
clarkbinfra-root ^ any idea who advancedhosters.com could be? and should I go ahead and delete the old server at this point?17:20
*** bobh has joined #openstack-infra17:20
*** markmcclain has quit IRC17:20
*** markmcclain has joined #openstack-infra17:21
*** pcaruana has quit IRC17:21
corvusdmsimard: yeah, there's one point that i fully agree with that on: always indent after a colon.  not doing that frequently leads people to errors (especially between lists and dicts), so we've adopted that as the house style in docs and zuul-jobs.  i think it might be okay to try out a linter to catch that and ensure consistent indentation on zuul-jobs and maybe the other repos.  i wouldn't want zuul itself to enforce indentation ...17:22
corvus... though (eg, 531307 should always be accepted by zuul, but we could locally enforce more consistency)17:22
*** dtantsur is now known as dtantsur|afk17:22
fungiclarkb: on the xen meltdown front, my take is that we still need guest kernel updates to secure processes on our guests from other processes on our guests, but that rackspace has probably mostly dodged the bullet from the perspective of guests being able to attack other guests or attack the hypervisor host17:23
clarkbfungi: right we still need to patch to protect within our VMs but other tenants are hopfully locked away because xen17:23
clarkbalso firefox is mitigating attack via javascript by reducing timer precision. So updating your firefox to 57 or greater is a good idea (even with a patched kernel)17:25
openstackgerritMerged openstack-infra/project-config master: Add api-ref job to blazar jobs  https://review.openstack.org/53005117:25
fungiclarkb: www.advancedhosters.com looks to be somewhere around amsterdam according to my traceroutes. dunno if that helps. at this point anyone who has continued hitting a host we took out of dns >12 hours ago (with a 1h ttl) can't exactly complain17:26
corvusi'm happy that we only run code from trusted projects on the executors17:26
fungicorvus: same here!17:26
fungiand the extra layers of effort we went to in securing secrets17:26
fungiso that zuul won't provide them to hosts running untrusted code17:27
fungi(at least insofar as our code review process manages to catch that)17:27
*** iyamahat has quit IRC17:30
*** trown is now known as trown|lunch17:30
clarkbalright I am deleting 39106fe5-2e12-4ca0-b5ba-b79e35abc590 in rax dfw now. This is the old git.openstack.org server17:32
clarkbthat is done, increasing the dns ttl back to 1 hour on git.o.o records17:35
fungihuh, so openbsd has had effective meltdown workarounds in place for a decade... https://bsd.network/@phessler/9929127411932930917:35
fungimakes me happy to continue placing my trust in it17:36
*** danpawlik has quit IRC17:37
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Fixed indentation typo in tempest plugin job template  https://review.openstack.org/53143617:37
fungialways nice when theo's rants create results https://marc.info/?l=openbsd-misc&m=118296441702631&w=217:38
*** florianf has quit IRC17:39
clarkb#status log Old git.openstack.org server has been deleted. New server's A and AAAA dns record TTLs bumped to an hour. We are now running with PTI enabled on all CentOS control plane servers.17:40
openstackstatusclarkb: finished logging17:40
mordredclarkb: woot17:40
* fungi wonders whether we can rename our project testing interface to "consistent testing interface" for purposes of disambiguation :P17:41
*** niska has quit IRC17:42
dmsimardeither that or we need a branding, logo and our own website for our version of pti17:44
dmsimardand two or three nice marketing slogans17:44
*** yamamoto has quit IRC17:47
johnsommordred Is there anything more that needs to happen with https://review.openstack.org/#/c/521211/ ?17:47
SamYaplefungi: too many bad experinces with/around theo :( turned me off of openbsd, but i wont disparange openbsd at all17:47
*** niska has joined #openstack-infra17:47
fungii've found him less vitriolic than linus in recent years ;)17:48
*** derekh has quit IRC17:48
SamYaplewell you arent wrong there. but i also dont deal/hear about linus on the daily when using linux. i do hear theo-isms in openbsd non-stop17:49
SamYaplei just want cool software ok!17:49
fungifair point17:49
openstackgerritMerged openstack-infra/project-config master: Use tempest-plugin-jobs template for cinder-tempest-plugin  https://review.openstack.org/53136117:49
pabelangerinfra-root: https://review.openstack.org/531401/ could use a review please, that should keep our centos servers packages up to date, as we do under ubuntu17:50
*** shardy is now known as shardy_afk17:53
*** jascott1 has joined #openstack-infra17:53
*** iyamahat has joined #openstack-infra17:54
*** Apoorva has joined #openstack-infra17:54
*** xarses_ has joined #openstack-infra17:54
*** jascott1 has quit IRC17:54
clarkbpabelanger: +2'd but left some comments on one behavior I noticed updating things yesterday and another is a minor puppet nit17:56
*** sree has joined #openstack-infra17:56
*** dave-mccowan has quit IRC17:57
*** lucasagomes is now known as lucas-afk18:02
*** yamamoto has joined #openstack-infra18:03
*** yamamoto has quit IRC18:07
*** harlowja has joined #openstack-infra18:09
*** sree has quit IRC18:09
AJaegerconfig-core, the tox-with-sudo jobs works, could mordred and myself get a second +2 on https://review.openstack.org/512653, please?18:13
clarkb"Ubuntu users of the 64-bit x86 architecture (aka, amd64) can expect updated kernels by the original January 9, 2018 coordinated release date, and sooner if possible."18:14
clarkbAJaeger: I will take a look18:14
*** ijw has joined #openstack-infra18:15
dmsimardyeah the coordinated release date was thrown out the window and they released the vuln info before the planned date18:16
dmsimardso it caught some vendors off guard18:16
dmsimardthe vulns were getting way too much exposure even before they were officially announced18:16
*** mriedem_luncharo is now known as mriedem18:17
*** sree has joined #openstack-infra18:17
clarkbAJaeger: mordred why does js require sudo18:17
*** jascott1 has joined #openstack-infra18:19
clarkboh is it because pre of parent runs before your pre? that must be it18:19
AJaegerclarkb: yes18:20
clarkbthey still revoke sudo just later in the lits of things18:20
openstackgerritMerged openstack-infra/project-config master: Add #openstack-self-healing to accessbot  https://review.openstack.org/53113518:20
*** sree has quit IRC18:26
*** chandankumar is now known as {archer}18:27
*** pcichy_ has joined #openstack-infra18:28
*** pcichy_ has quit IRC18:29
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Add openstack-tox-with-sudo test jobs  https://review.openstack.org/51265318:30
*** sree has joined #openstack-infra18:30
*** electrofelix has quit IRC18:30
*** pcichy_ has joined #openstack-infra18:31
openstackgerritMiguel Lavalle proposed openstack-infra/project-config master: Remove Neutron legacy jobs definition  https://review.openstack.org/53050018:33
*** sambetts is now known as sambetts|afk18:37
*** dtroyer has joined #openstack-infra18:37
*** armaan has quit IRC18:44
*** trown|lunch is now known as trown18:45
*** armaan has joined #openstack-infra18:47
*** sree has quit IRC18:47
*** yamamoto has joined #openstack-infra18:48
clarkbwith 512653 merged above I'm going to mark all the stuff on zuulv3-issues related to lack of sudo + tox as fixed (jobs will need to be updated to do the right thing but now there is a clear right thing for them to do)18:48
*** katkapilatova1 has quit IRC18:48
*** numans has quit IRC18:50
*** numans has joined #openstack-infra18:51
*** yamamoto has quit IRC18:52
*** pcichy_ has quit IRC18:53
*** rlandy has quit IRC18:56
AJaegerclarkb: thanks18:58
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147018:58
*** ilpianista_ has quit IRC19:03
*** ilpianista_ has joined #openstack-infra19:06
zxiirofungi: We'd like to cut a jjb 2.0 b3 release. Are you the one who can do that?19:06
fungizxiiro: the acl says this is the group which can do it: https://review.openstack.org/#/admin/groups/jenkins-job-builder-release19:08
fungizxiiro: i notice that zaro is in there explicitly, but we can certainly add more if clarkb says it's okay19:09
clarkbfungi: zxiiro ++ to delegating that to the active devs19:09
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147019:09
fungizxiiro: https://docs.openstack.org/infra/manual/drivers.html#tagging-a-release is the process for creating and pushing a tag19:10
clarkbI'm happy for zxiiro to do it if zxiiro is happy to do it19:10
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fake more of the github3 api  https://review.openstack.org/53147819:10
*** openstack has joined #openstack-infra21:14
*** ChanServ sets mode: +o openstack21:14
fungiinfra-root: how have we been dealing with these servers which got built with no swap space? were there more during the xenial upgrade sprint which ran into this?21:15
fungithe servers i built during the sprint have swap partitions, so it doesn't seem to have impacted everything21:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove explicit bindep parameters  https://review.openstack.org/53143021:18
AJaegermordred: please check my change ^21:18
* AJaeger calls it a day, would appreciate somebody telling me where i screwed up with https://review.openstack.org/531470 a21:19
fungithanks AJaeger!21:19
*** smatzek has quit IRC21:20
*** smatzek has joined #openstack-infra21:20
*** hughhalf has quit IRC21:20
*** tonyb has quit IRC21:21
fungifor some reason, there is no ephemeral disk on eavesdrop0121:23
*** smatzek has quit IRC21:25
*** numans has quit IRC21:25
*** erlon has quit IRC21:27
*** slaweq has joined #openstack-infra21:27
*** numans has joined #openstack-infra21:28
*** slaweq has quit IRC21:28
*** slaweq has joined #openstack-infra21:29
fungipabelanger: my irc log says you were the one who did the eavesdrop xenial rebuild during the sprint? do you happen to know if there was anything unusual about booting that which might have caused it to get no ephemeral disk?21:31
fungiin the interim, i'm going to create a swapfile on the rootfs. it has plenty of space21:32
openstackgerritMiguel Lavalle proposed openstack-infra/openstack-zuul-jobs master: Remove Neutron legacy jobs definition  https://review.openstack.org/53149621:33
*** ijw has joined #openstack-infra21:33
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Add regex support to project stanzas  https://review.openstack.org/51336821:35
fungi#status log added 2gb swap file to eavesdrop01 at /swapfile since it has no ephemeral disk21:35
openstackstatusfungi: finished logging21:35
dmsimardfungi: that's a hotfix but if 1GB isn't enough we should consider making it 2GB :)21:36
*** trown is now known as trown|outtypewww21:37
openstackgerritGage Hugo proposed openstack-infra/project-config master: Remove legacy jobs from Keystone  https://review.openstack.org/52423221:38
fungidmsimard: yeah, i'd like to hear back from pabelanger once he's recovered to discuss whether there was a specific reason we went with a 1gb flavor21:38
openstackgerritIhar Hrachyshka proposed openstack-infra/elastic-recheck master: grenade: wait for service logs before matching against patterns  https://review.openstack.org/49398721:39
dmsimardfungi, apevec: re - unreachable nodes, the first step was to establish if there was a pattern in which the issues occured and we're thus tracking them with elastic-recheck queries21:40
dmsimardsee http://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=76b9afa9d977a8e95cd349df2091f67e2cbbbf80 and http://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=9358e081deaed58f0a8a0e1972558752e2f707ef21:41
dmsimardI admittedly haven't spent time looking through those since the holidays21:41
openstackgerritGage Hugo proposed openstack-infra/project-config master: Remove legacy jobs from Keystone  https://review.openstack.org/52423221:41
*** hashar has joined #openstack-infra21:41
dmsimarde-r shows 55 hits in 10 days for privilege escalation timeout and, oh, yikes, 853 fails in 10 days for unreachable tasks21:42
dmsimardthere was a definitive spike yesterday, I'll see if we can understand what's happening.21:43
* dmsimard puts on good music21:43
fungidmsimard: so if you see the discussion we had about this particular incident, it wasn't unreachable, but was no longer responding to ansible task initiation and so timing out (if it had disconnected and attempted to reconnect over ssh, it would have seen a broken pipe after a ~2-minute hang, assuming my experience testing with an ssh client is indicative)21:43
dmsimardfungi: incident == packstack ?21:44
*** sree has joined #openstack-infra21:44
fungiyep21:44
dmsimardis this the issue where ansible basically hangs forever until the executor kills it ?21:44
fungisounds similar at least21:45
dmsimardI've been meaning to troubleshoot that. I've seen it happen a few times.21:45
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Share a fake pull request database across connections  https://review.openstack.org/53150621:47
dmsimardinfra-root: random finding: zuul-executor being oom' killed (on ze02)21:47
fungianyway, on eavesdrop01 seems it's been hovering around 0.7gb ram used most of the time. something happened today to cause memory utilization to spike suddenly, i think. maybe someone spammed it with privmsg or something21:47
*** gouthamr has joined #openstack-infra21:47
dmsimardhttp://paste.openstack.org/raw/639284/21:48
*** Apoorva has joined #openstack-infra21:48
dmsimardthat paste got truncated for some reason21:48
corvusdmsimard: that will probably keep happening until we have a ram governor21:48
fungidmsimard: yeah, lodgeit (what paste.o.o hosts) has a fairly smallish content size limit21:49
openstackgerritZara proposed openstack-infra/storyboard master: Pin version of webob to 1.7.4  https://review.openstack.org/53150721:49
dmsimardcorvus: it's thundering herd symptom you think ? should we consider adding a few executors ? looking at grafana.o.o they're pretty busy21:49
dmsimardsometimes running >80 concurrent jobs is probably asking a lot of ssh and throughput in general21:50
*** dave-mccowan has quit IRC21:50
Shrewsdmsimard: that's actually probably the finger daemon being killed, not the actual executor21:50
Shrewssince it's a child process21:50
dmsimardShrews: when Ansible runs, the output of the command (non-streamed) is buffered on the control node or on the client node ? I ask in terms of RAM usage21:51
Shrewsstreaming module just reads from the console.log file21:52
dmsimardyeah I know that, but I'm going to guess we're not actually discarding the original output21:52
dmsimardeither something's wrong with ze09 and ze10 or something's up with grafana.. their reported values are static21:54
corvusdmsimard: it's buffered on the remote side, then transmitted back json encoded in the normal way21:54
clarkbfungi: re eavesdrop I think we've typically just created a swap partition and shuffled data around as necessary to make that happen21:54
dmsimardoh sweet21:54
dmsimarddmsimard@ze09:~$ touch me21:54
dmsimardtouch: cannot touch 'me': Read-only file system21:54
clarkbfungi: but if not ephemeral disk is prsent that would be a problem. Is that a flavor thing?21:54
dmsimardinfra-root: I'm going to reboot ze0921:55
corvusdmsimard: i don't know off hand if ansbile stores all of that in memory when decoding it, or otherwise.21:55
dmsimardlooks like a live migration gone bad http://paste.openstack.org/raw/639285/21:55
*** Apoorva_ has joined #openstack-infra21:56
corvusdmsimard: if you can clear out /var/lib/zuul/builds before the executor restarts, that would be good21:56
fungidmsimard: check dmesg for info on why the filesystem was marked read-only?21:56
dmsimardcorvus: read only filesystem21:56
corvusdmsimard: i understand :)21:56
dmsimardfungi: dmesg in paste21:56
fungidmsimard: aha, i was still catching up21:57
dmsimardcorvus: I'll try and do it quick after the reboot?21:57
fungiand yes, sounds likely21:57
corvusdmsimard: so you'll need to remount it before rebooting, or configure it not to start on boot21:57
*** jtomasek has quit IRC21:57
corvusdmsimard: do not race you will lose21:57
dmsimardcorvus: I can systemctl disable21:57
clarkbdmsimard: evne on a ro fs?21:57
dmsimardah, guess I can't do that heh21:57
dmsimardhmm21:57
clarkbremounting rw is probably best option21:57
*** sree has quit IRC21:58
dmsimardok let me try21:58
corvus/var/lib/zuul/builds is not on the / fs, fwiw.21:58
dmsimardyeah I just noticed21:58
corvus(perhaps zuul-executor should clear out its build directory on start?)21:58
fungiyeah, i was just about to point that out21:58
fungi(the fact that /var/lib/zuul is on another device)21:59
clarkbcorvus: I like that idea, similar to nodepool clearing out transitional state entries in its db on start21:59
corvus(then 'reboot' would be an okay fix to problems like this)21:59
corvusclarkb: ya21:59
*** Apoorva has quit IRC21:59
*** spzala has quit IRC22:00
*** sree has joined #openstack-infra22:02
dmsimardwhat's the service name for zuul-executor ? systemctl list-unit-files isn't showing anything resembling zuul/executor22:04
dmsimardor is it upstart ?&22:04
*** yamamoto has joined #openstack-infra22:04
clarkbdmsimard: zuul-executor22:04
dmsimardfwiw ps aux | grep zuul isn't returning anything either22:04
clarkbits a sysv init script22:05
clarkbrun through systemd's compat laye22:05
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Configure all dvr nodes to dvr_snat agent mode  https://review.openstack.org/53150922:05
dmsimardah, that's prolly why list-unit-files isn't picking it up22:05
*** sree has quit IRC22:06
dmsimardok anyway, I was able to remount /var/lib/zuul rw after killing a few hung processed, cleaning /var/lib/zuul/builds and I'll reboot22:06
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron  https://review.openstack.org/43679822:06
clarkbnote depending on when the fs went ro the fixes to the init script may not have been applied yet22:07
clarkbso zuul-executor won't start on boot in that case, instead we need puppet to run then we can start it22:07
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Configure all dvr nodes to dvr_snat agent mode  https://review.openstack.org/53150922:07
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron  https://review.openstack.org/43679822:07
*** gouthamr has quit IRC22:07
*** bobh has quit IRC22:08
dmsimardeven remounting rw isn't working, dmesg complaining about filesystem errors.. going to run a fsck22:08
dmsimardwow that's a lot of errors :/22:09
*** Goneri has quit IRC22:10
dmsimardare the executors using volumes or ephemeral storage ?22:10
*** sree has joined #openstack-infra22:10
fungimgagne: looks like a partition on the ephemeral disk22:12
fungier, dmsimard ^22:12
fungisorry mgagne!22:12
*** gouthamr has joined #openstack-infra22:12
fungidmsimard: was /var/lib/zuul read-only too?22:13
dmsimardafter fsck: 4.7Glost+found/22:13
dmsimardfungi: yeah, it was in a pretty bad shape22:13
fungii can write to it on ze0922:13
dmsimardI just remounted it after finishing the fsck22:14
fungioh, you already checked and remounted22:14
fungik22:14
dmsimardmaybe we should consider using boot from volume, it helps especially for live migrations22:14
*** armaan_ has quit IRC22:14
fungidoes it in all cases?22:14
fungiagain, remember, this is xen22:14
dmsimardhmm, I don't know about xen -- but with a volume you're basically syncing the dirty page, disconnecting the volume from somewhere and connecting it elsewhere22:15
*** gouthamr has quit IRC22:15
fungii don't even know if rax has bfv support22:15
*** gouthamr has joined #openstack-infra22:15
dmsimardfor ephemeral (unless it's backed by centralized storage like ceph), you're doing a cold rsync of the backing disk22:16
dmsimardconsidering the executors have high i/o throughput, they're not very good candidates22:16
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies  https://review.openstack.org/53080622:16
clarkbthe executors are also ephemeral instances22:16
fungimakes sense not to use ephemeral for write-heavy use cases in that scenario, agreed22:16
clarkbso sticking them on long lived data safe volumes feels wrong22:16
*** gouthamr has quit IRC22:16
*** yamamoto has quit IRC22:17
*** bobh has joined #openstack-infra22:17
dmsimardit's not so much about the data being important than it is about making it easier to live migrate them22:17
clarkbdmsimard: right but there is no point in live migrating them at all22:17
clarkb(but no good way to communicate that to rax)22:17
dmsimardrax live migrates stuff when they do maintenance on compute nodes, it's routine22:17
fungistarting to wonder if that means putting swap on ephemeral disk might explain some of our migration failures/hangs too22:17
clarkbthe other issue with boot from volume is our volume quote is far more limited22:18
clarkb(because the idea is its for data you car eabout)22:18
clarkbwe would not be able to boot all our instances from volume with currentquota or any where close iirc22:18
fungiand also the minimum cinder volume size rax allows is 100gb22:18
dmsimardwell, anyway, something to keep at the back of our heads22:19
*** wolverineav has joined #openstack-infra22:19
dmsimardbuilds are clean, there's 4GB worth of stuff in lost+found, I doubt we have anything important to recover from there22:20
dmsimardrebooting22:20
clarkbdmsimard: we shouldn't since its just git repos and build data. The git repos are hosted elsewhere and builds we just rerun22:21
dmsimardhm, actually I suspect we'll need to do fsck in console -- do we have a root password ?22:21
clarkbdmsimard: we don't22:21
dmsimardok I'll set one22:21
dmsimardno wait I can't :D22:21
dmsimardoh well, here goes22:22
*** edmondsw has quit IRC22:22
fungiworst case we rebuild the instance22:22
dmsimardyeah22:22
fungithere's nothing of importance stored on it22:22
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron  https://review.openstack.org/43679822:22
*** sree has quit IRC22:22
*** edmondsw has joined #openstack-infra22:23
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Delete stale jobdirs at startup  https://review.openstack.org/53151022:23
*** bobh has quit IRC22:23
*** wolverineav has quit IRC22:24
clarkbubuntu has updated firefox for meltdown/spectre22:24
dmsimardew, straight to initramfs for ze0922:24
*** slaweq has quit IRC22:25
*** edmondsw has quit IRC22:26
*** smatzek has joined #openstack-infra22:27
*** sree has joined #openstack-infra22:27
fungiit's too bad i have to use firefox-esr to be able to continue using xul extensions22:28
fungiff57 no longer supports them at all22:28
fungihopefully they'll backport the protections to the esr branch22:29
*** bobh has joined #openstack-infra22:30
*** smatzek has quit IRC22:32
dmsimardze09's root partition is hosed22:34
dmsimardI'll check ze10 for the time being which is also out of commission22:34
*** bobh has quit IRC22:34
clarkbprobably best to just rebuild them then (assuming 10 is in similar state22:35
dmsimardyeah I can't ssh to ze10 so it doesn't look good22:35
clarkbif you can ssh to 10 then just try a reboot22:35
clarkbthats the typical fix for live migration woes22:35
*** spzala has joined #openstack-infra22:35
clarkb(and typically ssh doesn't work)22:35
dmsimardyeah it doesn't but it looks like ze10 might be recoverable, only see oom errors in console .. no filesystem errors22:36
dmsimardso I'll retry a reboot through the api22:37
*** slaweq has joined #openstack-infra22:39
dmsimardclarkb: procedure to rebuild a ze is similar to a logstash-worker ? :D i.e, build new node, change dns and that's pretty much it ?22:39
*** slaweq has quit IRC22:39
clarkbdmsimard: yes, I'm not sure of all the places where the firewall might need to be kicked to pick up the new node but otherwise yup22:40
dmsimardI think corvus fixed dns resolution22:40
dmsimardfor the firewall rules22:40
clarkbdmsimard: thats on boot, but places like the zookeeper server will need to be kicked22:40
clarkbI think nodepool.openstack.org and zuulv3.openstack.org are the two places with holes punched for zuul executors (zk and gearman respectively)22:41
*** hashar has quit IRC22:41
clarkbbut we can sort that out after the node is booting22:41
*** wolverineav has joined #openstack-infra22:41
openstackgerritIhar Hrachyshka proposed openstack-infra/devstack-gate master: Switch from lib/neutron-legacy to lib/neutron  https://review.openstack.org/43679822:42
*** e0ne has joined #openstack-infra22:42
*** bobh has joined #openstack-infra22:43
*** wolverin_ has joined #openstack-infra22:43
fungiwell, puppet is auto-resolving the hostnames to ip addresses and updating the rules with them at each pulse22:43
fungiso in theory it'll work as long as you can wait for puppet to do that?22:44
clarkboh right because its causing a hcange in the config file it will pulse it22:45
*** wolverin_ has quit IRC22:45
clarkbso ya ~1-45 minutes after dns updates it will update22:45
dmsimardze10 is back but zuul builds could not be cleaned in time :/22:45
*** wolverineav has quit IRC22:45
dmsimardserver otherwise looks healthy22:46
*** hongbin has quit IRC22:46
clarkbdmsimard: if you look at ls -l output you'll see the stale builds are quite stale and stand out based on timestamp22:46
dmsimardyeah that's what I had in mind22:47
clarkbwe should be able to delete anything older than today22:47
clarkboh also zuul isn't running22:47
clarkbso I don't think it raced?22:47
*** bobh has quit IRC22:47
*** wolverineav has joined #openstack-infra22:47
dmsimardI did a stop but not fast enough22:48
dmsimard:D22:48
clarkbah22:48
*** katkapilatova1 has joined #openstack-infra22:48
dmsimardthere's one build going afaict22:48
*** katkapilatova1 has left #openstack-infra22:48
dmsimardor 3, rather22:48
*** slaweq_ has quit IRC22:48
dmsimardthere's actually quite a few stale builds, going back as far as december 4th22:48
clarkbya22:48
dmsimardshould there be something to prune those ?22:48
*** bobh has joined #openstack-infra22:49
clarkbdmsimard: thats jeblairs change above22:49
clarkbhttps://review.openstack.org/53151022:49
*** sree has quit IRC22:49
dmsimardoh, neat22:49
*** wolverin_ has joined #openstack-infra22:50
*** jtomasek has joined #openstack-infra22:50
*** wolverineav has quit IRC22:52
clarkbcorvus: left a comment on ^22:53
*** bobh has quit IRC22:53
*** sree has joined #openstack-infra22:53
*** wolverin_ has quit IRC22:54
*** wolverineav has joined #openstack-infra22:55
*** wolverineav has quit IRC22:56
*** wolverineav has joined #openstack-infra22:56
dmsimardalso left another comment22:57
dmsimardneed to add a os.path.join22:57
*** wolverineav has quit IRC22:58
*** sree has quit IRC22:58
dmsimardok I started zuul-executor on ze10 after wiping old builds22:58
dmsimardgoing to relocate and start rebuilding ze0922:58
dmsimardclarkb: safe to assume I can already delete the existing instance ?22:58
clarkbdmsimard: I think so. corvus pabelanger fungi ^ can you think of any reason not to?22:59
*** wolverin_ has joined #openstack-infra22:59
*** bobh has joined #openstack-infra23:00
dmsimard#status log (dmsimard) ze10 was rebooted after being hung since january 4th23:00
openstackstatusdmsimard: finished logging23:00
*** sree has joined #openstack-infra23:02
*** bobh has quit IRC23:04
*** wolverin_ has quit IRC23:04
*** david-lyle has joined #openstack-infra23:04
*** wolverineav has joined #openstack-infra23:05
fungiyeah, i would delete ze09 and the existing dns records, then build another to replace it23:06
clarkbnote that puppet may fail if it runs when the old dns records are gone23:06
fungithough i suppose that could race puppet updating firewalls23:06
fungiyeah, that23:07
*** sree has quit IRC23:07
clarkb(probably not a huge issue as long as we get new records up soon and odn't forget)23:07
fungiso maybe just update dns23:07
fungibut probably fine either way23:07
clarkbfungi: care to review https://review.openstack.org/#/c/531249/ ?23:07
clarkbhoping that gets in before tonights periodic runs so I can double check that either over weekend or on monday23:07
*** wolverin_ has joined #openstack-infra23:08
*** wolverineav has quit IRC23:09
*** wolverineav has joined #openstack-infra23:10
*** wolverin_ has quit IRC23:10
fungisure23:10
*** sree has joined #openstack-infra23:11
dmsimardfungi, clarkb: is there logic involved in making /var/lib/zuul it's own partition ?23:11
corvusdmsimard: it's manual :(23:11
dmsimardI don't see any of that in puppet, it just creates the directory tree23:11
*** ltomasbo has quit IRC23:11
*** dalvarez has quit IRC23:11
dmsimardcorvus: ok so I shouldn't use launch-node because that'll get puppet to create the directory tree right away ?23:12
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Delete stale jobdirs at startup  https://review.openstack.org/53151023:12
dmsimardsorry, still kind of new to this :)23:12
corvusdmsimard: nah, use launch-node, but then rm or move it to the new partition23:13
dmsimardack23:13
corvusif you just rm it, puppet will fix on next run23:13
*** jpena|off has quit IRC23:14
*** wolverineav has quit IRC23:14
*** sree has quit IRC23:15
*** amoralej|off has quit IRC23:16
*** bobh has joined #openstack-infra23:18
*** bobh has quit IRC23:19
*** sree has joined #openstack-infra23:23
weshayanyone around? https://review.openstack.org/#/c/530983/ looks stuck in the gate23:24
weshayhttp://zuulv3.openstack.org/stream.html?uuid=4f3fec55ee9f4148a8360f6df56551da&logfile=console.log23:24
*** wolverineav has joined #openstack-infra23:26
*** wolverineav has quit IRC23:27
*** sree has quit IRC23:28
*** wolverineav has joined #openstack-infra23:28
*** Apoorva_ has quit IRC23:28
openstackgerritClark Boylan proposed openstack-infra/puppet-snmpd master: Enable snmpd so it works on systemd systems  https://review.openstack.org/53152023:29
clarkbinfra-root ^ noticed when poking at git farm performance on cacti (trying to see if there is a noticeable difference in system load and such)23:29
*** wolverin_ has joined #openstack-infra23:30
*** ltomasbo has joined #openstack-infra23:30
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Create artifacts dir regardless of afs rpm builds  https://review.openstack.org/53124923:30
clarkbweshay: I'm not sure its stuck. I think we've had prolems iwth the console log streaming not always working if the process has oomed23:31
clarkbweshay: is the blank console log the only data we have to assume its stuck?23:31
weshayya.. that's all I have23:31
corvusclarkb: want to drop ensure => running?23:31
clarkbcorvus: we can, in this case I didn't think it would hurt to have both23:31
*** wolveri__ has joined #openstack-infra23:31
*** sree has joined #openstack-infra23:32
corvusclarkb: probably not, so i +2d.  i'd just perfer to stomp out all instances.23:32
weshayclarkb, other jobs that should have started before this one, have completed the same job. that would be the other data point23:32
*** wolverineav has quit IRC23:32
dmsimardclarkb: I'm getting a permission denied error (causing launch-node to fail) when it tries to refresh the inventory but it works with sudo -- should I use sudo when using launch-node ? http://paste.openstack.org/show/639289/23:32
corvusdmsimard: are you in the right group?23:33
clarkbdmsimard: I think that means your user's group membership isn't correct. Ya check the launch node readme it has details on that23:33
corvusyeah, it's the pgraph at the top23:33
dmsimardhmm I was able to launch nodes before, let's see23:33
clarkbweshay: its possible the job had to be restarted for some reason. I can try and find where it is running really quick to at least track down the possible dead log streaming23:34
*** esberglu has quit IRC23:34
corvusdmsimard, clarkb: maybe the inventory file got remade with bad perms?23:34
*** wolverin_ has quit IRC23:34
corvusi can never remember where the cache is23:34
dmsimardI'm in admin afaict23:34
*** esberglu has joined #openstack-infra23:34
clarkbthat could be (and ya I have to reread the launch script to find it)23:34
corvus-rw-rw-r--  1 root admin 476869 Jan  5 23:30 ansible-inventory.cache23:35
*** gyee has quit IRC23:35
dmsimardcorvus: it complains about /var/cache/ansible-inventory*23:35
weshayclarkb, ok.. thank you.23:35
dmsimard /var/cache/ansible-inventory_openstackci-rax is kinda weird because apparently there's no such thing in /var/cache.. only /var/cache/ansible-inventory23:35
*** wolveri__ has quit IRC23:35
*** jtomasek has quit IRC23:36
corvusdid something change in os_client_config?23:37
corvusoh no, that's determined only by the inventory plugin23:38
corvus-rwxr-xr-x 1 root root 9771 Jan  2 16:16 /etc/ansible/hosts/openstack23:38
corvushow did that get updated?23:38
dmsimardJanuary 2nd..23:38
corvusit's puppeted -- copied in place from ansible23:39
*** esberglu has quit IRC23:39
corvusso it's a change to the openstack inventory file in ansible23:39
clarkbweshay: build is running from ze05.openstack.org and has processes running. If I log into the test instance it appears to be running quickstart23:39
*** slaweq has joined #openstack-infra23:39
dmsimardcorvus: we had a few changes to clouds.yaml due to OVH in and out23:39
weshayclarkb, ok.. thanks for checking :)23:39
dmsimardcan't think of anything else23:39
*** e0ne has quit IRC23:40
corvusthe commit that made that cache file format landed on august 10.23:41
corvusthe jan 2 change shouldn't matter23:42
clarkbisn't 664 root:admin correct? (I feel like I'm missing what we think changed in appropriately)23:42
clarkbShrews: what is process for debugging console streaming with the proxy in place? looks like nothing listens on port 79 anymore on the executors?23:43
corvusclarkb: see the paste.  it's trying to write to a cache dir that doesn't exist23:43
corvusclarkb: afaik, it's still on 79, but we will move it to an unprivileged port23:43
*** slaweq has quit IRC23:44
clarkbah ok ya ze01 has a :79 listener but ze05 does not23:44
corvuslikely oom killed23:44
clarkbcorvus: do we have to restart the zuul-executor daemon to correct that?23:44
corvusclarkb, dmsimard: here's the code regarding the cache: https://github.com/ansible/ansible/commit/d9d983160d9acc45b33a99ed7e570a9351eaf2c9#diff-c3e81a1e2b883ffdfff123da47a1c27dR19023:44
*** sree has quit IRC23:44
corvusclarkb: yes23:45
*** sree has joined #openstack-infra23:45
dmsimardcorvus: that landed in ansible 2.4, env is using 2.1..23:45
corvusdmsimard: we copy it from the git repo23:46
dmsimardoh23:46
clarkb[Fri Dec 22 13:54:01 2017] Killed process 2955 (zuul-executor) total-vm:653968kB, anon-rss:1476kB, file-rss:5344kB that is the cause I think23:47
clarkbweshay: ^ fyi23:47
clarkbits interesting how much smaller rss is but still enough to get it killed23:48
corvusclarkb: it's killing children of the largest process23:48
weshayk23:48
weshaythe poor children23:48
corvuswould be a good idea for apache, not so much for zuul23:48
dmsimardthe message is literally "sacrifice child" too.23:48
weshayif there is only zuul, the children must be sacrificed ;)23:49
corvusdmsimard: best i can figure, we somehow had an old version of the inventory plugin stuck on there, and now it's been updated23:49
dmsimardcorvus: yeah I was trying to see if there was something obvious in the puppet logs23:49
corvusdmsimard: i say run it as root for now... we'll probably need to update our cache directory creation in puppet to create those and set group ownership23:49
clarkbwell puppet wasn't working until we fixed ovh23:49
clarkbbut ovh wasn't broken tat long23:50
corvusclarkb: yeah, but that change was from many months ago23:50
*** ijw has quit IRC23:50
clarkbfwiw cacti doesn't show anything indicating pti is a major burden on the git servers23:53
*** caphrim007_ has joined #openstack-infra23:54
pabelangerfungi: dmsimard: re: eavesdrop01, no it sounds like there was an issue bringing the server online if no swap exists.  It was launched using launch-node.py, I guess we need to maybe add some logic to detect if swap is missing after we run puppet and fail so we don't launch servers without swap?23:54
*** caphrim007_ has quit IRC23:56
*** caphrim007 has quit IRC23:58
*** jascott1 has quit IRC23:59
*** sdague has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!