Monday, 2016-04-11

*** nelsnels_ has joined #openstack-infra00:06
*** Qiming has quit IRC00:07
*** nelsnelson has quit IRC00:07
*** _amrith_ is now known as amrith00:18
openstackgerritJamie Lennox proposed openstack/requirements: Allow maintaining extras from project
*** sflanigan has joined #openstack-infra00:22
*** chlong has quit IRC00:27
*** dchen has joined #openstack-infra00:28
*** ayoung has joined #openstack-infra00:32
*** dchen has quit IRC00:33
*** chlong has joined #openstack-infra00:34
openstackgerritMerged openstack-infra/subunit2sql: Remove DB existence check from open session termination
*** tiswanso has joined #openstack-infra00:45
*** gongysh has quit IRC00:47
*** tiswanso has quit IRC00:49
*** zz_dimtruck is now known as dimtruck00:50
*** mriedem has quit IRC00:50
*** nelsnelson has joined #openstack-infra00:51
*** nelsnels_ has quit IRC00:54
Nakatolifeless: Do you have any context on propose-reqs being disabled on stable branches?
*** Qiming has joined #openstack-infra01:06
*** julim has quit IRC01:09
*** julim has joined #openstack-infra01:11
*** tiswanso has joined #openstack-infra01:15
*** mylu has quit IRC01:15
lifelessNakato: yes; it pulls in new releases of openstack components01:15
lifelessNakato: and without an actual backwards compat policy, this is a problem01:16
lifelessNakato: all the non-openstack components tend to be more careful about backwards compat01:16
*** tiswanso has quit IRC01:16
lifelessNakato: but01:16
lifelessNakato: unless we state that e.g. mitaka libs should work to run a liberty cloud, its clearly wrong to do that01:16
*** tiswanso has joined #openstack-infra01:17
*** ramishra has joined #openstack-infra01:17
*** mylu has joined #openstack-infra01:17
Nakatolifeless: So, should we aim for a backwards compat policy, or aim to enhance that job to keep stable versions pinned?01:19
*** gongysh has joined #openstack-infra01:19
lifelessNakato: I have a backwards compat policy proposal drafted01:20
lifelessNakato: mriedm and I plan to go through it face to face in atlanta01:20
lifelesssince we're failing to get consensus via deuelling review comments01:20
lifelessNakato: in terms of stability, not modifying anything is actually fairly good for released code01:20
*** dimtruck is now known as zz_dimtruck01:23
*** krtaylor has quit IRC01:32
*** thorst has joined #openstack-infra01:35
*** amotoki has joined #openstack-infra01:38
Nakatolifeless: Thanks.01:39
*** thorst has quit IRC01:39
*** claudiub has quit IRC01:40
*** gildub has joined #openstack-infra01:41
lifelessNakato: collaborating on that discussion would be great01:44
*** achanda has joined #openstack-infra01:45
Nakatolifeless: Could you fling me the links?01:45
*** amotoki has quit IRC02:00
*** mylu has quit IRC02:01
*** Jeffrey4l_ has joined #openstack-infra02:02
*** thorst has joined #openstack-infra02:04
*** Jeffrey4l_ has quit IRC02:08
*** kengo_sakai has joined #openstack-infra02:10
*** amotoki has joined #openstack-infra02:10
*** kengo_sakai has quit IRC02:11
*** shu-mutou-AFK is now known as shu-mutou02:11
*** kengo_sakai has joined #openstack-infra02:11
*** nhandler has quit IRC02:14
*** tiswanso has quit IRC02:16
openstackgerritSachi King proposed openstack-infra/project-config: Update stable constraints url
*** ramishra has quit IRC02:21
*** ramishra has joined #openstack-infra02:21
*** Jeffrey4l_ has joined #openstack-infra02:22
*** sdake has joined #openstack-infra02:24
*** knikolla has quit IRC02:24
*** mylu has joined #openstack-infra02:24
*** thorst has quit IRC02:24
*** nhandler has joined #openstack-infra02:27
*** sdake_ has joined #openstack-infra02:27
*** sdake has quit IRC02:28
*** beisner has quit IRC02:30
*** beisner has joined #openstack-infra02:31
*** asettle has quit IRC02:33
*** kengo_sa_ has joined #openstack-infra02:34
*** kengo_sakai has quit IRC02:35
*** sflanigan has quit IRC02:37
*** sdake has joined #openstack-infra02:39
*** sdake_ has quit IRC02:43
*** salv-orlando has joined #openstack-infra02:45
*** rkukura has joined #openstack-infra02:45
*** yuanying has quit IRC02:46
*** akshai has quit IRC02:46
*** salv-orl_ has quit IRC02:47
*** FallenPegasus has joined #openstack-infra02:59
*** achanda has quit IRC02:59
*** mylu has quit IRC03:09
*** bkeller` has joined #openstack-infra03:10
*** Sukhdev has joined #openstack-infra03:10
*** mfisch has quit IRC03:11
*** mylu has joined #openstack-infra03:12
*** Guest89657 has joined #openstack-infra03:12
*** bkeller` has quit IRC03:12
*** yamahata has joined #openstack-infra03:13
*** Guest89657 has quit IRC03:13
*** Guest89657 has joined #openstack-infra03:13
*** Guest89657 is now known as mfisch03:14
*** rkukura has quit IRC03:14
*** sputnik13 has joined #openstack-infra03:19
lifelessfungi: could you comment on please03:21
*** zhurong has joined #openstack-infra03:25
*** harlowja_at_home has joined #openstack-infra03:32
ianwhmm, not sure if it's local, but git in my local image builds is hanging a lot03:33
*** mylu has quit IRC03:35
*** sputnik13 has quit IRC03:36
*** rkukura has joined #openstack-infra03:37
*** mylu has joined #openstack-infra03:37
*** rkukura has quit IRC03:41
*** abregman has quit IRC03:41
*** achanda has joined #openstack-infra03:44
*** zz_dimtruck is now known as dimtruck03:45
*** links has joined #openstack-infra03:45
*** achanda has quit IRC03:46
*** rkukura has joined #openstack-infra03:47
*** yuanying has joined #openstack-infra03:49
*** Nikolay_St has quit IRC03:57
*** FallenPegasus has quit IRC03:58
*** treaki_ has joined #openstack-infra03:59
prometheanfireianw: hi :P04:01
*** harlowja_at_home has quit IRC04:02
*** treaki has quit IRC04:02
*** jamesmcarthur has joined #openstack-infra04:04
*** jamesmcarthur has quit IRC04:08
*** oanson has joined #openstack-infra04:10
*** zhurong has quit IRC04:15
ianwok, maybe it was local04:16
*** achanda has joined #openstack-infra04:18
*** kengo_sakai_mba has joined #openstack-infra04:21
*** zhurong has joined #openstack-infra04:21
*** amotoki has quit IRC04:22
*** amotoki has joined #openstack-infra04:24
*** thorst has joined #openstack-infra04:25
*** thorst has quit IRC04:29
*** mylu has quit IRC04:30
*** kengo_sakai_mba has quit IRC04:31
*** mylu has joined #openstack-infra04:32
*** zhurong has quit IRC04:33
*** Nikolay_St has joined #openstack-infra04:38
*** gongysh has quit IRC04:38
openstackgerritMerged openstack-infra/tripleo-ci: Upload the ironic-python-agent images to cache
*** jaosorior has joined #openstack-infra04:46
*** kengo_sakai_mba has joined #openstack-infra04:58
*** sflanigan has joined #openstack-infra05:00
*** sflanigan has joined #openstack-infra05:00
*** ramishra has quit IRC05:04
*** ramishra has joined #openstack-infra05:04
*** vmbrasseur has left #openstack-infra05:05
*** abregman has joined #openstack-infra05:08
*** grue_pm has joined #openstack-infra05:15
*** grue_pm has quit IRC05:16
*** sdake_ has joined #openstack-infra05:23
*** sdake has quit IRC05:24
*** akaszuba has joined #openstack-infra05:27
*** asettle has joined #openstack-infra05:28
*** claudiub has joined #openstack-infra05:29
*** hdd has joined #openstack-infra05:31
*** Nikolay_St has quit IRC05:36
*** mylu has quit IRC05:36
*** kushal has joined #openstack-infra05:39
*** pcaruana has joined #openstack-infra05:39
*** gongysh has joined #openstack-infra05:40
*** dimsum__ has quit IRC05:40
*** achanda has quit IRC05:43
*** sdake_ is now known as sdake05:45
*** dims has joined #openstack-infra05:47
*** kengo_sakai_mba has quit IRC05:49
openstackgerritMerged openstack-infra/shade: Use event_utils.poll_for_events for stack polling
*** dimtruck is now known as zz_dimtruck05:54
openstackgerritMerged openstack-infra/shade: Wrap stack operations in a heat_exceptions
*** tricksters has joined #openstack-infra05:56
openstackgerritMerged openstack-infra/shade: Delete image objects after failed upload
openstackgerritMerged openstack-infra/shade: Add release note about the swift Large Object changes
*** eric_lopez has quit IRC05:58
*** oanson has quit IRC06:01
*** zeih has joined #openstack-infra06:03
*** tphummel has joined #openstack-infra06:11
*** hrubi has joined #openstack-infra06:12
openstackgerritMerged openstack/diskimage-builder: Turn down tracing for source-repo cache
*** chlong has quit IRC06:13
*** achanda has joined #openstack-infra06:15
*** oanson has joined #openstack-infra06:15
*** rcernin has joined #openstack-infra06:15
*** achanda has quit IRC06:20
yolandagood morning06:22
openstackgerritSteve Kowalik proposed openstack-infra/system-config: Switch to new release of puppet-mysql
*** abregman has quit IRC06:28
AJaegermorning, yolanda !06:29
*** abregman has joined #openstack-infra06:30
*** scheuran has joined #openstack-infra06:30
*** dims has quit IRC06:31
*** tesseract has joined #openstack-infra06:35
*** tphummel has quit IRC06:35
*** tesseract is now known as Guest898906:35
*** roxanaghe has joined #openstack-infra06:37
openstackgerritRyota MIBU proposed openstack-infra/project-config: [ceilometer] enable ceilo tempest plugin tests
*** jtomasek has joined #openstack-infra06:39
*** roxanaghe has quit IRC06:41
*** achanda has joined #openstack-infra06:44
*** fedexo has quit IRC06:44
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Create Merger Git repos using connection names
*** Hal has joined #openstack-infra06:45
*** Hal is now known as Guest1961406:46
*** gildub has quit IRC06:48
*** sputnik13 has joined #openstack-infra06:49
*** sputnik13 has joined #openstack-infra06:49
*** daemontool has joined #openstack-infra06:53
*** dims has joined #openstack-infra06:54
*** esikachev has joined #openstack-infra06:55
*** kengo_sakai_mba has joined #openstack-infra06:56
*** sputnik13 has quit IRC06:56
*** kushal has quit IRC06:56
*** dims has quit IRC06:58
*** arxcruz has joined #openstack-infra07:01
*** markvoelker has joined #openstack-infra07:01
*** ihrachys has joined #openstack-infra07:03
*** jaosorior has quit IRC07:03
*** jaosorior has joined #openstack-infra07:04
*** kengo_sakai_mba has quit IRC07:04
*** markvoelker has quit IRC07:06
*** dtardivel has joined #openstack-infra07:07
*** shashank_hegde has quit IRC07:08
*** mikelk has joined #openstack-infra07:11
*** jamielennox is now known as jamielennox|away07:15
*** vincentll has joined #openstack-infra07:17
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Fix get_test_metadata to use test_id
*** jed56 has joined #openstack-infra07:21
*** fhubik has joined #openstack-infra07:24
*** placko_ has joined #openstack-infra07:26
*** Sukhdev has quit IRC07:28
*** chlong has joined #openstack-infra07:31
*** nmagnezi has joined #openstack-infra07:31
*** fhubik has quit IRC07:34
*** shardy has joined #openstack-infra07:34
*** jlanoux has joined #openstack-infra07:35
yolandaAJaeger,thx for the change to increase heap size on gerrit, i missed the conversation yesterday07:41
*** dims has joined #openstack-infra07:45
*** kushal has joined #openstack-infra07:45
*** jaosorior has quit IRC07:47
*** jaosorior has joined #openstack-infra07:48
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Fix get_test_metadata to use test_id
*** jaosorior has joined #openstack-infra07:49
*** kairat has joined #openstack-infra07:50
*** achanda has quit IRC07:56
kairatAJaeger, Hi, do you know if reno cache release notes somewhere?07:57
*** pblaho has joined #openstack-infra07:57
*** chem has joined #openstack-infra07:57
*** e0ne has joined #openstack-infra08:02
AJaegerkairat: what do you mean with cache?08:04
AJaegeryolanda: you're welcome! Just was briefly online when that discussion happened...08:04
kairatI am trying to understand why note broke translation but it seems that changes in release note doesn't affect tests08:04
kairatso i am trying fix the test08:04
AJaegerkairat: the glance_store problem?08:04
kairatbut when I execute tox -e releasenotes again08:04
kairatit somehow uses old release note definition08:05
kairatYou can try to reproduce it on your local env08:05
kairatjust change the release note that breaks the gate08:06
AJaegerthanks for digging into this. Sphinx does some caching, so thta might be involved here.08:06
kairatand you will see that error is at the same place08:06
AJaegerBut in the gates we always start with clean workspace, so there is no caching involved08:06
AJaegerI reproduced it locally - that's why I send the email.08:06
kairatI am trying to find the root cause08:07
kairatSo i am interesting in cleaning that cache08:07
AJaegergive me a sec, I need to double check something08:08
kairatPerhaps, you know how to do that cause I am not familiar with sphinx08:08
openstackgerritRyota MIBU proposed openstack-infra/project-config: [telemetry] enable ceilo/aodh tempest plugin tests
*** asettle has quit IRC08:09
AJaegerkairat:  releasenotes/build/.doctrees is the cache, just rm -rf it08:09
*** yamahata has quit IRC08:10
*** BobBall_AWOL is now known as BobBall08:11
AJaegerkairat: to me it looks like the mitaka releasenotes are broken - and I wonder how that could have passed the gates. You might need to discuss with dhellmann how to avoid this in the future...08:11
*** BobBall is now known as Guest8010008:11
kairatAJaeger, I tried that already08:11
kairatWith no effect08:12
kairatI also cleaned releasenotes env in .tox08:12
AJaegerkairat: for me it fails everytime reruning tox - no need to do anything08:12
*** jordanP has joined #openstack-infra08:12
kairatCan you update release not08:13
kairatwill error message be changed after that?08:13
kairatfor example you can insert several additional strings08:13
AJaegerYes, you should be able to update the release notes but since reno parses the git history I'm not sure whether testing will get this really. Best talk to dhellmann ...08:15
kairatOk, thanks08:16
kairatActuallym that was my change08:16
kairatso I am interesting to fix that ASAP08:16
kairatAnd understand what was the trouble08:16
AJaegerAh ;)08:17
AJaegerkairat: I'm interested mainly to learn why the gates did not complain earlier - why it was possible to merge this at all...08:18
*** dingyichen has quit IRC08:22
*** achanda has joined #openstack-infra08:22
*** Guest80100 is now known as BobBall^08:22
*** yamamoto has joined #openstack-infra08:24
*** roxanaghe has joined #openstack-infra08:25
*** vgridnev has joined #openstack-infra08:25
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
*** bauwser is now known as bauzas08:29
*** roxanaghe has quit IRC08:29
AJaegerkairat: Ah, I see - glance_store has no releasenotes jobs defined at all. Let me add them now...08:31
openstackgerritMerged openstack/diskimage-builder: Debian: dont set always the hostname to debian
*** jistr has joined #openstack-infra08:32
*** vikram_ has joined #openstack-infra08:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add release note publishing to glance_store
vikram_can someone plz help in approving
AJaegerkairat, nikhil_k, flaper87: See 303962, please08:33
*** derekh has joined #openstack-infra08:33
AJaegervikram_: please describe your change so that the right people can review it.08:34
vikram_AJaeger: The change is about creating a new project for Huawei's ML2 plugin code for interacting with neutron08:35
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
AJaegervikram_: LGTM, approved08:37
vikram_AJaeger: Thanks ;)08:37
*** jlanoux has quit IRC08:38
vikram_AJaeger: One query.. when the new repo will be up08:38
*** jlanoux has joined #openstack-infra08:38
AJaegervikram_: this is all driven via puppet, expect it to be up 1h after the change is merged08:39
*** ramishra has quit IRC08:39
vikram_AJaeger: Ok08:39
AJaegervikram_: and then ask here for an admin to add you to ACLs - note I'm not an admin and cannot do it.08:39
vikram_AJaeger: Ok.. After the project is created will do that..08:40
AJaegerkairat: since there're no gate jobs for releasenotes yet, it's no wonder that this could go in as is.08:40
AJaegerkairat: But that also means you can easily fix it - and then we enable the gate jobs ;)08:40
*** mrmartin has joined #openstack-infra08:43
*** salv-orl_ has joined #openstack-infra08:44
*** salv-orlando has quit IRC08:47
*** dims has quit IRC08:48
openstackgerritLenny Verkhovsky proposed openstack-infra/project-config-example: First commit. README update
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors
openstackgerritLenny Verkhovsky proposed openstack-infra/project-config-example: First commit. README update
*** skolekonov|away is now known as skolekonov08:55
*** dims has joined #openstack-infra08:55
*** pcaruana has quit IRC09:00
openstackgerritThomas Herve proposed openstack-dev/pbr: Fix wsgiref script use with oslo.config
*** zhurong has joined #openstack-infra09:02
*** markvoelker has joined #openstack-infra09:02
*** nadya has joined #openstack-infra09:02
*** rossella_s has quit IRC09:03
openstackgerritMerged openstack-infra/project-config: Adding networking-huawei project.
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
*** rossella_s has joined #openstack-infra09:03
*** ociuhandu has quit IRC09:04
*** abregman has quit IRC09:05
*** markvoelker has quit IRC09:07
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors
*** sambetts|afk is now known as sambetts09:12
openstackgerritMateusz Matuszkowiak proposed openstack-infra/puppet-nodepool: Added nodepool::mysql class
*** ramishra has joined #openstack-infra09:16
*** ramishra has quit IRC09:16
*** abregman has joined #openstack-infra09:18
*** zhurong_ has joined #openstack-infra09:18
*** Nikolay_St has joined #openstack-infra09:21
*** ramishra has joined #openstack-infra09:21
*** zhurong has quit IRC09:22
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
openstackgerritGiulio Fidente proposed openstack-infra/project-config: Add os-{refresh,collect,net}-config projects
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
*** briancline has quit IRC09:40
*** briancline has joined #openstack-infra09:42
openstackgerritGiulio Fidente proposed openstack/requirements: Bump os-net-config to 0.2.4
*** ociuhandu has joined #openstack-infra09:43
esikachevyolanda: hi! take a look on what did you think?09:43
yolandaah, as I saw the -1 from Khai i didn't review sorry. Taking a look now09:44
esikachevok, thanks09:46
*** gfidente has joined #openstack-infra09:46
*** ociuhandu has quit IRC09:47
gfidentehi, do you guys know the nick for Andreas Jaeger ?09:48
*** kairat has left #openstack-infra09:48
gfidenteping AJaeger09:49
AJaegergfidente: You guessed correctly ;)09:52
AJaegergfidente: how can I help?09:52
gfidenteAJaeger sorry but it was my first attempt09:53
gfidentethanks for reviewing09:53
gfidenteso I think what I wanted is to map os-net-config to its separate ACL file which I am adding09:53
gfidenteand to update the mappings for a couple of those projects, let me try another update09:54
AJaegergfidente: *why* do you want to do this?09:55
gfidentebecause we wanted to split up permissions for os-net-config09:55
*** tosky has joined #openstack-infra09:55
AJaegergfidente: so, have a separate group for os-net-config? Ok, then add that information to the commit message, please09:56
gfidenteI think initially though it'll be tripleo-core anyway09:56
BobBall^So it seems that libvirt-python now depends on having libvirt installed in order to install itself as of a few days ago... libvirt-python is in global requirements, but possibly should not be since it is not requried for non-libvirt environments.  How do we normally handle this?09:57
AJaegergfidente: all of you three new files have the same content.09:57
gfidenteAJaeger yeah I removed those for os-collect and os-refresh09:57
AJaegerThat's something to avoid, just add it once and reference it.09:57
AJaegergfidente: ok09:57
gfidenteAJaeger yeah thanks09:57
AJaegergfidente: looking forward to the next iteration ;)09:57
gfidentealso I noticed os-collect-config is missing the reference to os-apply-config.config ?09:58
AJaegergfidente: indeed, that looks wrong ;(09:58
openstackgerritMerged openstack/requirements: Add Windows as supported operating system
openstackgerritGiulio Fidente proposed openstack-infra/project-config: Add ACLs for os-net-config
gfidenteAJaeger thanks for helping, hope it looks better now10:01
AJaegergfidente: gerrit/acls/openstack/tripleo.config is the same as your file, isn't it?10:01
AJaegergfidente: Please don't duplicate ;)10:01
*** kengo_sakai_mba has joined #openstack-infra10:02
gfidenteAJaeger yes it is but that's the purpose of the split10:02
gfidentethough until we have a split to do it's not worth it10:02
gfidenteso I'll step back on that for now10:02
AJaegergfidente: You can either create a new group - or reference the acl file...10:02
AJaegergfidente: but send the fix for os-collect-config, please10:03
*** cdent has joined #openstack-infra10:03
*** thorst has joined #openstack-infra10:04
*** ociuhandu has joined #openstack-infra10:06
openstackgerritGiulio Fidente proposed openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config
*** roxanaghe has joined #openstack-infra10:13
*** kengo_sakai_mba has quit IRC10:14
*** ams_ has joined #openstack-infra10:16
*** roxanaghe has quit IRC10:18
*** Guest19614 has quit IRC10:19
*** ramishra has quit IRC10:19
*** kengo_sakai_mba has joined #openstack-infra10:20
*** Qiming has quit IRC10:20
*** ramishra has joined #openstack-infra10:24
*** briancline has quit IRC10:25
*** briancline has joined #openstack-infra10:25
openstackgerritAlexey Stepanov proposed openstack-infra/project-config: fuel-qa: Add vote to pylint checks
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in for cleanup_pingtest method
*** thorst has quit IRC10:27
*** thorst has joined #openstack-infra10:27
*** briancline has quit IRC10:30
*** briancline has joined #openstack-infra10:31
*** sdague has joined #openstack-infra10:31
*** thorst has quit IRC10:32
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in for cleanup_pingtest method
*** penguinolog has joined #openstack-infra10:36
penguinologHello! Review, plz:
*** vikram_ has quit IRC10:45
*** sdake_ has joined #openstack-infra10:45
*** thorst has joined #openstack-infra10:46
*** jcoufal has joined #openstack-infra10:48
*** sdake has quit IRC10:48
*** kushal has quit IRC10:50
*** ifarkas has joined #openstack-infra10:50
openstackgerritVictor Stinner proposed openstack-infra/project-config: Make gate-trove-python34-db voting; add gate job
*** jaypipes has joined #openstack-infra10:57
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
*** markvoelker has joined #openstack-infra11:03
*** maishsk has joined #openstack-infra11:03
*** zhurong_ has quit IRC11:04
*** thorst has quit IRC11:04
*** thorst has joined #openstack-infra11:05
*** bkeller` has joined #openstack-infra11:05
*** mat128_ is now known as mat12811:06
*** markvoelker has quit IRC11:07
*** nadya has quit IRC11:09
*** thorst has quit IRC11:09
*** ramishra has quit IRC11:16
*** tpsilva has joined #openstack-infra11:18
*** kushal has joined #openstack-infra11:18
*** esikachev has quit IRC11:20
*** vtapia_ has quit IRC11:20
*** vtapia_ has joined #openstack-infra11:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository
*** bkeller` has quit IRC11:22
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
*** Goneri has joined #openstack-infra11:24
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors
*** ramishra has joined #openstack-infra11:27
Zarafungi: a while ago you mentioned that we had a limited time to improve the storyboard email setup before it stopped working; I can't remember the details (I think to do with how we were handling bounces?) do you have a moment to go through it with me again? worried since bugsquash is coming up...11:28
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
*** esikachev has joined #openstack-infra11:31
*** weshay has quit IRC11:36
*** ldnunes has joined #openstack-infra11:37
*** cdent has quit IRC11:40
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax
*** gordc has joined #openstack-infra11:42
*** Qiming has joined #openstack-infra11:43
*** binbincong has quit IRC11:43
*** ramishra has quit IRC11:43
*** yamamoto has quit IRC11:46
*** vtapia_ has quit IRC11:46
*** nadya has joined #openstack-infra11:47
*** esikachev has quit IRC11:47
*** vtapia_ has joined #openstack-infra11:48
*** camunoz_ has joined #openstack-infra11:49
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax
*** baoli_ has joined #openstack-infra11:50
*** flaper87 has quit IRC11:51
*** flaper87 has joined #openstack-infra11:51
*** sshnaidm has quit IRC11:51
*** flepied1 has joined #openstack-infra11:51
*** baoli has quit IRC11:53
*** esikachev has joined #openstack-infra11:53
*** flepied has quit IRC11:54
*** binbincong has joined #openstack-infra11:56
*** sshnaidm has joined #openstack-infra11:57
openstackgerritAparna proposed openstack/diskimage-builder: Support to add certificate in ironic-agent
*** amotoki has quit IRC11:58
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test_unset_flavor_specs to use betamax
*** sdake_ is now known as sdake11:59
*** roxanaghe has joined #openstack-infra12:01
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Update deprecated commands in (Old heat * calls)
*** thorst has joined #openstack-infra12:04
*** icey has joined #openstack-infra12:04
*** xiangxinyong has quit IRC12:04
*** eliqiao has quit IRC12:05
AJaegerodyssey4me: are you around for your retirement change?12:06
*** eliqiao has joined #openstack-infra12:06
*** roxanaghe has quit IRC12:06
*** xiangxinyong has joined #openstack-infra12:06
*** icey has quit IRC12:06
*** esikachev has quit IRC12:07
*** cdent has joined #openstack-infra12:07
AJaegerodyssey4me: commented on review12:07
*** aysyd has joined #openstack-infra12:09
*** amotoki has joined #openstack-infra12:09
*** gongysh has quit IRC12:09
*** sdake_ has joined #openstack-infra12:10
openstackgerritAparna proposed openstack/diskimage-builder: Support to add certificate in ironic-agent
*** icey has joined #openstack-infra12:11
*** icey has quit IRC12:11
*** sdake has quit IRC12:12
*** rodrigods has quit IRC12:13
*** aysyd has quit IRC12:13
*** rodrigods has joined #openstack-infra12:13
*** aysyd has joined #openstack-infra12:14
*** icey has joined #openstack-infra12:16
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add release note publishing to glance_store
*** pradk has joined #openstack-infra12:17
*** sdake_ has quit IRC12:18
*** achanda has quit IRC12:18
*** markvoelker has joined #openstack-infra12:19
*** dprince has joined #openstack-infra12:21
*** yamamoto has joined #openstack-infra12:22
*** dtantsur is now known as dtantsur|mtg12:25
*** weshay has joined #openstack-infra12:25
*** edmondsw has joined #openstack-infra12:26
*** camunoz_ has quit IRC12:26
*** amotoki has quit IRC12:27
*** rlandy has joined #openstack-infra12:30
*** Hal has joined #openstack-infra12:30
*** Hal is now known as Guest555912:30
*** zumbi has joined #openstack-infra12:32
odyssey4meAJaeger I saw the thing about making it read-only, but if I do that it'll apply to all of our repositories I think. Is there a way to do that for only one of the repositories as they all share the same ACL file?12:34
*** achanda has joined #openstack-infra12:35
*** achanda has quit IRC12:35
*** zhurong has joined #openstack-infra12:35
AJaegerodyssey4me: sure.;)12:36
*** zhurong has quit IRC12:36
AJaegerodyssey4me: let me check your setup12:36
*** kengo_sakai_mba has quit IRC12:36
*** salv-orl_ has quit IRC12:36
AJaegerodyssey4me: you have in gerrit/projects.yaml "acl-config: /home/gerrit2/acls/openstack/openstack-ansible.config"12:36
*** zhurong has joined #openstack-infra12:37
*** zhurong has quit IRC12:37
AJaegerodyssey4me: create a new acl file gerrit/acls/openstack/openstack-ansible_swift_sync.config with the read only content from the ifnra manual and remove the acl-config line above.12:37
odyssey4meah ok, that makes sense - let me revise that patch now with that advice - thanks!12:38
*** burgerk has joined #openstack-infra12:38
*** eharney has quit IRC12:39
AJaegerodyssey4me: also, do you have a governance change? Please reference it12:39
*** BobBall^ is now known as BobBall12:39
odyssey4mesure, will do12:40
*** kengo_sa_ has quit IRC12:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository
AJaegerodyssey4me: no need to add information about defaults, just remove the acl-config line...12:42
odyssey4meah ok, so the default acl file is the repo name?12:43
*** amrith is now known as _amrith_12:44
*** achanda has joined #openstack-infra12:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-os_swift_sync repository
*** gongysh has joined #openstack-infra12:46
mrmartinthe groups.o.o build seems to be broken, because the bundle package is missing, and compass requires that to compile the sass files. What is the proper way to add the missing package? Am I need to add it as a bindep dependency?12:46
AJaegerodyssey4me: yes12:46
odyssey4menifty, and sensible :)12:46
AJaegermrmartin: you could add an other-requirements.txt file that lists all the requirements your repo has.12:47
AJaegermrmartin: we don't want to enhance bindep-fallback in general12:47
AJaegermrmartin: package name is bundle?12:47
AJaegermorning, pabelanger. Could you read the last 6 mins of backscroll and help mrmartin, please?12:48
pabelangerAJaeger: Sure, mrmartin do you mind posting a log file12:48
mrmartinAJaeger, pabelanger : yeap, the package name is "bundler"
mrmartinthis one12:48
mrmartinand the log:
pabelangermrmartin: Oh, okay. I would suggest doing what puppet jobs do and sandbox your bundler install12:50
AJaegermrmartin: the log file says "SUCCESS" - that looks strange12:50
krotscheckIs the openstack-infra namespace still something I should use for infra-specific projects, or should I use the openstack namespace?12:50
mrmartinyeah, because bash not stopped with a proper exit code12:50
pabelangermrmartin: I can get a patch for JJB to do it12:50
*** akshai has joined #openstack-infra12:51
pabelangeractually, where is scripts/ located?12:51
pabelangershould be either to add logic to add bundler there12:51
mrmartinpabelanger: ok we need to add this one only: gem install bundler --no-rdoc --no-ri --verbose12:51
pabelangermrmartin: oh, it is a path issue in your script12:52
pabelanger1 sec for patch12:53
openstackgerritPaul Belanger proposed openstack-infra/groups: Use path name for bundle
pabelangermrmartin: that should help12:54
pabelangerlets see what the gate does12:54
pabelanger1 sec12:55
*** xyang1 has joined #openstack-infra12:55
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] disable releasenotes jobs for liberty and kilo branch
openstackgerritPaul Belanger proposed openstack-infra/groups: Install bundler with gem
vgridnevfolks please take a look on the ; it should unlock our gate for stable/liberty12:57
pabelangerokay, that should be better12:57
*** jsavak has joined #openstack-infra12:58
pabelangerlooks like jenkins02 is down again. 3 mondays in a row, I guess something is wrong with our ansible playbook12:58
pabelangerI'll fix it up shortly12:58
AJaegerpabelanger: it's time to move jenkins02 to a new server...13:01
AJaegerpabelanger: thanks for noticing and fixing it!13:01
AJaegervgridnev: looking13:02
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to opentack infra
pabelangerAJaeger: Ya, clarkb mentioned that last week too. Going to bring it up at tomorrows meetings and see if people are on board with upgrade13:03
pabelangermrmartin: mat128: Okay, should be fixed with 30410413:04
ZZelleHi everyone!13:04
AJaegerpabelanger: I think all are on board - it was just not high enough on anybody's priority list yet13:04
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
krotscheckThere we go13:04
pabelangerAJaeger: Ya, I also think we have a quota issue too13:04
pabelangerokay, starting on jenkins0213:05
AJaegerkrotscheck: that change needs a governance change as well...13:05
openstackgerritMerged openstack-infra/groups: Install bundler with gem
*** maishsk has quit IRC13:07
*** amotoki has joined #openstack-infra13:08
krotscheckAJaeger: yep, workin on it.13:09
*** kzaitsev_mb has joined #openstack-infra13:09
AJaegerkrotscheck: once you have it, add Needed-By to your commit message, please13:09
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Refactor releasenote branch condition
krotscheckNeeded by? Is that new?13:10
ZZelleThe redirection from review.o.o to launchpad for oauth fails, am i the only one impacted?13:11
*** maishsk has joined #openstack-infra13:11
pabelangerokay, jenkins02 back online13:12
AJaegerkrotscheck: introduced together with depends-on - and documented in Infra Manual governance change section
*** trown|outtypewww is now known as trown13:12
krotscheckAJaeger: Odd. In the past you've told me to put depends-on in the governance patch, not needed-by in the project_config patch13:12
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] configure jobs for sahara-tests
AJaegerkrotscheck: you should do *both*13:13
*** zhurong has joined #openstack-infra13:14
AJaegerneeded-by is the backwards pointer to depends-on13:14
AJaegerkrotscheck: the depends-on is essential, the needed-by very good practice ;)13:14
mrmartinpabelanger: looks better now, thnx:
krotscheckThis seems like an awful lot of red tape to jump through.13:14
fungiZara: it was more of a concern that in the very distant future, the /var/spool/mail/storyboard mbox file might get so full of bounces and unanticipated replies that it consumes all of the available space on the root filesystem. since those messages are small bits of text and we have 34gb available space on the server, it's not especially urgent and can probably wait (even years) until storyboard has13:14
fungiinbound e-mail processing13:15
pabelangermrmartin: np13:15
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests
*** _ari_ has joined #openstack-infra13:16
AJaegerkrotscheck: I need to governance review to +2 your change. So, putting it in the commit message is the best place for it.13:17
Zarafungi: ahhh, okay. :D are there any circumstances under which it would fill up faster? eg: if messages could contain pictures, etc? (not that we have plans to do this, but that way my brain will flag it if it comes up in the future)13:17
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Temporarily disable mistral on the undercloud
fungimrmartin: pabelanger: it does appear that we `gem install bundler` in our puppet jobs... is bundler a deployment requirement for the groups webapp too, or just used in testing it?13:17
mrmartinfungi: it is using to compile SaSS to CSS, so we are using it for packaging actually13:18
mrmartinit is / we are13:19
penguinologreview please:
pabelangerfungi: mrmartin: Ya, for the most part, puppet jobs will just sandbox bundler, since they have boostrap script in the gate.  Looks like groups maybe have just overlooked it from the start13:20
pabelangerfungi: moving to ubuntu-trusty exposed the missing dependency13:20
AJaegerfungi, needs infra PTL+1, please put it on your queue - this is krotscheck's change13:20
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: clening up python3 jobs for sahara-tests
*** links has quit IRC13:21
*** piet has joined #openstack-infra13:21
*** tiswanso has joined #openstack-infra13:21
AJaegerpenguinolog: if that's critical for you, please explain what it is and why you urgently need it...13:21
krotscheckAJaeger Except it's broken.13:21
krotscheckJust a minute13:21
AJaegerkrotscheck: do you mind capitalizing OpenStack in gerrit/projects.yaml then as well?13:22
fungipabelanger: AJaeger: clarkb thought that might fix it. merged yesterday at 07:14 utc, about 7 hours after the playbook ran13:23
penguinologAJaeger, it's not urgent, but now we have >1 week positive status on this check and after voting in team we selected to enable voting13:23
krotscheckAJaeger- That's why it's broken13:23
krotscheckSilly infra, whyfore do you explicitly reference an executable by path? /bin/mktemp: No such file or directory13:24
*** resker has quit IRC13:25
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
*** flip214 has quit IRC13:25
fungiZara: i suppose if storyboard started sending very large messages with image attachments and people's mtas were bouncing them with all attachments intact that could conceivably fill up a little quicker, but not enough i'd be worried about it making a huge difference13:25
pabelangerfungi: clarkb: maybe we should default master executors to 0 with a grovvy script:
pabelangervs offlining them13:26
*** flip214 has joined #openstack-infra13:26
*** pkoniszewski has joined #openstack-infra13:26
Zarafungi: cool, I'll file it under 'not gonna worry about this' (I'd like to get inbound email processing working for other reasons, but yeah). thank you. :)13:27
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Use the new launch client in zuul server
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Don't launch jobs in threads
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Use the seeded repos when launching a job
anteayaI think for the longest time (I think this is a recent change) we had a word for the people who work in this channel and we had a separate word for the tooling we support13:27
*** mtanino has joined #openstack-infra13:27
AJaegermorning, anteaya !13:27
anteayahaving two separate words enabled folks to respect and support the people while communicating frustration and disappointment with the tooling13:28
anteayaAJaeger: morning13:28
anteayathat has changed13:28
anteayaand now folks use one word (not sure why)13:28
anteayato express frustration and disapointment13:28
anteayawhich now extends to the people13:28
anteayawhich, I have to say, becomes tiring fast13:28
* AJaeger agrees with anteaya ...13:29
*** krtaylor has joined #openstack-infra13:29
anteayaAJaeger: thanks, glad you have seen it happen as well13:29
AJaegeranteaya: doyou want to advise also to people what to use instead of just "infra" for both?13:31
*** oanson has quit IRC13:31
*** Guest5559 has quit IRC13:31
* mordred calls himself mordred13:31
anteayaAJaeger: no13:32
*** eharney has joined #openstack-infra13:32
anteayaI'm carrying the standard of using the word folks people and group, as well as friends and collegues13:32
anteayaI can't take this on as well13:32
anteayamostly I'm saying that word choice makes a big difference in group health13:32
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required
anteayaand I'm observing this13:32
anteayamordred: morning mordred13:33
* AJaeger welcomes mordred called mordred13:33
mordredyah. I've got floating ips, and "openstack is not the place for cloud native workloads" - so I'm also a bit full on conceptual flags13:33
mordredmorning anteaya and AJaeger !13:33
AJaegeranteaya: I agree, word choice is important13:34
openstackgerrityolanda.robla proposed openstack-infra/shade: Add test for delete flavor exception with betamax
anteayaAJaeger: but in this instance I don't have a suggestion for a solution13:34
*** e0ne has quit IRC13:34
trownhola infra folks, I do not see anything in the project creator's guide about creating groups in gerrit, could someone point me in the right direction?13:34
fungitrown: gerrit groups are automatically created if gerrit acls refer to them13:34
trownfungi: ah, cool, who gets edit permissions for group membership?13:35
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
*** mylu has joined #openstack-infra13:35
AJaegertrown: if the content is not clear, please send a patch to enlighten the next one reading it...13:35
trownAJaeger: will do :)13:35
*** mriedem has joined #openstack-infra13:35
trownin general it is very good13:35
AJaegertrown: did yo uread ?13:36
fungitrown: normally they're self-managed, and (upon request) a gerrit admin will add the gerrit account of the contributor who submitted the acl addition as the initial member13:36
openstackgerritAlexis Lee proposed openstack-infra/shade: Add test for delete flavor exception with betamax
*** thorst has quit IRC13:36
fungitrown: or the gerrit account of the ptl in the event that the acl is for repos under an official team in openstack governance13:36
trownAJaeger: that is precisely what I was looking for thanks!13:37
*** salv-orlando has joined #openstack-infra13:37
AJaegerfungi, thanks for approving the translation script change yesterday! No need for any rerun of post jobs...13:37
AJaegerfungi, New Babel release had broken handling of multiple keywords on command line ;(13:38
AJaegerupstream patch to fix it exists, so I hope to be able to revert my change soon...13:38
*** e0ne has joined #openstack-infra13:39
*** amitgandhinz has joined #openstack-infra13:40
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NPM mirror hosting to unified mirrors
mordredderekh: morning!13:40
openstackgerritMerged openstack-infra/project-config: fuel-qa: Add vote to pylint checks
mordredderekh: so - when you get a sec, I encountered an issue with the tripleo cloud over the weekend. we have a workaround for it, but I thought you should know13:40
derekhmordred: howdy, whats up?13:40
*** jtomasek has quit IRC13:41
mordredderekh: basically, if you create a floating ip via the neutron api handing it the port id of the server in question - the cloud happily makes and attaches the floating ip to the server but seems to not inform nova about it13:41
mordredderekh: I've verified that the same calls work on the rax/intel osic, on dreamhost and on our bluebox cloud (which all require floating ips) so I _think_ it's something wonky on the tripleo cloud13:42
krotscheckI think I've got all the npm things ready.13:42
*** tonytan4ever has joined #openstack-infra13:42
anteayamorning yolanda, I'm taking a look at the etherpad for the gerrit migration today:
mordredderekh: but like I said - not a big deal, we have a config flag we can use to change our behavior to use the nova proxy api - mostly wanted to heads-up you13:42
yolandahi anteaya13:42
mordredkrotscheck: zomg. that's super awesome13:43
*** salv-orlando has quit IRC13:43
anteayayolanda: did you have an opportunity to read the suggestions jeblair left on it?13:43
mriedemare the openstack irc channel topics stored in an infra repo somewhere?13:43
yolandayes, i took a look today13:43
anteayayolanda: the line above step 1 and the line below step 113:43
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests
krotscheckI don't suppose I can get someone with an AFS keytab to go in and clear out the /mirror/npm directory? It's got some old things in it.13:44
AJaegermriedem: not currentlyy13:44
krotscheckmordred ^^13:44
yolandayep, i actually wanted to check with infra-roots today about that. As we use trove, wanted to check what's the better way for creating backups13:44
anteayayolanda: and I was wondering if you wanted to paste in the commands you expect to run for each step13:44
anteayayolanda: like we do with the rename workflow13:44
mordredkrotscheck: sure. one sec13:44
anteayayolanda: great13:44
derekhmordred: ok, are the calls your making in nodepool somewhere and I'll see if I can reproduce? Also we are hoping to redeploy that cloud shortly after the summit so if its not easily fixable the new cloud should do the right thing13:45
AJaegermriedem: but would be a neat idea, jhesketh was considering it last week as well when he restarted the bot and had to resurrect all channel messages - the bot had changed the topics temporraily and didn't know to set it back ;(13:45
mriedemAJaeger: yeah i noticed the nova channel topic changed13:45
yolandaabout the dns, i also have the question if we change it just when we start the maintenance window, or after the updates have been done13:45
fungiyolanda: i'd do it at the start of the maintenance window (think i said that in the meeting as well)13:46
anteayayolanda: can you add your questions to the etherpad?13:46
fungithat way it gets more time to propagate13:46
AJaegermriedem: jhesketh grepped the topic from eavesdrop and set it back again13:46
yolandai added my question to the ehterpad13:46
jheskethmriedem: the statusbot stores it in memory13:46
anteayayolanda: that way when you have answers everyone can see13:46
anteayayolanda: thanks13:46
jheskethmriedem: topics also land in the logs which is where I extracted all the ones from to restore them13:46
yolandaok, updated the document with dns change13:47
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required
jhesketha tool to pull them from logs would be useful, or otherwise a data store of sorts for the bot to manage13:47
anteayajhesketh: hey there!13:47
fungijhesketh: yep, that's how we've restored it in the past if statusbot gets restarted in the middle of an alert condition13:47
yolandafungi, also how about the backups? how do we do it on trove?13:47
anteayayolanda: thanks13:47
*** kgiusti has joined #openstack-infra13:47
jheskethI did it by hand because I figured it might be easier than writing a bot, but I think given the time to do it I may have been wrong13:47
jheskethanteaya: howdy13:47
fungiyolanda: there's a daily cronjob in root's crontab (our puppet mysql backup module creates it)13:47
yolandaso we already have the backups?13:47
mordredkrotscheck: you said you want all of the things in mirror/npm deleted?13:47
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required
yolandaor do we launch one just before?13:47
fungiyolanda: might be a good idea to do one more just before we start13:48
fungiyolanda: but can simply reuse the syntax from the crontab13:48
*** jtomasek has joined #openstack-infra13:48
yolandaok i will add it13:48
*** ddieterly has joined #openstack-infra13:49
*** jtomasek has quit IRC13:50
*** achanda has quit IRC13:50
*** jtomasek has joined #openstack-infra13:50
yolandaanteaya, i'm adding a bit more of details to the etherpad, although i cannot create an step by step instruction. But will be enough to get a clear idea13:51
*** mtanino has quit IRC13:51
krotscheckmordred: yep.13:51
krotscheckmordred: And while you're in there, can you delete my access rights?13:51
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Correct error message when domain is required
openstackgerritMerged openstack-infra/project-config: Update regex for tempest-dsvm-ironic-inspector job
AJaegerdhellmann: what's the best way to fix broken releasenotes? See - does it need a backport to mitaka like in ?13:53
openstackgerritJohn Trowbridge proposed openstack-infra/project-config: Create tripleo-quickstart subteam
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix filtering of automatic worklist contents
*** _amrith_ is now known as amrith13:54
mordredkrotscheck: done13:54
krotscheckmordred: Awesome, thanks13:54
yolandafungi, can you double check the etherpad again, to ensure that all steps are valid?
amrithyolanda, trove does backups for you ...13:55
*** esker has joined #openstack-infra13:56
*** esker has quit IRC13:56
*** esker has joined #openstack-infra13:57
yolandaamrith, thanks. I got the commands to do a local backup before the maintenance13:57
*** nmagnezi has quit IRC13:57
*** avarner has joined #openstack-infra13:57
*** achanda has joined #openstack-infra13:58
*** xarses has quit IRC13:58
openstackgerritMonty Taylor proposed openstack-infra/shade: Consume config values for NAT destination
openstackgerritMonty Taylor proposed openstack-infra/shade: Search subnets for gateway_ip to discover NAT dest
openstackgerritMonty Taylor proposed openstack-infra/shade: WIP Start stamping the has_service debug messages
openstackgerritMonty Taylor proposed openstack-infra/shade: Refactor guts of _find_interesting_networks
openstackgerritMonty Taylor proposed openstack-infra/shade: Refactor the port search logic
openstackgerritMonty Taylor proposed openstack-infra/shade: Add nat_destination filter to floating IP creation
openstackgerritMonty Taylor proposed openstack-infra/shade: Allow passing nat_destination to get_active_server
AJaegeramrith: what's up with trove-dashboard? The translation imports do not pass your tests: and . Would be great to merge these, please13:58
openstackgerritMerged openstack-infra/shade: Make delete_project to call get_project
*** rkukura has quit IRC13:59
*** doug-fish has joined #openstack-infra14:00
*** jsavak has quit IRC14:01
*** e0ne has quit IRC14:01
*** jsavak has joined #openstack-infra14:01
*** woodster_ has joined #openstack-infra14:03
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add simple script to help debug shade image uploads
*** e0ne has joined #openstack-infra14:04
*** yamamoto has quit IRC14:06
*** achanda has quit IRC14:07
*** nadya has quit IRC14:07
*** burgerk has quit IRC14:07
*** bkeller` has joined #openstack-infra14:08
openstackgerritJohn Trowbridge proposed openstack-infra/project-config: Create tripleo-quickstart subteam
*** yamamoto has joined #openstack-infra14:10
*** jtomasek has quit IRC14:11
openstackgerrityolanda.robla proposed openstack-infra/shade: Update delete flavor test to work with betamax
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test for deleting non-existing flavors
*** vgridnev has quit IRC14:14
mrmartinpabelanger: I'm checking the last puppet run on groups-dev.o.o and it was at 9:08:
*** roxanaghe has joined #openstack-infra14:18
pabelangermrmartin: Hmm, looks like ansible is blocked again14:18
*** dizquierdo has joined #openstack-infra14:18
jeblairgood morning14:19
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test for setting flavor specs to use betamax
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project
pabelangerHmm, looks to be hung on the past facts task since 07:00 UTC14:20
jheskethjeblair: I'm about to head off, but if you have time this week I think we should go over a few of the open zuul changes and mark some for consideration to merge before v3.. there are a few that we may want in v3 and some we may just want sooner etc14:20
*** zz_dimtruck is now known as dimtruck14:20
*** ajmiller has joined #openstack-infra14:20
*** rvasilets has left #openstack-infra14:20
mrmartinpabelanger: does it affect groups.o.o only, or everything else?14:21
jheskethjeblair: I /think/ this one should help with the memory leak
openstackgerrityolanda.robla proposed openstack-infra/shade: Update test_unset_flavor_specs to use betamax
pabelangermrmartin: everything14:21
*** ddieterly is now known as ddieterly[away]14:21
jheskethjeblair, fungi: which, btw, it looks like zuul might run out of memory any minute now :-s (
jeblairjhesketh: okay -- it sounds like you've started -- you want to send me a list of ones you think i should look at?14:22
*** roxanaghe has quit IRC14:22
pabelangermrmartin: okay, looks to be unblocked, should start running again in 8mins14:22
jheskethjeblair: yeah I went over the zuul reviews yesterday but didn't mark any particulars down... I'll do that tomorrow :-)14:22
*** zhurong has quit IRC14:22
jeblairjhesketh, fungi: that suggests that the foreign project change is probably not responsible for the leak -- we've been running with that manually reverted since the last restart14:22
mrmartinpabelanger: thnx again14:23
fungijhesketh: jeblair: okay, so the foreign project for pipeline revert we have running on zuul.o.o right now doesn't seem to have helped the leak i guess? does that place the ball more squarely in the connections change court?14:23
fungier, what jeblair just said14:23
jheskethjeblair: oh I didn't know that was applied.. interesting14:23
*** ddieterly[away] is now known as ddieterly14:23
jheskethfungi, jeblair: I wonder if it's worth doing the same experiment with 28116614:23
*** jsavak has quit IRC14:24
jeblairjhesketh: i think we can experiment with it -- but i don't want to land that patch because it is a feature regression; it's definitely intended that everything should be reloaded14:24
*** jsavak has joined #openstack-infra14:24
pabelangerI wonder if we should pass --timeout=SECONDS to the  ansible-playbook command to help improving our connection issues14:24
jheskethjeblair: well the .conf was reloaded previously, but the connection to gerrit (for example) was never re-established14:25
*** vgridnev has joined #openstack-infra14:25
jheskethso updated credentials were never picked up14:25
jheskethand I think it owuld have been the same for smtp connection (since the class is spawned at the start)14:25
jeblairjhesketh: ok.  we may need to handle connections separately (we don't want to needlessly restart them) -- but i also don't want to lose updates to the logging config, or log urls, or other options.14:26
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project
*** avarner has quit IRC14:27
jheskethjeblair: hmm, I wonder if just not stopping and registering connections is enough...14:28
*** tongli has joined #openstack-infra14:28
jeblairjhesketh: maybe so?14:28
pabelanger#status log ansible-playbook blocked since 0700 UTC, manually killed ansible-playbook process to unblock puppet_run_all.lock14:28
jheskethyeah I'm thinking so... I'm just taking a look14:28
*** avarner has joined #openstack-infra14:28
openstackgerrityolanda.robla proposed openstack-infra/shade: Add new test with betamax for create flavors
openstackgerrityolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks
mordredyolanda: we still waiting on betamax release?14:29
*** mtanino has joined #openstack-infra14:29
yolandamordred yes, i have pinged sigmavirus24, but i see him offline since 7th april14:29
jeblairyolanda: where's the gerrit upgrade etherpad?14:30
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Only reload layout on HUP
jheskethjeblair: so nothing obvious in terms of using the connections from the config... so hopefully ^ is safe14:31
jhesketh(note, untested)14:31
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project
jeblairjhesketh: cool; we can try restarting with that next time14:31
jheskethjeblair: sounds good... I wonder if it's worth preempting the memory running out and doing it in a quiet period (eg tomorrow night)14:32
jheskethassuming it hasn't ran out in the mean time14:32
*** regXboi has joined #openstack-infra14:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs
*** sc68cal has joined #openstack-infra14:33
jheskethjeblair: or maybe during the gerrit switchover14:33
jeblairjhesketh: that would be a good one14:33
AJaegerthe change above will fail until is in - reviews for both are welcome, please14:33
*** rkukura has joined #openstack-infra14:33
AJaegerproject-config cores, 304164 checks for all problems I noticed in reviews today - and a few more ;)14:34
*** jamesmcarthur has joined #openstack-infra14:35
*** yamahata has joined #openstack-infra14:36
jeblairAJaeger: somehow 304164 conflicts with 30398414:38
AJaegerjeblair, interesting...14:38
*** burgerk has joined #openstack-infra14:38
AJaegerI'll rebase/recheck once 303984 is in14:38
*** knikolla has joined #openstack-infra14:38
jeblairgit does not think so, but gerrit does14:39
*** greghaynes has quit IRC14:39
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
AJaegerjeblair, jhesketh : We currently have nearly 6 MB of log files for zuul/layout.yaml, is there a good way to "compress" it? I'm speaking here about the project-config job. What about removing DEBUG values?14:39
*** haypo has joined #openstack-infra14:40
AJaegerjeblair: ok, let me rebase now...14:40
*** tpeoplesvacation is now known as tpeoples14:40
haypohi. "recheck" asks to run again all checks, but is it possible to only ask to retry a single job? on my job gate-grenade-dsvm-neutron failed on apt-get: "Cannot initiate the connection to"14:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs
anteayahaypo: no14:42
anteayahaypo: all jobs run on purpose14:42
*** jaosorior is now known as jaosorior_away14:42
anteayaas rechecking jobs individually will allow patches that have races to be merged14:42
*** vikram_ has joined #openstack-infra14:42
anteayacheck your patch, you might be triggering or introducing a nondeterministic failure14:43
anteayajust rechecking until your patch merges introduces races in the code14:43
haypoanteaya: what?14:43
jeblairkrotscheck, fungi: if we felt like bikeshedding -- afs-blob-store isn't actually afs-specific, is it?  it could as well be 'posix-blob-store' or 'fs-blob-store', yeah?  having said that, i don't mind having more things in the world that say afs, so i'm just going to +2 304108 unless one of you are like 'aha!'14:43
anteayaand makes the code base worse for everyone14:43
anteayahaypo: what part of my reply did you not understand?14:43
haypoanteaya: apt-get failed. my patch only changes the neutron code, not the code to create/setup the VM14:44
*** maishsk has quit IRC14:44
anteayahaypo: right, I'm giving you the reason we have all the jobs run every time14:44
anteayahaypo: you asked can you trigger one job at a time, I said no and told you why not14:45
haypoanteaya: my question is specific to random failures which are unrelated to my change, when i checked and i know that it's really unrelated to my change and "random"14:45
krotscheckjeblair: Well, fs-blob-store is an interface in the node world.14:45
krotscheckso xxx-blob-store is the usual naming convention for thing-that-implements-blob-store-for-thing14:45
anteayahaypo: right, we have no provision for checking on job for 'random' failures14:46
vikram_Hi Admin14:46
*** jtomasek has joined #openstack-infra14:46
jeblairkrotscheck: oh, 'fs-blob-store' is the one that implements an unhierarchical blob store on the fs?14:46
krotscheckjeblair: Yep.14:46
vikram_Can someone please add me to ACL's for the newly created project
krotscheckjeblair: If you look at the implementation that I did, the afs thing actually sits on top of fs-blob-store14:46
jeblairkrotscheck: then i guess i'm left with suggesting 'uninsane-blob-store' so 'afs-blob-store' is probably the better name.14:46
haypoanteaya: last weeks, i started to check each time why a test failed, and it's very common that a job fails before running tests, it fails to setup the VM :-/14:46
krotscheckAnd, well, it's not afs specific, no, but it's at that odd intersection of a-little-too-precise-use-case.14:47
vikram_fungi: Can you plz help me out14:47
*** rbrndt has joined #openstack-infra14:47
anteayahaypo: did you file a bug?14:47
anteayahaypo: or investigate bugs for that failure?14:47
vikram_fungi: Adding me to ACL's for the newly created project
krotscheckjeblair: Right, but if you consider what the name says to the rest of the world, one could assume that "Oh, this is a thing that'll run vos-release automatically"14:47
krotscheckWhich it doesn't.14:47
rcarrillocruzmordred: i was chatting the other day with Shrews about (yeah, revisiting designate again :D ). We agreed to add the designate plugins on all the dsvm jobs, rather than adding a new specific designate-dsvm job14:48
jeblairkrotscheck: yeah, but leaves the door open.14:48
rcarrillocruzall good?14:48
mordredkrotscheck: so - fwiw - it looks to me like maybe you didn't git add in your repo14:48
mordredkrotscheck: all I see is tests and code to handle arguments14:48
rcarrillocruzme thinks we may eventually want to have a non-core services dsvm job14:48
*** placko_ has quit IRC14:48
rcarrillocruzand leave neutron/nova/keystone to just test that14:48
mordredbut it's also probably my js foo is not as awesome as I'd like and I'm missing something14:48
jeblairmordred: javascript is just that amazing.14:48
haypoanteaya: oh... i'm not sure that a bug report would help... today i noticed two failures (1) gpg failed to retrieve a key. i tried manually: it works. it looks like a network issue or timeout issue (2) apt-get failed to download Releases.gpg -- network issue or issue with the package server14:48
jeblairmordred: you *only* have to write tests.14:48
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs
rcarrillocruzwhere we put designate, heat, $new_service14:49
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config
mordredjeblair: dude. we should totally write that framework14:49
haypoanteaya: for (1), the resource is external, it's the public GPG server hkp://keys.gnupg.net14:49
openstackgerrityolanda.robla proposed openstack-infra/shade: WIP: Add shade call to get flavor access list
*** GheRivero_vaca is now known as GheRivero14:49
mordredjeblair: write the tests, and we'll infer and generate the code that satisfies the tests14:49
*** piet has quit IRC14:49
AJaegerjhesketh, jeblair: Thanks for review, fixed the issues and removed basing on unrelated change14:49
haypoanteaya: the (2), i don't know who handles and it expects a reply "issue already fixed" or "network issue, retry"14:49
haypoanteaya: for* (2)14:49
anteayahaypo: okay well if you think a bug report won't help, then me thinking it will won't result in you filing a bug report14:50
haypoanteaya: i will open a bug if i start to see regulary the same failure multiple times14:50
anteayahaypo: okay14:50
haypoanteaya: sadly, the last errors i saw were all unique!14:50
haypo(and very different)14:50
vikram_AJaeger: The project is created now.. Whom I should contact now14:51
*** Jeffrey4l_ has quit IRC14:51
*** yamamoto has quit IRC14:51
Zaraclarkb: saw your feature-request in the logs, have put it in this story:!/story/2000545 ; not sure when I'll get to improving subscriptions but I'd like to look at it at some point.14:51
krotscheckjeblair: Anything since 'uninsane-blob-store'? I had a bouncer hiccup14:51
jheskethAJaeger: I think that'll fail Jenkins sorry14:51
jhesketh(see comment)14:51
vikram_AJaeger: Can you plz show me the way :)14:52
mordredkrotscheck: yes ...14:52
AJaegerjhesketh: I thought I fixed id ;( Will push again..14:52
jeblairkrotscheck: from me: general agreement with your forward looking statements about potentially adding more afs to afs-blob-store14:52
AJaegervikram_: ask here for "infra-root" help, and reference the commit14:52
mordredkrotscheck: I see nothing in your project other than tests and args.js ... maybe missing git add ?14:52
*** greghaynes has joined #openstack-infra14:52
krotscheckHrm. IRC job is failing.14:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add checks for gerrit ACLs
AJaegerjhesketh: better?14:52
vikram_AJaeger: Ok14:53
AJaegerkrotscheck: fungi fixed it for you, didn't he?14:53
krotscheckmordred: index.json14:53
krotscheckmordred: index.js14:53
jheskethAJaeger: lgtm, thanks14:53
mordredkrotscheck: ah - in the root and not in the lib dir14:53
AJaegerthanks, jhesketh !14:53
vikram_Need infra-root for . Can someone plz help14:54
krotscheckAJaeger: Oh, cool. Didn't see that14:54
fungikrotscheck: yeah, gate-project-config-irc-access looks like it's passing (according to zuul status) since my tweak, but it hasn't reported yet because gate-project-config-compare-xml is still running14:54
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for phantomjs / horizon lint job bug 1568325
openstackbug 1568325 in OpenStack Dashboard (Horizon) "Phantom is causing JS tests to fail constantly" [High,In progress] - Assigned to Rob Cresswell (robcresswell)14:54
*** esikachev has joined #openstack-infra14:54
krotscheckfungi: cool, thanks for stepping in :)14:55
krotscheckrobcresswell: For ^^, what solution are y'all going after?14:56
fungikrotscheck: any last-minute alterations/additions you needed t make to that repo before i approve it for import?14:57
krotscheckfungi: Nope. I know there's some work to be done, i.e. add license headers, but I can do those afterwards.14:58
krotscheckUnless mordred, AJaeger, or jeblair have comments.14:58
mordredI think it's awesome14:58
mordredof course, I was in favor of it the instant it had afs in the name14:58
fungiafs is the reviewer trump card14:58
fungi[approved because: afs]14:59
*** jtomasek has quit IRC14:59
krotscheckmordred: So, if I created an app called afs-rm-dash-r-star, you'd approve it?14:59
*** cdent has quit IRC14:59
mordredkrotscheck: totally. and run it on my laptop14:59
krotscheck....ooookay. I'll be right back.14:59
mordredkrotscheck: of course, it didn't say afs-rm-dash-r-star-then-vos-release ... so it's safe ;)15:00
krotscheckmordred: Well, we can address that issue in v215:00
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit initial json
*** bkeller` has quit IRC15:01
*** fawadkhaliq has joined #openstack-infra15:01
fungivikram_: done15:01
mordredkrotscheck: I'd say we could talk about design for it at #cubaconference - but it conflicts with the summit15:01
*** bkeller` has joined #openstack-infra15:02
vikram_fungi : Thanks15:02
*** amitgandhinz has quit IRC15:02
krotscheckmordred: whaaa.... apparently that's a thing.15:02
*** amitgandhinz has joined #openstack-infra15:02
krotscheckmordred: Have I told you recently about my conversation with the phantomjs people vis-a-vis having a mirror more reliable than bitbucket?15:03
krotscheckmordred: Their response was: We have a mirror, here it is *points at china*15:03
*** salv-orlando has joined #openstack-infra15:03
*** salv-orlando has quit IRC15:03
*** jtomasek has joined #openstack-infra15:04
*** salv-orlando has joined #openstack-infra15:04
robcresswellkrotscheck: Using firefox rather than phantom for now. Its only used for the UTs, so we can remove it from the dependencies quite easily.15:04
mordredkrotscheck: wow15:05
krotscheckrobcresswell: Alrightey.15:05
mordredkrotscheck: so, is our npm mirror about to become the world's best npm mirror?15:05
krotscheckrobcresswell: I'll keep an eye on it to see if anything ever progresses.15:05
*** apoorvad has joined #openstack-infra15:05
krotscheckmordred: I dunno about that.15:05
krotscheckmordred: But it's about to become the world's only public npm mirror.15:05
mordredoh. well15:06
mordredif it's te world's only public npm mirror15:06
krotschecknpm-inc asked all the other mirrors to shut down because CDN15:06
mordredthen I think we win 'best' by default :)15:06
openstackgerritMerged openstack-infra/project-config: Add missing acl-config reference for os-collect-config to os-apply-config.config
mordredthe old "cdn solves everything" talke15:06
krotscheckWhich, of course, really hasn't impacted their reliability at all.15:06
krotscheckPlus or minus15:06
AJaegerjeblair: could you review again, please? I updated based on jhesketh's review...15:06
*** abregman has quit IRC15:08
krotscheckmordred: Speaking of which, go bleed on this with a red pen ->
openstackgerritMerged openstack-infra/openstackid-resources: Summit initial json
*** jsavak has quit IRC15:09
* krotscheck is switching internets. His home network is terribad right now.15:10
*** jsavak has joined #openstack-infra15:10
fungithe internets are all terribad15:10
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** sputnik13 has joined #openstack-infra15:11
fungithese are not the bits you're looking for15:11
mordredkrotscheck: registry-static errors on incomplete mirroring runs I'm assuming?15:11
openstackgerritMerged openstack-infra/project-config: Add js-afs-blob-store (afs-blob-store) to openstack-infra
vincentllwhen I attempt to set my contact information I get a "Server Error - Cannot store contact information". Is it a known error?15:12
*** hdd has quit IRC15:13
*** xarses has joined #openstack-infra15:13
*** apoorvad has quit IRC15:14
*** yamahata has quit IRC15:15
*** arxcruz has quit IRC15:15
*** Jeffrey4l_ has joined #openstack-infra15:15
*** achanda has joined #openstack-infra15:15
*** sputnik13 has quit IRC15:16
openstackgerritPaul Belanger proposed openstack-infra/devstack-gate: Test commit
*** akaszuba has quit IRC15:17
*** piet has joined #openstack-infra15:17
*** openstackgerrit has quit IRC15:18
*** openstackgerrit has joined #openstack-infra15:18
jheskethjeblair: decided to create a list for v2 now:
*** fawadkhaliq has quit IRC15:18
jhesketh(clearly not urgent, but please put your thoughts in there when you can)15:19
*** hdd has joined #openstack-infra15:19
jheskethjeblair: I've been leaving zuulv2 patches for you to merge.. but maybe we should come up with a rule as to what is okay (if anything) to merge into master while we're working on v315:19
jheskethjeblair: and on the topic of zuul, I've put up a couple of small changes to the v3 branch fyi15:19
*** thorst has joined #openstack-infra15:20
jeblairjhesketh: cool, thanks!15:20
*** fawadkhaliq has joined #openstack-infra15:21
*** yamamoto has joined #openstack-infra15:21
anteayavincentll: yes, look at
*** e0ne has quit IRC15:21
anteayavincentll: and see if it helps you15:21
amrithAJaeger, I've logged it in LP and have pinged the folks working on dashboard to take a look15:22
openstackLaunchpad bug 1568876 in OpenStack DBaaS (Trove) "trove-dashboard python27 gate job fails" [High,New]15:22
fungijpmaxman: jamesmcarthur: i don't see smarcet in here at the moment, but i approved just now. please take note of the conditions mentioned in my approval comment (i'll remind smarcet the next time i see him in here as well)15:22
vincentllanteaya: thanks i'll have a look15:22
*** e0ne has joined #openstack-infra15:23
anteayavincentll: may I ask what documentation you were following to set up your account?15:23
*** matt-borland has joined #openstack-infra15:23
*** haypo has left #openstack-infra15:23
*** zeih has quit IRC15:23
*** njohnston has joined #openstack-infra15:24
vincentllanteaya: I was actually trying to do my first "git review" on a new stackforge project, the prompt asked me first to accept the Contributor Agreement, and then the contact information15:24
anteayavincentll: ah, so no documentation, might I suggest some?15:24
vincentllanteaya: yes please15:24
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Don't reload connections on HUP
anteayavincentll: read everything about setting up accounts very closely15:25
anteayathere are no extraneous steps in that documenation15:25
anteayavincentll: and welcome to openstack15:25
*** thorst has quit IRC15:25
vincentllanteaya: thanks!15:26
*** daemontool_ has joined #openstack-infra15:26
openstackgerritVictor Stinner proposed openstack-infra/project-config: Make gate-trove-python34-db voting; add gate job
anteayavincentll: welcome15:26
*** daemontool_ has quit IRC15:26
*** daemontool_ has joined #openstack-infra15:27
*** yamamoto has quit IRC15:27
*** daemontool has quit IRC15:28
vincentlloh, one more question, when importing my project into the newly created stackforge project, is it ok to keep the git history from my project? Or should I squash all the commits into one (as the git review prompt suggested)?15:28
anteayamorning clarkb15:28
*** hdd has quit IRC15:28
anteayavincentll: what do you mean importing your project into the newly created stackforge project?15:29
anteayavincentll: which stackforge project and which is your project?15:29
anteayawe have a step where a project is imported when a project is created on our gerrit15:29
anteayadid you know about the project import step during project creation?15:29
*** mylu has quit IRC15:30
clarkbis the gerrit lucene index on the volume device(s)?15:30
clarkbwondering ifwe will need to reindex15:30
anteayaclarkb: that is a good question15:31
*** exploreshaifali has joined #openstack-infra15:31
anteayaI don't think anyone has asked that yet15:31
mordredclarkb: hrm. I have no idea where that index is stored :)15:32
mordredShrews, infra-root: woot! is green and ready for review15:32
*** ajmiller has quit IRC15:33
clarkbit is in review_site/index so will move with the volumes15:33
vincentllanteaya: yes, I did know about the import step, but at that time our git repo was not ready to be imported (the stackforge project is
anteayaclarkb: are you purple on the etherpad?15:33
*** lakshmiS has joined #openstack-infra15:34
clarkbanteaya: yes15:34
anteayaclarkb: wonderful15:34
openstackgerritMerged openstack-infra/devstack-gate: Archive /etc/openstack/clouds.yaml in gate runs
fungivincentll: you can propose all your commits as individual changes for review and approve them one by one, or you can squash them all into one change. it's really up to you and your contributors/reviewers preference-wise15:34
openstackgerritMerged openstack-infra/project-config: Add checks for gerrit ACLs
openstackgerritMerged openstack-infra/system-config: Add user smarcet to openstackid production server
vincentllfungi: ok, thanks for the clarification15:35
*** Guest8989 has quit IRC15:35
krotscheckmordred: I believe so. I will admit that I haven't run a full mirror, and therefore don't know if there's any annoying edge cases in the guts of the nm registry15:35
mordredkrotscheck: kk15:36
mordredkrotscheck: we'll learn about them15:36
jheskethokay, I'm off... good luck with the gerrit switchover, sorry I'm not around to help15:36
mordredjhesketh: woot! enjoy your night!15:36
anteayajhesketh: enjoy sleep!15:36
krotscheckmordred: I've got the personal email of the maintainer over at yahoo, so I can ask him for help if we run into anything.15:36
jpmaxmanCool fungi thank you. Comments noted.15:36
*** kengo_sakai_mba has joined #openstack-infra15:36
*** jsavak has quit IRC15:38
*** rbrndt__ has joined #openstack-infra15:38
*** harlowja_at_home has joined #openstack-infra15:38
*** mylu has joined #openstack-infra15:38
*** rbrndt has quit IRC15:39
fungiyolanda: i added reverse dns entries on the replacement gerrit server just now. that would have caused a lot of e-mail to not get delivered15:40
fungibut this is far enough in advance the negative cache ttl should expire15:41
anteayafungi: yolanda how long will it take to create the db backup do you think?15:41
clarkbanteaya: about 5 minutes15:41
anteayaawesome, thank you15:41
krotscheckinfra-core: js-afs-blob-store just merged and was replicated, could I get someone to update the ACL's to add infra-core as approvers, and add myself to the js-afs-blob-store +2 group?15:41
*** Swami has joined #openstack-infra15:41
*** kengo_sakai_mba has quit IRC15:41
*** yamamoto has joined #openstack-infra15:42
fungiyolanda: any reason not to do step 4 before the maintenance (like preferably 30 minutes to an hour before)?15:42
*** mylu has quit IRC15:43
yolandafungi yes, can be done before15:44
*** chem has quit IRC15:44
*** dtantsur|mtg is now known as dtantsur|afk15:44
*** gyee has joined #openstack-infra15:45
yolandai will connect an hour before the maintenance to start with prev steps15:45
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Return boolean from delete_project
*** Qiming has quit IRC15:45
fungiyolanda: or one of us can do it so you don't need to be around that soon. just mostly concerned we'll end up waiting for an in-progress puppet run15:46
yolandafungi, i'm reading about the volume changes, how do you have normally done in the past? using some script for that, or just using the ui?15:46
*** vincentll has quit IRC15:46
*** chem has joined #openstack-infra15:46
*** bpokorny has joined #openstack-infra15:46
fungiyolanda: usually use nova volume detach/attach15:46
fungifrom the puppetmaster15:47
*** fawadkhaliq has quit IRC15:47
*** jaosorior_away has quit IRC15:47
fungiyolanda: there are some examples at
*** ddieterly is now known as ddieterly[away]15:48
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] configure publist-to-pypi jobs for sahara-tests
*** jaosorior_away has joined #openstack-infra15:48
*** mylu has joined #openstack-infra15:48
*** ifarkas has quit IRC15:49
*** jlanoux has quit IRC15:49
*** mrmartin has quit IRC15:49
yolandafungi thx15:49
fungiyolanda: in this case, we'll also need to do some lvm setup on the new server i think, since this isn't just a straight filesystem on a cinder block device but rather an logical volume. though vgscan will _probably) pick it up once attached i think15:49
openstackgerritMerged openstack-infra/elastic-recheck: Add query for phantomjs / horizon lint job bug 1568325
openstackbug 1568325 in OpenStack Dashboard (Horizon) "Phantom is causing JS tests to fail constantly" [High,In progress] - Assigned to Rob Cresswell (robcresswell)15:50
yolandaok going to add some of these bits to the etherpad15:50
*** ddieterly[away] is now known as ddieterly15:51
*** Sukhdev has joined #openstack-infra15:51
*** bpokorny has quit IRC15:51
fungialso we had an old gerrit homedir (lv name was "gerritold") mounted at /mnt but i'm in the process of getting rid of that now so it doesn't confuse us later today15:52
*** yamamoto has quit IRC15:52
*** bpokorny has joined #openstack-infra15:52
*** dtroyer_zz has quit IRC15:52
clarkbfungi: thats what the second volume on the etherpad was, I was about to start recording info about that but you were ahead of me :)15:52
*** dtroyer has joined #openstack-infra15:52
*** thorst has joined #openstack-infra15:56
clarkbfungi: you should update the volume attach and detach commands if one of those volumes is that old fs15:56
fungiclarkb: oh, yep, this needs further cleanup. apparently we added the old and new cinder devices into one vg15:56
fungistill detangling15:56
clarkbin that case probably easier to clean up after the fact?15:56
* clarkb defers to you as you are working it15:57
*** dprince has quit IRC15:57
fungiwell, i'm removing the old logvol, and then will see what's involved in a vgreduce (whether we have blocks for one lv on both pvs, which we likely don't)15:57
*** dprince has joined #openstack-infra15:58
fungiclarkb: chances are we want a larger cinder volume anyway as we're at 83% full on one 100gb volume for production already15:58
*** mikelk has quit IRC15:58
*** jsavak has joined #openstack-infra15:59
*** Nikolay_St has quit IRC15:59
fungiwe could just lvextend this onto the second cinder volume, or pvmove onto a new larger one15:59
fungialso both are things which can be done online after the maintenance15:59
yolandafungi clarkb, so we just need the main-gerrit volume, not the gerritold? i was documenting everything assuming two volumes, but looking at review, fstab, etc... and your comments, seems that one is old and trash?15:59
fungiyolanda: yes, i'm still working on cleaning that up so it should simplify teh maintenance steps16:00
*** dizquierdo has quit IRC16:01
*** dizquierdo has joined #openstack-infra16:02
anteayaso 4 hours until gerrit goes offline16:02
fungiyep, after an lvremove of main-gerritold, /dev/xvdb1 is 100% free so i can vgreduce it out of main16:02
anteayahow do folks feel about issuing a channel status alert to folks as a reminder?16:02
fungianteaya: i was going to suggest a #status notice (not alert) at around now since it's the 4-hour mark16:02
yolandai'll step out for a while and will come back later16:02
fungiyolanda: thanks for working on this!16:03
anteayafungi: ah, sorry to get in your way16:03
anteayafungi: yes, I used the wrong word, notice16:03
*** zeih has joined #openstack-infra16:03
*** armax has joined #openstack-infra16:03
anteayafungi: and I agree with your suggestion of a notice16:03
clarkbimage builds failed this morning on a new git repo (I think there is a small race between creating the repo on git.o.o backends and on review.o.o and te replication to the backends16:04
*** jaosorior_away has quit IRC16:04
clarkbnot the end of the world16:04
fungiyes, there is a race there16:04
pabelangerclarkb: Ya, ansible was hung for about 12 hours16:05
*** jaosorior has joined #openstack-infra16:05
anteayaclarkb: yes, I think that was what wznoinsk was trying to draw attention to last week when he was talking about dib breakages16:05
clarkbpabelanger: similar thing as to when noepool went away?16:05
clarkbanteaya: those were related to the nodepool server going away and ansible getting unhappy about it16:05
fungidid we decide on a duration for the maintenance window? one hour? more?16:05
clarkbanteaya: and were corrected same day16:05
anteayaclarkb: ah, thank you16:05
pabelangerclarkb: yes, looking into the --timeout flag for ansible-playbook to see if that helps resolve the blockage16:05
anteayafungi: I think we had decided one hour but I'm not sure that was communicated16:06
clarkbfungi: considering we shouldn't need to reindenx or any of that an hour should be plenty16:06
anteayafungi: just the start time was communicated in the initial email16:06
fungisomething like #status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement
AJaegerthanks, amrith for the bug!16:06
anteayathe followup email also doesn't appear to communicate duration16:06
pleia2good morning16:07
anteayafungi: works for me, thank you16:07
anteayapleia2: good morning16:07
*** jsavak has quit IRC16:07
*** jsavak has joined #openstack-infra16:07
fungiokay, after a vgreduce and pvremove of /dev/xvdb1 on review.o.o i'm ready to detach that cinder volume16:08
fungi#status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement
anteayawherefore art thou statusbot?16:08
fungioh, i forgot statusbot was sleeping in. moments while i wake it up16:09
anteayait is Monday after all16:09
anteayameetbot seems to be online16:09
*** openstackstatus has joined #openstack-infra16:10
*** ChanServ sets mode: +v openstackstatus16:10
*** tphummel has joined #openstack-infra16:11
*** jordanP has quit IRC16:11
fungii'm starting to think that we should configure statusbot not to hang out in the meeting channels. we tend to cause fights between it and meetbot over channel topics, and it's arguably disruptive to have it chiming in during meetings at all16:11
fungion the other hand, there's the chance that a meeting channel is the only place some users are lurking16:11
fungithough i think that's probably a corner case not worth concerning ourselves with16:11
*** Hal has joined #openstack-infra16:12
*** Hal is now known as Guest6132316:13
anteayacan statusbot still report in meeting channels but defer to meetbot on the topics?16:13
*** jcoufal has quit IRC16:14
*** julim has quit IRC16:14
pleia2it doesn't currently have that capability16:14
*** david-lyle_ is now known as david-lyle16:15
anteayaah okay, well then I agree with the corner case not worth concerning ourselves with16:15
fungisome infra-root want to check behind me before i `nova volume-detach 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1` ?16:15
*** fawadkhaliq has joined #openstack-infra16:16
*** armax has quit IRC16:16
fungicinder show says it's /dev/xvdb on 87741fa9-92cb-4d1e-8195-6841e2468fb2 (the old, current production review.o.o)16:16
pleia2I'll have a look, what should it be?16:16
*** fawadkhaliq has quit IRC16:16
clarkbthats unfortunate /dev/by-id doesn't have the cinder info for these devices (guessing due to lvm?)16:16
fungii'm just trying to make doubly-sure i'm not about to detach the in use cinder volume for /home/gerrit216:17
fungi#status notice Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement
openstackstatusfungi: sending notice16:17
fungithe name and creation date on that volume are also earlier than the presumed active one, so i think it should be fine16:18
clarkbfungi: sudo pvs says xvdc1 is the only device backing lvm right now. I am assuming xvdb is the one you are trying to remove, I think a nova volume-show will confirm that16:18
*** gongysh has quit IRC16:18
-openstackstatus- NOTICE: Reminder, Gerrit will be offline from 20:00 to 21:00 UTC for a server replacement
* clarkb runs thta command to check16:18
*** e0ne has quit IRC16:18
robcresswellHey all. What's the process for adding to horizon-stable-maint? Is it a dev mailer process, or patched etc?16:19
robcresswellSorry if this isn't the correct place to ask.16:19
anteayarobcresswell: adding what to horizon-stable-maint?16:19
anteayaa person?16:19
fungiclarkb: hrm, it complains i don't have a cinder v2 endopoint in the catalog16:20
AJaegerrobcresswell: the ACL is self-maintained, ask one of the people in it to add you if you're not in there - and if you're there, do it yourself ;)16:20
openstackstatusfungi: finished sending notice16:20
openstackgerritMerged openstack-infra/project-config: [sahara] disable releasenotes jobs for liberty and kilo branch
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: cleaning up python3 jobs for sahara-tests
clarkbfungi: | 4ae4bfbf-c90e-4bad-a9d8-4c1f70604ac1 |          | in-use    |  100 | Attached to on /dev/xvdb           |16:20
clarkbfungi: I had to use openstack client16:20
robcresswellanteaya: Yes16:20
anteayarobcresswell: great, I think AJaeger's response is probably the best one16:20
anteayazaro: morning zaro16:21
anteayazaro: we are here:
robcresswellAJaeger: None of the existing members are able to add; the group itself is owned by stable-maint-core so I assumed it was managaed by those.16:21
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: cleaning up python3 jobs for sahara-tests
*** vgridnev has quit IRC16:21
AJaegerrobcresswell: then ask the stable team for it16:21
robcresswellAJaeger: Will do, thanks.16:21
fungiclarkb: which osc verb do i want? is it under server?16:21
clarkbfungi: openstack --os-cloud openstackci-rax --os-region-name DFW volume list16:22
fungiclarkb: that's calling into volume not server though, so equivalent to cinderclient not novaclient?16:22
clarkbthe message I got from nova client was use cinderclient or osc16:22
fungii got the same info out of cinder show already16:22
*** cdent has joined #openstack-infra16:23
*** jogo has quit IRC16:24
fungiso i think i'll amend the instructions to only move the volume we're 99+% sure is the one in use, and then delete the other one when we delete the old server16:24
fungiif we make the db backup into /home/gerrit2 then we can also use that as a canary to let us know we've got the right volume16:25
anteayado we say in the instructions how long we are going to wait before we delete the old server?16:25
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Remove statusbot from meeting channels
*** jogo has joined #openstack-infra16:25
fungianteaya: we don't say, but i'd give it a couple days16:25
AJaegerteam , here's the suggested statusbot removal on meeting channels ^16:25
anteayafungi: great16:25
anteayaAJaeger: oh I had thought we had said we weren't going to bother16:26
*** esker has quit IRC16:26
anteayaAJaeger: as there are some cases where having it report in meeting channels can be useful16:26
pabelangerclarkb: have we ever talked about using async in our playbooks for puppet before?16:26
pleia2anteaya: I think AJaeger is right, keeping it *in* the channel was a corner case16:26
fungiyes, that's what i attempted to say (perhaps not clearly)16:27
*** scheuran has quit IRC16:27
clarkbpabelanger: I don't think so, not sure we would need to with only ~10 connections at any time16:27
AJaegeranteaya: let me reread the dsicussion, seems I misunderstood it.16:27
anteayaAJaeger: or perhaps I did16:27
*** yamahata has joined #openstack-infra16:27
fungiAJaeger: nope, your change is what i was suggesting. i'm in favor, i just think we should collect some input on it before we approve16:28
esikachevguys, we need review this needs for release16:28
*** fawadkhaliq has joined #openstack-infra16:28
*** harlowja_at_home has quit IRC16:29
AJaegeranteaya: I interpreted your answer the way that you're in favor of the change I did  ("I agree with the corner case not worth concerning ourselves with")16:29
pabelangerclarkb: Ya, my thought ansible gets less flaky when the far end disappears when using async, since it reconnects ever x seconds.16:29
*** armax has joined #openstack-infra16:29
AJaegeresikachev: please explain what the change is, which repo it touches and why it blocks you.16:29
pabelangerbut not sure that is the best solution either16:29
AJaegeresikachev: not everybody here reviews everything...16:29
fungi"people only lurking in meeting channels and nowhere else" was what i was saying i felt was a corner case we could reasonably ignore16:30
*** abregman has joined #openstack-infra16:30
*** bkeller` has quit IRC16:30
anteayafungi: ah I mis-understood you16:30
anteayaI've commented on the patch16:30
anteayait is possible I am in the minority16:30
fungianteaya: not at all, that's why i encouraged further discussion even though i'm in favor16:31
anteayayup, thanks16:31
fungifeel free to make your comment a -1 too if you have a strong enough opinion on it16:31
*** bkeller` has joined #openstack-infra16:31
anteayano not strong enough for a -116:32
anteayajust registering what I see as a usecase16:32
*** apoorvad has joined #openstack-infra16:33
*** hdd has joined #openstack-infra16:33
mfischcan we get openstack-security as a tag that I can subscibe to?16:33
mfischsince my mail to openstack-sec..@lists was redirected to -dev16:33
*** jaosorior has quit IRC16:33
clarkbmfisch: you need to contact the list admins for the list in question16:33
AJaegerbtw. if my change merges, let's merge - kolla to #openstack-kolla rename - and (powervm) as well ,please16:33
clarkbmfisch: they should be listed on the page for that specific list16:34
fungimfisch: official subject tags need to be set by the list admin, so ttx in this case according to the details at the bottom of
zarofungi: what do you mean 'all content' from /home/gerrit2?16:34
mfischlooks like thieerry is admin16:34
fungizaro: anything inside /home/gerrit2 on the root filesystem, which will get shadowed when we mount the logical volume there16:35
zarofungi: what else is in there besides review_site?16:36
fungizaro: dotfiles/dotdirs from /etc/skel at a minimum16:37
zarofungi: ohh, i see, you mean clean it out in prep for existing gerrit2 content16:38
zaroyeah, good idea i guess.16:38
*** Guest61323 has quit IRC16:38
*** rcernin has quit IRC16:40
fungiright, it's hard to delete files off the parent filesystem after you mount a new filesystem on top of them16:43
fungiso easier to move or delete them out of the way before anything is mounted there16:43
*** jaosorior has joined #openstack-infra16:44
fungiclarkb: thinking more about the reboot idea, there's a bit of a chicken and egg with turning puppet back on16:45
clarkbfungi: is it?16:45
*** sputnik13 has joined #openstack-infra16:45
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
fungiclarkb: we want to reenable puppet and test that it's working before we reboot i think? since services (gerrit/gerritbot) are likely to start automatically at boot16:45
fungior maybe we've already got them well disabled. checking16:46
clarkbin that case I think we can just run puppet first, then reboot?16:46
clarkbbasically we should reboto at some point during the time window16:46
fungii agree we should reboot to test that it's safe16:47
fungii'm just wondering to what extent we can rely on the reboot itself to do things for us (like finding, activating and mounting the logvol)16:47
anteayadid we discuss at all if we are going to address the zuul memory usage issue during the gerrit maintenance window?16:47
clarkbfungi: you mean the /home/gerrit2 volume?16:48
fungiclarkb: yep16:48
*** thiagop has joined #openstack-infra16:48
clarkbya may need to just mount -a after editing fstab, run puppet, then reboot later16:48
*** fawadkhaliq has quit IRC16:48
fungianteaya: it's probably a good time to do a zuul restart (in fact, i was thinking that we probably need to stop zuul before this maintenance anyway and start it again after), though no idea if there were patches jhesketh and jeblair wanted to try getting in place in advance of that restart16:49
*** fawadkhaliq has joined #openstack-infra16:49
anteayafungi: can you add some zuul steps in the etherpad, then I can see if I can track down patch urls if any apply?16:49
openstackgerritMerged openstack-infra/tripleo-ci: Temporarily disable mistral on the undercloud
*** infra-red has joined #openstack-infra16:51
fungianteaya: just added, yes16:51
anteayafungi: thank you16:51
anteayajeblair: I have lost track, I saw you and jhesketh discussing earlier, did you want that patch to merge prior to restarting zuul today?16:51
*** yamamoto has joined #openstack-infra16:52
*** esikachev has quit IRC16:53
*** bkeller` has quit IRC16:53
*** dutsmoc is now known as comstud16:54
*** dtantsur|afk is now known as dtantsur16:55
*** derekh has quit IRC16:57
*** dprince has quit IRC16:57
*** daemontool__ has joined #openstack-infra16:57
*** dprince has joined #openstack-infra16:58
*** julim has joined #openstack-infra16:59
*** yamamoto has quit IRC17:00
*** daemontool_ has quit IRC17:00
jeblairanteaya, fungi: yeah, i will restart zuul with that patch manually applied17:02
jeblairanteaya, fungi: (and only that patch -- so the foreign project support will be back in)17:03
*** vgridnev has joined #openstack-infra17:04
fungii've cleaned up/expanded on the maintenance steps further to include the steps for mounting the homedir and suggested reboot17:04
fungiand moved the puppet disablement to pre-maintenance17:05
*** flepied has joined #openstack-infra17:05
jeblairfungi: oh, we should remove the init script links on old gerrit17:05
*** fawadkhaliq has quit IRC17:06
jeblair('reboot' made me think of that)17:06
fungigood call17:06
anteayajeblair: thank you17:06
*** jamesmcarthur has quit IRC17:07
*** Hal has joined #openstack-infra17:07
*** Hal is now known as Guest516217:07
*** hdd has quit IRC17:07
fungijeblair: added in step 1.117:08
*** edmondsw has quit IRC17:08
fungiright after we stop the services17:08
*** flepied1 has quit IRC17:08
anteayaI have edited step 7.5 to added manually applying the contents of 281166 to zuul prior to restart17:08
funginoted explicitly to stop gerritbot on the old server too, that was missed17:08
fungithanks anteaya17:09
anteayathank you17:09
fungiat this point all the suggestions i can think of are addressed in the etherpad17:09
anteayafungi: yay17:09
*** fawadkhaliq has joined #openstack-infra17:09
fungibut we still have a few hours to go if anyone realizes something's missing/wrong17:09
*** dizquierdo has quit IRC17:10
AJaegeranteaya, fungi: is applying on the agenda?17:10
anteayaAJaeger: yes, line 6417:11
AJaegerThat's the heap size change17:11
AJaegerthanks, anteaya !17:11
AJaegershould I remove my WIP?17:11
anteayawell it isn't a step but the patch is noted17:11
*** bpokorny has quit IRC17:11
anteayaAJaeger: I thinking removing the WIP at this point should be okay17:11
AJaegerthanks, anteaya17:11
anteayathanks for the patch!17:12
*** ihrachys has quit IRC17:13
*** jsavak has quit IRC17:13
fungiwondering whether that should happen after we disable puppet for the old gerrit server but before the maintenance? or if we should plan to add that after gerrit's back up and then do a separate gerrit restart to pick it up17:13
*** jsavak has joined #openstack-infra17:13
*** e0ne has joined #openstack-infra17:14
anteayawould there be any benefit to adding it after gerrit is back up?17:14
anteayaI'm leaning towards one gerrit stop/start myself17:15
fungiit's one more thing to have to revert if we roll back to the old server17:15
anteayaokay well I guess I quick gerrit restart afterwards isn't a huge inconvience17:15
clarkbconsidering it is the reason for this whole swith we should try to get it in soon17:16
anteayaand having a clean roll back path is important17:16
*** tonytan4ever has quit IRC17:16
anteayaclarkb: oh I think we are agreeing the change happens right after the migration17:16
anteayajust not the same as the migration17:16
anteayaat least that I what I'm agreeing to17:17
fungimainly a question of whether we do only one gerrit restart and so squeeze it in before we take zuul and old gerrit offline, or whether we do a second gerrit restart with that chance in place after we've verified it's up and working17:18
*** nadya has joined #openstack-infra17:18
anteayavote option two17:18
jeblairvote option one17:19
fungii could go either way, though option will make the maintenance slightly shorter (assuming we don't need to roll back)17:20
*** bpokorny has joined #openstack-infra17:20
fungier, option one will17:20
*** skolekonov is now known as skolekonov_away17:20
dimsfungi : is it easy to tickle the "propose-requirements-constraints-master" periodic job? the run this morning failed and i wanted to see if it happens again. there's no rush at all, just curiousity17:22
*** tonytan4ever has joined #openstack-infra17:22
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Metrics tracking for TripleO deployment tasks
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add common bash functions to help track metrics.
nibalizerI think option one as well17:25
clarkbya I think we should just do it17:26
clarkbit is a really easy revert17:26
krotscheckinfra-root: I'd like to grab the attention of someone this week to help land - does anyone have a few hours, say, wednesday?17:26
nibalizersafe change i think17:26
krotscheckWork would involve copy/pasting logs so i can debug them.17:26
pleia2wednesday is our bug day17:27
*** e0ne has quit IRC17:27
pabelangerpleia2: nice, forgot about that17:27
krotscheckTomorrow's my meeting day17:27
*** lakshmiS has quit IRC17:27
pabelangerkrotscheck: if you haven't found anybody else, feel free to ping me17:27
krotscheckpabelanger: Awesome, thanks :)17:28
fungidims: not sure if we can reenqueue periodic jobs, but i can give it a shot17:28
*** trown is now known as trown|lunch17:28
nibalizerkrotscheck: i have cycles this week :)17:28
krotschecknibalizer: Do you have cycles... right now? ;)17:28
*** lakshmiS has joined #openstack-infra17:29
*** jsavak has quit IRC17:29
dimsthanks fungi17:29
*** Rockyg has joined #openstack-infra17:29
*** e0ne has joined #openstack-infra17:30
nibalizerkrotscheck: no, today is break gerrit day17:30
krotschecknibalizer: Wednesday is bug day. Tomorrow I've got meetings all other the place, so I can't pay 100% attention to what's going on.17:31
jeblairkrotscheck: 617 needs an update17:31
krotscheckjeblair: Checking17:31
*** jaosorior has quit IRC17:31
nibalizerkrotscheck: thursday?17:31
fungidims: looks like zuul enqueue-ref was able to do it17:31
fungisudo zuul enqueue-ref --trigger timer --pipeline periodic --project openstack/requirements --ref refs/heads/master --newrev d4326d4ced84677ed37f5709cf0637b092044e9e17:31
*** ddieterly is now known as ddieterly[away]17:31
fungidims: i see it running on the zuul status page now17:31
dimsah nice! thanks fungi17:32
*** jsavak has joined #openstack-infra17:32
fungiCommand '['/tmp/tmpcJLYgc/bin/pip', 'install', '-r', 'global-requirements.txt']' returned non-zero exit status 217:32
krotschecknibalizer: ...thursday I have a thing at the Google offices here in kirkland, but if you're available around 7AM, I'm sure we can get it landed :)17:32
dimssaved it in case ...
jeblairkrotscheck: does the principal exist?17:32
dimsy that was the same problem from before17:32
dimsfungi : thanks, will try to repro locally17:33
krotscheckjeblair: I don't think so.17:33
fungidims: so this may be related to the "need to install libvirt to pip install python-libvert latest release?17:33
fungidims: we probably also collect the tox logs from this. checking now17:33
jeblairkrotscheck: ok; that should be done before landing it17:33
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update
fungidims: ahh, nope, we just collect the console log for that job17:34
dimsfungi : ack17:34
nibalizerso is this the only review for today's activity ?
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to mirror_update
krotscheckjeblair: Added a note to the commit message, and addressed your issues ^^17:34
jeblairkrotscheck: w00t; +217:35
anteayanibalizer: you can also review: which will be manually applied to zuul prior to restart as outlined in step 7.517:35
anteayanibalizer: but yes 303846 is the only patch we need to merge today17:35
fungidims: anyway, i saw someone say that global requirements containing libvirt-python now means you have to install the libvirt (library? headers?) from your distro since the 1.3.3 release late last week17:36
dimsfungi : thanks let me see if i can find anything17:37
*** vikram_ has quit IRC17:38
AJaegerteam, seems proposal slave is out of space see
AJaeger"IOError: [Errno 28] No space left on device"17:38
fungiAJaeger: oh, that could explain dims's issue too17:38
anteayaAJaeger: :/17:38
fungilooking into it now17:38
AJaegerthanks, fungi!17:38
fungisays /dev/xvda1       40G   37G  544M  99% / (so almost full anyway)17:39
dimsAJaeger : yay nice find17:39
AJaegerdims: I just saw some translation jobs failing - and those shouldn't fail ;)17:39
AJaegerdims: but wouldn't have checked without your inquiery17:40
AJaegerdims, thanks for asking!17:40
fungichances are we just don't have enough space in the homedir for all the workspaces which build up over time, but i should know the culprit in a bit17:40
fungi28gb in /home/jenkins17:40
fungi(this is one of the reasons we moved most jobs to single-use workers)17:41
fungiyeah, by far the bulk of the rootfs utilization is from /home17:41
AJaegerwe create now a venv for the releasenotes and don't clean that one up ;(17:42
fungipleia2: you have nearly 2gb of data in your homedir on proposal.slave.o.o, not sure what it is or whether you still need it17:42
*** dangers_away has quit IRC17:42
AJaegerI could remove that one...17:42
*** oanson has joined #openstack-infra17:42
*** bkeller` has joined #openstack-infra17:42
*** sc68cal has quit IRC17:43
pleia2fungi: thanks, probably some translations stuff, I'll get it sorted17:43
odyssey4mepabelanger are we in a position to do experimental xenial checks on any jobs yet?17:43
pabelangerodyssey4me: Yup, we are asking projects not to move them from experimental to check / gate until after the conference.17:44
fungipropose-requirements-updates is by far the largest workspace at 1.8gb17:44
pleia2fungi: ok, better now17:44
clarkbfungi: it clones all of the repos iirc17:44
*** mrmartin has joined #openstack-infra17:44
clarkbfungi: in order to generate the transitive dep list17:44
pabelangerodyssey4me: so, we could likely add experimental job for ansible17:44
odyssey4mepabelanger ah ok, cool - that'd be nice :)17:45
fungithough a lot of the translation-update jobs are big too17:45
fungihundreds of mb each17:45
odyssey4mepabelanger another thing I needed to discuss - I'd like to add release announcement and release note jobs to the roles - should we add that to the template, or to individual jobs?17:45
pabelangerodyssey4me: should be easy to add it into ansible-role-jobs.yaml17:45
anteayaodyssey4me: just to give you an update on the rename of your repo: we haven't scheduled it yet17:46
pabelangerodyssey4me: Ya, we should have a template for sure.17:46
fungifor now, i'm going to take proposal.slave.o.o offline in jenkins for a few minutes and as soon as it's not running any jobs, clear out its workspace cache. separately i'll start working on adding a cinder volume for /home/jenkins on it17:46
anteayaodyssey4me: currently renames require gerrit to be offline for 4 hours17:46
odyssey4meanteaya thanks for the update - any idea on when we'll be able to set a date?17:46
pabelangerodyssey4me: I haven't looked into reno jobs as of yet but if you get a patch up, I can help review17:46
anteayaodyssey4me: so timing renames is more onourous that it used to be17:46
AJaegerfungi, we delete some of the translations every day. Is there something else I should delete for translations?17:46
anteayaodyssey4me: well it won't happen prior to summit17:47
openstackgerritJohn L. Villalovos proposed openstack-infra/irc-meetings: Use a boostrap grid to list all the meetings
odyssey4meanteaya yeah, I'm aware of that - but we wanted to get it into the queue to happen as soon as possible, whenever a time can be scheduled for it17:47
anteayaodyssey4me: and we like to group them if we can since it is 4 hours per rename, regardless of how many we do17:47
*** Rockyg has quit IRC17:47
anteayaodyssey4me: okay great, just thought I'd let you know17:48
odyssey4me4 hours *per rename* - ouch!17:48
odyssey4meI thought it was 4 hours overall. :/17:48
fungiAJaeger: the openstack-manuals translations seem to be the largest at anywhere from 0.5-1gb in size (and there are 7 that i see, so possibly cruft from multiple job renames?)17:48
anteayaodyssey4me: sorry, no that is what I mean17:48
anteaya4 hours for every rename maintenance17:48
AJaegerfungi, do you have some names for me?17:48
jlvillalttx: I removed the WIP message from:   No other changes.17:48
anteayaodyssey4me: when do you expect a tag will be used for the repo? milestone 1?17:48
*** infra-red has quit IRC17:48
openstackgerritHenrique Truta proposed openstack-infra/project-config: Run identity v3 only job in nova
*** shardy has quit IRC17:49
fungiAJaeger: sizes in mb for the top 19 job workspaces on that server
*** shashank_hegde has joined #openstack-infra17:50
AJaegerfungi, anything "*-zanata" can be deleted completely17:50
AJaegerfungi, and we did rename the jobs - and they are all recreated, aren't they? So get rid of all translation jobs...17:51
*** hdd has joined #openstack-infra17:51
*** jistr has quit IRC17:51
fungiAJaeger: removing those seems to have freed up about 3.3gb17:51
clarkbI think we should start with just remove all workspaces17:51
clarkblet it steady state again and if we need to then add a volume17:51
fungiyeah, can do17:51
*** tonytan4ever has quit IRC17:51
clarkbits possible we are well under the threshold we just have old cruft17:51
clarkb(jenkins doesn't delete workspaces when jobs are deleted?)17:51
fungii don't think it does, no17:52
*** tonytan4ever has joined #openstack-infra17:52
odyssey4meanteaya for the moment we'll be referring to the github URL for the repo in all docs and code, as the rename will cause a redirect when accessing github so it'll work before and after17:52
reedwho manages the Corporate CLA nowadays? this page should have a contact point:
odyssey4meanteaya once the rename is done, we'll switch to using git.o.o in all branches17:52
fungiokay, i'm blowing away all cached workspaces there and then we can look at it again later in the week to see what's going on there17:52
clarkbreed: danny does iirc17:52
anteayaodyssey4me: your rename commit message mentions tags breaking17:52
fungireed: danny careno17:52
clarkbreed: I will let him know to update the wiki17:53
reedclarkb, fungi, k, thanks17:53
fungireed: thanks for spotting that17:53
anteayaodyssey4me: when do you expect applying a tag to that repo?17:53
*** ddieterly[away] is now known as ddieterly17:53
reedi'd update it but Launchpad is taking a break17:53
odyssey4meanteaya we'll be doing the first tag before the summit, so we had to make a plan and work with what we have17:53
*** shardy has joined #openstack-infra17:53
*** achanda has quit IRC17:53
fungiokay, cached workspaces have been cleaned up on proposal.slave.o.o and it's reenabled on jenkins.o.o now17:54
anteayaodyssey4me: oh well the commit message just mentions a tag, I didn't know you would be tagging prior to summit17:54
anteayaodyssey4me: when are you taggin?17:54
*** pvaneck has joined #openstack-infra17:54
odyssey4meanteaya hopefully at the end of this week - but failing that it'll be by the end of next week17:54
anteayaodyssey4me: can you say so in the commit message?17:55
odyssey4meanteaya sure, I'll fix that up17:55
anteayaodyssey4me: we can discuss it again at tomorrow's meeting, if someone feels like doing a rename prior to you tagging then it saves a lot of trouble17:55
anteayaodyssey4me: I don't think anyone knew your tagging plans so it was hard to keep them in mind when discussing maintenance17:56
*** lakshmiS has quit IRC17:56
*** achanda has joined #openstack-infra17:56
*** lakshmiS has joined #openstack-infra17:56
*** tonytan_brb has joined #openstack-infra17:57
anteayamyself I was assuming milestone 117:57
fungidims: i've retriggered it again, console log is streaming now at
dimscool fungi : will watch that17:57
openstackgerritTim Burke proposed openstack-infra/git-review: Remove worthless print
fungidims: or, you know, do something productive and check it again in a little while ;)17:58
fungiit's likely not a fast job17:58
dimsfungi : LOL17:58
*** amotoki has quit IRC17:59
*** tonytan4ever has quit IRC17:59
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Rename openstack-ansible-ironic to openstack-ansible-os_ironic
odyssey4meanteaya ^ is that suitable?18:00
*** exploreshaifali has quit IRC18:00
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
fungidims: oh, it is a fast job after all, and has already succeeded and produced
anteayawell you said end of this week in irc, so I'll add that as a comment on the patch18:01
dimswasn't bad at all! 4 mins18:01
clarkbfungi: how big is the workspace now?18:01
*** sambetts is now known as sambetts|afk18:01
dimsthanks. review it generated looks good fungi18:01
*** doug-fis_ has joined #openstack-infra18:01
clarkbI wonder if that job needs more git clean action18:01
fungiclarkb: 6mb18:01
anteayaodyssey4me: we are just all really tired and would like to avoid more maintenance if we can, but we also don't want to break things if we can avoid it18:01
*** dtantsur is now known as dtantsur|afk18:01
fungiclarkb: though that wasn't one of the big workspaces before18:02
clarkboh I thought it was for some reason18:02
fungiclarkb: propose-requirements-updates was the big one18:02
*** doug-fi__ has joined #openstack-infra18:02
odyssey4meanteaya yep, completely understood - we're happy to roll with it and will fit in however we can - let me know if there's anything I can do to help18:03
*** jsavak has quit IRC18:03
anteayaodyssey4me: well know I know your tagging intentions that helps me understand, thank you18:03
fungiclarkb: my console history says it was 23mb before cleanup18:03
*** jsavak has joined #openstack-infra18:03
anteayaodyssey4me: if you are available tomorrow for the infra meeting at 19:00 utc that can't hurt as others may have questions18:03
*** dimtruck is now known as zz_dimtruck18:03
*** achanda_ has joined #openstack-infra18:04
*** achanda has quit IRC18:04
odyssey4meanteaya sure, let me add a reminder18:04
anteayaodyssey4me: great, thank you18:04
fungiclarkb: so crufty, but not one of the huge "problem" workspaces18:04
*** doug-f___ has joined #openstack-infra18:04
*** doug-fish has quit IRC18:04
odyssey4meanteaya which room is the meeting in?18:05
anteayaodyssey4me: in #openstack-meeting18:05
*** maishsk has joined #openstack-infra18:05
odyssey4meanteaya cool, reminder set18:05
anteayaodyssey4me: great, thank you18:05
*** doug-f___ is now known as doug-fish18:06
*** doug-fis_ has quit IRC18:06
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: remove release note building venv
AJaegerfungi, that venv was 230 MB just for nova on my local system...18:07
*** doug-fi__ has quit IRC18:07
fungiAJaeger: wow, pretty big18:07
*** Sukhdev has quit IRC18:07
*** denisra has joined #openstack-infra18:07
AJaegerbut venv is used for tarballs as well, so is it good to remove it?18:08
AJaegerwait, tarballs run on a different node, don't they?18:08
openstackgerritHenrique Truta proposed openstack-infra/project-config: Run identity v3 only job in cinder
*** jcooley_ has joined #openstack-infra18:09
*** daemontool__ has quit IRC18:10
*** avarner has quit IRC18:11
fungiAJaeger: tarball generation happens on single-use workers because it runs arbitrary code from the projects18:12
fungicourtesy of the python ecosystem relying on as its package building entrypoint18:13
*** jcooley_ has quit IRC18:13
*** jcooley has joined #openstack-infra18:13
fungithis is why we split out package uploading to pypi as a separate job18:13
*** trown|lunch is now known as trown18:14
fungiso that we don't compromise our upload credentials18:14
AJaegerfungi, ok. In Austin, I'd like to sit with somebody together to review the translation jobs setup and figure out whether there's any risk...18:14
*** tiswanso has quit IRC18:14
*** tricksters is now known as elopez18:14
anteayaAJaeger: do you want to add that to the sessions etherpad?18:14
*** tiswanso has joined #openstack-infra18:15
*** elopez is now known as eric_lopez18:15
AJaegeranteaya: good idea. Do you have the link handy?18:15
fungiAJaeger: well, we can't _rely_ on cleanup steps in jobs to fire (for example if the job breaks and terminates early), but a git clean at the end probably can't hurt18:15
AJaegerfungi, will think about more about cleanup...18:15
AJaegeranteaya: thanks18:15
*** ddieterly is now known as ddieterly[away]18:16
anteayaAJaeger: thank you18:18
*** sdake has joined #openstack-infra18:18
openstackgerritHenrique Truta proposed openstack-infra/project-config: Run identity v3 only job in swift
crinkleif an infra-core has the time it would be neat to have feedback on , since it'll be easier to start work on liberty->mitaka after we have functional tests18:19
*** kzaitsev_mb has quit IRC18:20
*** fawadkhaliq has quit IRC18:20
*** shardy is now known as shardy_afk18:20
*** fawadkhaliq has joined #openstack-infra18:21
anteayacrinkle: I've added that link as a reference for clarkb's proposed sprint session on line 116:
*** achanda_ has quit IRC18:21
*** achanda has joined #openstack-infra18:21
AJaegerfungi, we currently have jenkins/scripts/ that deletes once a day the .venv directory18:22
AJaegershould I extend this to run "git clean"?18:22
AJaegerOr remove after each run?18:23
*** cody-somerville has quit IRC18:23
openstackgerritMerged openstack/requirements: Bump os-net-config to 0.2.4
*** dangers_away has joined #openstack-infra18:24
*** mrmartin has quit IRC18:24
*** avarner has joined #openstack-infra18:25
crinkleanteaya: thanks, though it's only four patches and most of them are reasonably small so i was hoping they'd get attention a little sooner than the summit18:25
fungiAJaeger: oh, we have jobs creating a ~jenkins/.venv too? git clean won't help with that (and it's also not contributing to workspace sizes)18:26
anteayacrinkle: oh yes, I agree, but I think that same query will apply for new infra-cloud patches, will it not?18:26
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** mrmartin has joined #openstack-infra18:26
fungiAJaeger: also that ~jenkins/.venv is basically empty (just contains the CREATED marker file)18:27
*** achanda has quit IRC18:27
crinkleanteaya: actually a better query would be (topic rather than project), I'll go ahead and change it18:27
AJaegerfungi, I wonder whether it's the wrong directory that gets cleaned...18:27
anteayacrinkle: awesome, thank you18:27
*** tiswanso has quit IRC18:28
fungiAJaeger: what jobs are creating this that you're trying to clean up after?18:28
* AJaeger currently tries to remember and digs into git logs...18:28
*** tiswanso has joined #openstack-infra18:28
*** ociuhandu has quit IRC18:29
*** infra-red has joined #openstack-infra18:29
AJaegerfungi, do you have a ~/.m2 file on the slave? How large is that?18:29
AJaegerdirectory I mean...18:29
*** achanda has joined #openstack-infra18:30
fungiAJaeger: looks like the cleanup is only called from these jobs running the proposal-slave-cleanup macro
AJaegerfungi, yes, correct.18:31
fungiAJaeger: there is a ~jenkins/.m2 but all it contains is a 1kb settings.xml file18:31
*** esikachev has joined #openstack-infra18:31
sdakethere is speculation that bluebox runs multiple kolla jobs at once and the multicast is enabled on their networks causing keepalived interference18:31
AJaegerfungi: good ;) So, do not collect lots of java files there18:31
fungisdake: aha! races18:31
sdakefungi not relaly races, kolla defaults to multicat address18:32
fungisdake: it's possible bluebox is _not_ filtering multipcast your jobs are leaking onto the provider network but other providers are?18:32
sdakeour keepalive config is password protected18:32
sdakeso it may be a bug in keepalived18:32
sdakefungi roger thats the speculation18:32
fungisdake: so yeah, lack of isolation from the outer environment in this case i guess18:32
sdakefungi we dont know for certain18:32
*** maishsk has quit IRC18:33
sdakethe other gate problem we had has to do with erlang crashing when compield with ipv618:33
sdakewhich was just done for centos about 2 weeks ago18:33
fungisdake: but i did see invalid password errors in your keepalived logs on multiple providers, not just bluebox18:33
* AJaeger thinks the slave script is just broken ;(18:33
sdakewhich coincides with major gate flakiness18:33
*** fawadkhaliq has quit IRC18:33
sdakefungi oh really?18:33
fungisdake: yeah, your jobs archive the keepalived log18:33
clarkbsdake: fungi our iptables rules should prevent that multicast18:34
*** fawadkhaliq has joined #openstack-infra18:34
sdakeclarkb unless we are using a 2x8 setup hopefully ;)18:34
fungiclarkb: well, unless that job is modifying iptables on the worker18:34
clarkbfungi: ya good point18:34
sdakewe do not modify ptables I am prety sure18:34
anteayaAJaeger: yeah, that might be the case18:34
*** nadya has quit IRC18:34
sdakefungi anyway just speculation at this point18:35
sdakeI thought I'd share since the password thing indicates the lack of isolation18:35
sdakewhich is maybe not what infra is after as clarkb indicates18:35
sdakethe password error is real though18:35
clarkbsdake: basically our tests should be isolated from each oter due to our firewall rules18:35
sdakeso something is either getting through or keepalived has gone buggy18:35
clarkbsdake: regardless of provider18:35
AJaegerfungi, anteaya : The idea was to delete the generated environments once a day (as optimization). I'll change this now to a git clean with some debugging output...18:36
fungisdake: well, assuming your jobs are testing correctly, you should just be able to recheck 304080 until it hits a bluebox worker to see whether that helps, but i agree better identification of the underlying cause is warranted18:36
sdakefungi yup thats what the plan is18:36
anteayaAJaeger: ah debugging, that sounds like the ticket18:36
anteayaAJaeger: let's have it tell us what it is doing18:36
*** achanda has quit IRC18:36
sdakefungi if you saw it on othre cloud providers, I am curious why only bluebox fails18:36
fungisdake: right, that may point to this being unrelated18:37
sdakemaybe the password lekaage is different from the keepalive faillign to go to master issue18:37
sdakerather multicast leakage18:37
fungibut anyway, should be able to tell by looking at the logs from job runs on this change whether the errors continue to appear in the keepalived logs18:37
sdakeits pretty consistent on bluebox18:38
sdakebut typically need lots of kolla patches running in the gate at the same time18:38
sdakei guess i'll find a  slection of patches to recheck :)18:38
fungisdake: the reason i remember is that i at first thought it might be related to the issue we were discussing, so i compared to a keepalived from a passing job run and it had the same invalid password errors18:38
sdakethis gate bug is not nearly as onerous as the ipv6 problem18:38
sdakefungi that is relaly intereting - I didn't think of that at the time18:39
sdakefungi i'll report back when we find out the underlyign issue18:39
sdakefungi (if it has effect for infra)18:39
*** piet has quit IRC18:40
*** jaypipes has quit IRC18:40
*** LaT0rtue has joined #openstack-infra18:41
*** julim has quit IRC18:42
fungicool, thanks!18:42
*** infra-re_ has joined #openstack-infra18:43
*** ddieterly[away] is now known as ddieterly18:44
*** daemontool has joined #openstack-infra18:46
*** med_ has quit IRC18:46
anteayait is an hour and 14 minutes before we begin our maintenance window, does everyone who is playing along know what they will be doing?
pabelangersdake: please take a moment to review as you are the kolla PTL18:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rework
*** infra-red has quit IRC18:47
AJaegeranteaya: I'll disappear before ;)18:47
anteayaAJaeger: figured you would have a plan18:47
AJaegerfungi, anteaya: Here's my rework ^18:47
*** mylu has quit IRC18:47
sdakepabelanger will do moment18:48
nibalizeranteaya: i'm planning on being in channel, I don't have any tasks right now18:48
nibalizerwhich is fine18:48
*** infra-re_ has quit IRC18:49
anteayanibalizer: great, thank you18:49
anteayanibalizer: thanks for reviewing the patch(es)18:49
*** tonytan_brb has quit IRC18:49
*** infra-red has joined #openstack-infra18:49
fungii'm planning to add the unpuppeting for old gerrit in a few minutes18:49
*** cdent has quit IRC18:49
sdakepabelanger ugh image changes t-5 days until release18:49
anteayafungi: awesome18:50
sdakepabelanger our release is the 15th, so this is our final go to week to fix bugs18:50
sdakewe need a functioanl gate, and from what I saw of the experimental jobs, bindep was failing18:50
*** mrmartin has quit IRC18:50
sdakei am happy to temporarily tear apart the gate after the 15th18:50
*** mylu has joined #openstack-infra18:50
*** tonytan4ever has joined #openstack-infra18:51
sdakeI hope that can fit in with your schedule?18:51
pabelangersdake: I am confident they are passing. they are only failing because of hardcoded tox settings. see:
fungibut it does seem prudent to wait another 4 days until their release work is done18:52
fungii'll be around all next week too and am happy to help get thise final image changes put through and debugged as needed18:53
sdakepabelanger nice18:53
sdakepabelanger ok that wfm then18:53
sdakestill a little frightening :)18:53
pabelangerfair enough we can hold off until kolla releases18:53
pabelangerthey are the last project on devstack-centos718:53
clarkbwe aren't pressured for disk space any longer18:54
sdakepabelanger yes we are asking the tc for a general one week extension for all deployment projects in release_with_milestones18:54
sdakealthough we re not in this release model yet we intend to be shortly18:54
*** med_ has joined #openstack-infra18:54
*** med_ has quit IRC18:54
*** med_ has joined #openstack-infra18:54
pabelangerfungi: okay, -1 workflow until you +A commit18:55
*** rbrndt has joined #openstack-infra18:55
pabelangerclarkb: Ya, that is true18:55
*** rbrndt__ has quit IRC18:55
anteayaAJaeger: reading the -d flag, do you want to use the -f flag twice in your command?18:55
sdakepabelanger ^18:56
sdakepabelanger i will add data to review about our deadlines18:56
*** tiswanso has quit IRC18:56
clarkbI should be around next week as well18:57
*** tonytan4ever has quit IRC18:57
clarkbits the best time to get work done18:57
*** doug-fish has quit IRC18:57
*** infra-red has quit IRC18:57
fungisince preference seemed to be around merging the jgit heap size bump prior to the maintenance, i've squeezed that in to happen soon after i disable puppet for the old server18:57
*** tonytan4ever has joined #openstack-infra18:58
*** doug-fish has joined #openstack-infra18:58
*** tiswanso has joined #openstack-infra18:58
*** infra-red has joined #openstack-infra18:58
AJaegeranteaya: using it twice is only needed "if an untracked directory is managed by a different Git repository" - so, we should be fine with doing it once. And the du output will show is if we really need it twice in case I missed anything...18:58
anteayaAJaeger: awesome, thank you18:58
*** sputnik13 has quit IRC18:59
AJaegeranteaya: thanks for double checking19:00
anteayaI do try19:00
*** yarkot_ has joined #openstack-infra19:00
anteayaand you're welcome19:00
*** julim has joined #openstack-infra19:01
yolandahi, i am around now. fungi, did you have progress with volumes?19:01
*** doug-fis_ has joined #openstack-infra19:01
fungiyolanda: yep, instructions updated19:01
fungiyolanda: i've also disabled puppet for the old uuid19:01
fungiand am tailing logs to make sure it stuck19:02
*** edmondsw has joined #openstack-infra19:02
*** sputnik13 has joined #openstack-infra19:02
yolandaok so everything looks ready for the replacement in one hour?19:02
*** doug-fish has quit IRC19:03
anteayafungi: so you have done step 0.2?19:03
fungii'll approve the heap size bump change in a few minutes if i don't see any evidence that the puppet disablement didn't take19:03
anteayayes you crossed it out19:03
fungianteaya: yep19:03
*** amitgandhinz has quit IRC19:03
anteayathank you19:03
fungianteaya: semi-done. i'm watching logs to make sure it worked as intended19:03
*** amitgandhinz has joined #openstack-infra19:03
anteayafungi: ah yes, saw you say that above, I just missed it was step 0.2 under discussion, sorry19:04
fungiother than approving there's not much else to do until moments before 20:0019:04
*** tonytan4ever has quit IRC19:04
openstackgerritMerged openstack-infra/puppet-infracloud: Reconcile compute and controller classes
openstackgerritMerged openstack-infra/puppet-infracloud: Parameterize libvirt virt type
AJaegerfungi, do you want to review my slave cleanup change while waiting? I'm still a bit around to discuss it if needed - but it's not urgent...19:05
*** doug-fis_ has quit IRC19:06
AJaegerwe could add this one also to all the propose jobs once we're sure it does the right thing19:06
openstackgerritHongbin Lu proposed openstack/requirements: Downgrade docker-py from 1.8.0 to 1.7.2
*** tonytan4ever has joined #openstack-infra19:07
fungiAJaeger: does `du -h --max-depth=1 .` have any benefit over the simpler `du -hs .`?19:07
fungiAJaeger: oh, i see you're getting the next level trees under it19:08
fungi-s is more like --max-depth=019:08
AJaegerfungi, correct19:11
* AJaeger wanted more details19:11
*** Sukhdev has joined #openstack-infra19:12
fungiseems fine. any reason to keep the daily limit/marker hassle?19:12
fungirather than just run this on each job invocation?19:12
AJaegerfungi, I can remove it. Idea was initially to reduce tox setup time...19:13
*** krtaylor has quit IRC19:13
fungiAJaeger: now that pip caches wheels, i doubt tox setup time is an issue any longer, but we can keep it around for now and you can compare i guess19:13
fungifirst run of the day will reflect the "longer" tox setup time19:14
AJaegerfungi, yep...19:14
fungi2016-04-11 19:11:11,218 p=4541 u=root |  ERROR! Unexpected Exception: [Errno 12] Cannot allocate memory19:15
fungijust saw that show up in the ansible run-all log19:15
funginot sure which host that was for19:15
fungioh, local!19:16
fungi[Mon Apr 11 19:11:09 2016] Killed process 18885 (ansible-playboo) total-vm:1188152kB, anon-rss:153304kB, file-rss:4kB19:16
funginibalizer: mordred: ^ if you want to dig into that juiciness...19:16
bkerooh boy oom19:17
fungisystem has 4gb virtual memory split between ram/swap19:18
fungii don't see a ton of resident memory on current processes according to top/free19:18
fungiwonder if we're trying to parallelize too much?19:19
*** jamesmcarthur has joined #openstack-infra19:19
clarkbfungi: or if shade is pulling too much data from clouds?19:20
AJaegerfungi, clarkb, do you consider updating of 40 jobs fine or might it involve manual JJB runs? I didn't approve yet...19:20
*** salv-orlando has quit IRC19:21
clarkb40 jobs should be fine19:21
fungiAJaeger: should be fine19:21
AJaegerso, approve now before downtime - or better tomorrow?19:21
funginow's probably fine19:22
fungigate seems pretty quiet19:22
AJaegeranteaya: I guess you like - adds voting job to gate that was only in check...19:23
clarkbI am about to eat a quick lunch and will be back fro the fun19:23
AJaegerfungi, clarkb : Thanks, approved19:23
openstackgerritMerged openstack-infra/project-config: Rework
fungii've approved 303846... hoping that puppet is really disabled for the old and it's not simply getting skipped because of the oom on the puppetmaster19:24
*** baoli_ has quit IRC19:25
*** xiangxinyong has quit IRC19:25
openstackgerritMerged openstack-infra/tripleo-ci: Dump end of logs to console.log on failure
*** x00350071 has joined #openstack-infra19:26
anteayaAJaeger: thank you19:26
nibalizerfungi: ok19:26
anteayafungi: ack19:26
*** piet has joined #openstack-infra19:27
*** ddieterly is now known as ddieterly[away]19:27
*** stevemar has quit IRC19:27
AJaegerthanks, anteaya19:28
*** krtaylor has joined #openstack-infra19:28
nibalizerlooks like we oom on ansible pretty frequently19:28
anteayawelcome, thanks for tracking that change19:28
*** stevemar has joined #openstack-infra19:29
openstackgerritMerged openstack-infra/project-config: Move gate-tempest-dsvm-neutron-dvr to gate queue
*** mrmartin has joined #openstack-infra19:30
*** nadya has joined #openstack-infra19:30
openstackgerritMerged openstack-infra/system-config: Raise gerrit heaplimit to 30 GB
*** Jeffrey4l__ has joined #openstack-infra19:32
mordredfungi: oh yay19:32
*** e0ne has quit IRC19:32
*** Jeffrey4l_ has quit IRC19:34
fungiand now that's merged, we're clear for another ~20 minutes before jeblair works on stopping zuul19:35
openstackgerritMerged openstack-infra/subunit2sql: Add more db api tests part 2
*** ihrachys has joined #openstack-infra19:36
openstackgerritMerged openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists
*** jamesmcarthur has quit IRC19:37
*** cdent has joined #openstack-infra19:37
*** mixos has joined #openstack-infra19:38
*** jamesmcarthur has joined #openstack-infra19:38
*** fawadkhaliq has quit IRC19:39
*** fawadkhaliq has joined #openstack-infra19:40
*** jamesmcarthur has quit IRC19:40
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add documentation about openstack-health deployment
*** jamesmcarthur has joined #openstack-infra19:41
mtreinishanteaya, yolanda: sry for the multiple month turnaround on ^^^19:42
*** jamesmcarthur has quit IRC19:42
mtreinishI forgot it was there19:42
anteayamtreinish: ha ha ha19:43
anteayamtreinish: I had forgotten what I had said19:43
anteayalike you say in comments, by the time it is merged your original may be correct19:43
* AJaeger waves good bye and wishes a successful gerrit move19:43
mtreinishwell on the rename comment it definitely was19:44
mtreinishI don't think anything else has really changed19:44
fungithanks AJaeger! have a good evening19:44
anteayaAJaeger: thank you, enjoy19:45
anteayamtreinish: :)19:45
jeblairback from lunch19:45
clarkbI am back from lunch as well19:45
fungijeblair: we're on track for stopping zuul in 10-15 minutes from now. nothing else pending at this point19:45
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add nodepool request framework
anteayajeblair: clarkb welcome back19:46
openstackgerritJames E. Blair proposed openstack-infra/puppet-subunit2sql: Ensure all jobs send a completion packet
jeblairmtreinish: i noticed this while reviewing your patch on the plane19:46
*** hdd has quit IRC19:47
*** jamesmcarthur has joined #openstack-infra19:48
fungi#status notice Gerrit will be offline from 20:00 to 21:00 UTC (starting 10 minutes from now) for a server replacement
openstackstatusfungi: sending notice19:49
mtreinishjeblair: hmm, sending a jobException on gearman doesn't clear it from server?19:49
* mtreinish isn't super familiar with the protocol19:49
-openstackstatus- NOTICE: Gerrit will be offline from 20:00 to 21:00 UTC (starting 10 minutes from now) for a server replacement
*** ddieterly[away] is now known as ddieterly19:50
jeblairmtreinish: it does -- the patch fixes the case where aborted jobs don't send anything19:50
*** fawadkhaliq has quit IRC19:51
jeblairmtreinish: also, i just -1'd it :)19:51
*** fawadkhaliq has joined #openstack-infra19:51
*** maishsk has joined #openstack-infra19:51
openstackstatusfungi: finished sending notice19:51
*** ajmiller has joined #openstack-infra19:52
*** ddieterly is now known as ddieterly[away]19:52
mtreinishjeblair: heh, ok19:52
jeblairmtreinish: so i should say, *ideally* the patch fixes that case :)19:52
*** cdent has quit IRC19:52
*** mylu has quit IRC19:53
*** salv-orlando has joined #openstack-infra19:54
jeblairfungi: i'll prepare to stop zuul and stage 28116619:54
*** fawadkhaliq has quit IRC19:54
fungijeblair: thanks!19:54
*** zz_dimtruck is now known as dimtruck19:54
*** ZZelle_ has joined #openstack-infra19:54
jeblair(if i can log in [ram crunch])19:54
*** fawadkhaliq has joined #openstack-infra19:55
fungias soon as you confirm zuul is dead (one way or another!) we'll press forward with step 119:55
*** ddieterly[away] is now known as ddieterly19:55
fungiyolanda: did you want to drive the gerrit server bits starting in 5 minutes?19:55
mtreinishjeblair: ah, ok with that comment I understand better now. I was just wondering about the L124 changing the sendWorkException() to a raise19:55
fungii'll get the maintenance alert queued up19:55
* clarkb is here to offer a helping hand where necessary19:55
* mordred is here to occasionally offer humorous interjections19:56
mtreinishjeblair: I was thinking that would have a similar effect by not clearing it from the server when we encounter a missing file19:56
jeblairmtreinish: ah, that's just to make the flow simpler so we don't have to separately handle that case19:56
*** maishsk has quit IRC19:56
*** ddutta has joined #openstack-infra19:56
mtreinishbecause it'll break out of the function before anything is sent19:56
yolandafungi, yes, i'm fine with driving that, with guidance of other infra-root just in case19:56
fungiyolanda: at least several of us are here to support you19:56
*** maishsk has joined #openstack-infra19:56
* mordred hands infra-root a not entirely eaten cake19:56
jeblairmtreinish: with the raise, it'll be caught by the outer execption handler that will send the work exception19:56
yolandaless pressure then :)19:56
jeblairanyone happen to be logged into zuul.o.o?19:57
yolandai am already logged to old and new gerrit19:57
* nibalizer on deck 19:57
clarkbjeblair: not me19:57
*** Nikolay_St has joined #openstack-infra19:57
nibalizerjeblair: not me19:57
fungijeblair: i was maybe an hour ago19:57
mtreinishjeblair: ah, so it will. I should have looked a bit lower down in the file :)19:57
fungibut i logged out19:57
* jeblair is looking for someone with an active shell there :|19:57
*** jamesmcarthur has quit IRC19:57
fungiyeah, i figured19:57
fungilooks like we picked the wrong moment to try to log in
anteayaoh my19:59
*** kzaitsev_mb has joined #openstack-infra19:59
*** achanda has joined #openstack-infra19:59
fungihigh system and iowait, load average of 30+19:59
*** tonytan4ever has quit IRC19:59
jeblairit's also not serving status.json, so i can't remotely save it either19:59
fungioddly paging hasn't jumped up19:59
jeblairi say we declare it a lost cause, and i'll reboot it externally after gerrit is stopped19:59
*** nadya has quit IRC20:00
fungino increase in swap utilization or spike in paging activity20:00
anteayasounds accurate20:00
yolandaok, are we good to start?20:00
fungiso this is slammed filesystem from something else presumably20:00
*** jsavak has quit IRC20:00
jeblairfungi: i suspect it's still ram related20:00
fungiyolanda: well, we were trying to get zuul pipelines saved and zuul stopped before taking gerrit down20:00
jeblairfungi: (could be related to oom killer or something)20:00
*** jsavak has joined #openstack-infra20:00
* pleia2 stands by20:01
yolandaok i'll wait until that20:01
*** trown is now known as trown|outtypewww20:01
openstackgerritHenrique Truta proposed openstack-infra/project-config: Adds ironic v3 only job
jeblairfungi: i don't recommend waiting for zuul; i recommend proceeding without it.  but i will continue to try if you like.20:01
fungii/o activity on xvda spiked, which is the rootfs20:01
fungijeblair: well, you can keep trying to get into it while we take gerrit down i guess20:02
fungii doubt it's doing much else at the moment20:02
fungithe spike on xvda has started to trail off (i think)20:02
*** Sukhdev has quit IRC20:02
fungiso maybe cpu/load/iowait will fall her ein a few20:02
fungii'll go ahead with the alert20:03
fungi#status alert Gerrit is offline until 21:00 UTC for a server replacement
openstackstatusfungi: sending alert20:03
openstackgerritHenrique Truta proposed openstack-infra/project-config: Run identity v3 only job in ironic
fungiyolanda: go ahead and stop gerrit following step 1 in the plan20:04
*** dimtruck is now known as zz_dimtruck20:04
*** openstackgerrit has quit IRC20:05
anteayawe are following this etherpad for those watching at home:
yolandaok first step done20:05
*** maishsk has quit IRC20:05
*** shamail has joined #openstack-infra20:05
-openstackstatus- NOTICE: Gerrit is offline until 21:00 UTC for a server replacement
*** ChanServ changes topic to "Gerrit is offline until 21:00 UTC for a server replacement"20:05
fungiyolanda: and per jeblair's suggestio, delete the initscripts there at step 1.120:06
fungiyolanda: then feel free to proceed with the db backup at step 220:06
funginote i updated the path to be on the cinder volume, so we can also use that to make sure we've got the right volume at step 5.420:07
sdakeyou probalaby know this but getting 503 on review.oo20:07
clarkbfungi: yolanda should we keep a local copy too so that part of the backup isn't invovled in the move?20:07
yolandaah good thought20:07
anteayasdake: yes:
*** bkeller` has quit IRC20:07
yolandabackup is processing20:07
clarkb(I don't expect there to be problems moving volumes but I am also paranoid)20:07
anteayasdake: we are replacing the server20:07
*** jamesmcarthur has joined #openstack-infra20:07
mordred(see irc channel status)20:07
sdakeok thanks ;)20:07
sdakei see now20:08
sdakeapologies for noise20:08
fungisdake: and mailing list announcements going back a month ;)20:08
* mordred hands anteaya an extra cuddly bunny20:08
sdakefungi yup i remember those actually :)20:08
*** tosky has quit IRC20:08
anteayamordred: awe, thank you20:08
jeblairsdake: we should add #openstack-kolla to the statusbot config so you'll see the announcements there20:08
fungiclarkb: yolanda: yep, lets cp that dump file into ~root/ too20:08
openstackstatusfungi: finished sending alert20:08
*** tosky has joined #openstack-infra20:09
fungii'll add a step 2.1 for that20:09
*** shamail has left #openstack-infra20:09
anteayajeblair: the patch is up to system-config to do that20:09
*** yarkot_ has quit IRC20:09
*** dirk has joined #openstack-infra20:09
*** amrith is now known as _amrith_20:09
*** bkeller` has joined #openstack-infra20:09
yolandabackup is taking its time20:10
sdakejeblair i'll do that20:10
*** maishsk has joined #openstack-infra20:10
dirkhi, does anyone know whether is still current?20:10
sdakejeblair it was atually done already i think20:10
sdakewaiting for no meetings to go on for its final ack I think20:10
dirkwhen I follow the instructions, I get a gerrit error "503 service unavailable"20:10
*** mylu has joined #openstack-infra20:10
sdakejeblair i'd point out review - but gerrit is in maintenance ;)20:10
jeblairsdake, anteaya: yep, 299009; i just approved it.20:10
*** oomichi has joined #openstack-infra20:10
clarkbdirk: gerrit is offline for a srver upgrade (see channel topic)20:10
anteayajeblair: thank you20:11
fungidirk: see channel topic in here and in most official channels20:11
fungidirk: and prior announcements to the -dev and -infra mailing lists20:11
dirkclarkb: fungi : d'oh. sorry for the noise20:11
jeblairsdake, anteaya: (i don't let little details like gerrit being offline stop me from approving patches ;)20:11
anteayajeblair: I see that20:11
yolandaok backup done20:11
anteayayou are prepared20:11
fungithree cheers for asynchronous reviewing20:11
yolandagoing to proceed with the volume umount20:11
anteayahip hip20:11
fungiyolanda: you copied the backup file to root's homedir too (new step 2.1)?20:12
fungiahh, i see you marked it out. perfect20:12
yolanda/home/gerrit2 device is busy20:12
sdakebest wishes for success on the upgrade folks20:12
sdakei guess i'll go do something else - not sure what :)20:12
fungiyolanda: can use lsof to see if someone/something has an open file handle in there20:12
yolandasomeone is tailing logs20:12
anteayasdake: some folks like bug triage20:12
yolandaok to kill it?20:12
anteayaI hear it is a popular choice during gerrit outages20:13
sdakeanteaya ya that is what i was doing with gerrit at the time :)20:13
mordredfungi, yolanda: next step is dns - should one of us do that asynchronously while the backups and stuff are happening?20:13
fungiyolanda: yeah, kill any processes you see with an open fd in there20:13
sdaketo make our plate more accurately measured for htis week ;)20:13
fungimordred: sure, you want to go ahead and do that?20:13
fungidetails are all there in the pad20:13
mordredfungi: on it20:13
yolandaumount /home/gerrit220:14
yolandaroot@review:/home/yolanda# vgchange -a n20:14
yolanda  0 logical volume(s) in volume group "main" now active20:14
fungiyolanda: step 5 next i guess, since mordred's working on 3 and 4 got moved to pre-maintenance20:15
yolandaok to proceed with volume changes?20:15
*** esikachev has quit IRC20:15
fungiyolanda: yep, should be all set20:15
yolandafungi, shall root on puppetmaster have the right creds? getting an error on nova, missing username20:16
jeblairzuul is still awol, i'm just going to reboot the server20:16
anteayajeblair: makes sense20:16
yolandai shall export OS_CLOUD ?20:16
mordredDNS done20:16
fungijeblair: yep, i see i/o still falling (and it's started responding to snmp again, at least briefly) but no guarantees this will subside in any way other than the server falling over anyway20:17
mordredyolanda: OS_CLOUD only works with openstack client20:17
jeblairfungi: or take >1hr to do so20:17
mordredyolanda: you either need to source the variable or use osc20:17
fungiyolanda: . ~root/ci-launch/openstackci-rs-nova.sh20:17
yolandaah thankx20:17
fungiyolanda: . ~root/ci-launch/cinder.sh20:17
fungiyou want both of those20:17
fungithe latter makes cinderclient work20:17
*** yamahata has quit IRC20:17
fungithough i guess you only need novaclient for this part20:18
mordredwhen we're done with this - I would like the overhaul the way we do those things20:18
*** shardy_afk has quit IRC20:18
fungiso the first one is i guess sufficient after all unless you want to use cinderclient to diagnose anything20:18
yolandaok, volumes switched20:18
anteayamordred: those things == volume commands?20:18
fungimordred: agreed, we can update to use `export OS_CLOUD=openstackci-rax` i guess20:19
*** maishsk has quit IRC20:19
fungianteaya: server actually20:19
mordredwell, any of them20:19
jeblairmordred: yeah, the files are in place now, we just need to update the docs i think20:19
mordredjeblair: yup20:19
clarkbya I use them almost exclusively now20:19
anteayafungi: thank you20:19
*** jamesmcarthur has quit IRC20:19
fungioh, i thought anteaya was asking about the specific novaclient commands translating to openstackclient20:19
clarkbthere are a lot of minor issues that can be dangerous though20:19
mordredI'd actually like for us to be able to write little ansible playbooks for these things20:19
clarkbthe most recent one I found was the paging by default so you don't actually operate on a complete set of data20:20
mordredso that we can review and the manual step is "run the playbook for the maint"20:20
fungiyolanda: should also check dmesg on the new server and the dev fs20:20
mordredclarkb: *headdesk*20:20
clarkbmordred: having done that for the elasticsearch upgrade it works slightly well20:20
fungiyolanda: i see [Mon Apr 11 20:18:47 2016]  xvdb: xvdb120:20
clarkbmordred: but then we bikeshed the details after the upgrade and never merge the playbook20:20
mordredI mean20:20
clarkbmordred: I think if we go that route we have to be more "this is what we did" and less "this is ZOMG perfect"20:20
mordredit's not a perfect process yet - merely an aspirational desire20:20
fungiyolanda: also i see a /dev/xvdb1 on the new server20:21
jeblairlet's discuss this later?20:21
mordredjeblair: ++20:21
jeblairi'm going to tell nodepool to delete all20:21
*** zz_dimtruck is now known as dimtruck20:21
fungiyolanda: pvs, vgs and lvs also show what i'd expect on the new server now20:21
yolandafungi yes, it looks good20:21
fungiyolanda: note the step we added at 5.3 per clarkb's suggestion to clear out the mountpoint before we mount there20:22
jeblairdone; nodepool is purging20:22
yolandayes, doing step by step20:22
*** achanda has quit IRC20:22
yolandait's nice to have everything documented like that20:22
fungiso we've found ;)20:22
yolandaok 5.3 done20:23
fungimakes it easier to remember what we did months down the road too20:23
fungiyolanda: yep, looks good!20:23
fungii guess go ahead and mount it20:23
yolandawe have volume with the backup on place20:23
fungiand we'll make sure we see the db backup you created earlier in there20:23
*** jsavak has quit IRC20:23
fungithere is is. perfect20:23
*** jamesmcarthur has joined #openstack-infra20:23
fungier, there it is20:23
yolandaok, the volume steps are done20:24
yolandaare we ok to start with puppetmaster?20:24
fungishould be fine to start puppeting them (undisable the new server, make sure we can sudo ssh to it by name, run ansible)20:24
*** trukise has joined #openstack-infra20:25
fungimordred: do we need to clear the ansible host cache at this point too?20:25
fungier, ansible inventory cache20:25
mordredfungi: might as well20:25
*** maishsk has joined #openstack-infra20:25
jeblairzuul change 281166 staged on zuul.o.o20:25
mordredyh, waiut20:25
mordredit will not do anything20:25
mordredthe old review,o.o still exists20:25
*** sridhar_ram has joined #openstack-infra20:25
fungioh, right, the inventory is independent of enabled/disabled status in groups20:25
*** trukise has quit IRC20:26
anteayajeblair: yay20:26
yolandaok running puppet20:26
*** openstackgerrit has joined #openstack-infra20:27
yolandafatal: [afb0cdb2-4800-4268-af83-0b2d5f68a635]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"environment": null, "facter_basename": "ansible", "facts": {"project_config_ref": "5c6ee77f47d05ae730f7335dc6a90d01e209eb99"}, "logdest": "syslog", "manifest": "/opt/system-config/production/manifests/site.pp", "puppetmaster": null, "show_diff": false, "timeout": "30m"}, "module_name": "puppet"}, "msg": "/usr/bin/timeout -20:27
yolandas 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 6", "rc": 6, "stderr": "", "stdout": "", "stdout_lines": []}20:27
yolandaso mordred, what do we do with that? we run puppet apply inside the vm?20:28
mordrederror code 6 ...20:28
clarkblook at the puppet logs in syslog20:28
clarkbthat means puppet itself failed20:28
*** julim has quit IRC20:28
jeblairApr 11 20:27:28 review puppet-user[19602]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]) Failed to call refresh: /usr/local/bin/manage-projects -v -l /var/log/manage_projects.log returned 1 instead of one of [0]20:28
jeblairApr 11 20:27:28 review puppet-user[19602]: (/Stage[main]/Openstack_project::Gerrit/Exec[manage_projects]/returns) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 29418 on  or
jeblairit tried to run m-p20:28
*** jsavak has joined #openstack-infra20:29
nibalizerits a traceback in manage-projects20:29
mordredah. yes. it's not going to be able to do that20:29
yolandagerrit is stopped of course..20:29
fungiohh, we have uid/gid mismatch20:29
nibalizerparamiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 29418 on  or
fungiyolanda: need to `sudo chown -R gerrit2:gerrit2 /home/gerrit2`20:29
clarkboh fun20:29
fungion old review, gerrit2 was 1001/1001, on new review it's 3000/300020:30
nibalizerheh, nice20:30
fungiso puppet is trying to start things as gerrit2 and getting permissions issues20:30
yolandachanging ownership20:30
nibalizerfungi: it figured out some stuff on its own :P (/Stage[main]/Openstack_project::Gerrit/File[/home/gerrit2/acls/openstack/fuel-ostf.config]/owner) owner changed '1001' to 'gerrit2'20:31
clarkbdont't forget to chown the top level dir as well20:31
funginibalizer: indeed it did, just... not enough stuff20:31
clarkber that command will do it nevermind20:31
fungiclarkb: it should get that, but worth double-checking20:31
*** jamesmcarthur has quit IRC20:32
anteayaI added the chown command to the etherpad on line 32 if someone wants to knit that into step 6 after we confirm it works for archival purposes20:32
*** amphiprions has joined #openstack-infra20:32
jeblairare we counting on the puppet-apply to do anything important for us?20:33
anteayaworks, for archival purposes20:33
clarkbjeblair: I think just to make sure puppet is happy20:33
*** jsavak has quit IRC20:33
jeblairin other words, do we need to engineer how to run it before the window is over, or wait until then?20:33
clarkband remove any delta between old and new servers if that exists20:33
fungiwell, it's had puppet disabled on it for a while20:33
reedjeblair, your comment about being able to apt-get install git-review on windows is quite premature :)20:33
jeblairreed: but funny, right? :)20:33
fungiand also it's applying the heap limit bump change for us20:33
reedtotally :)20:34
*** jsavak has joined #openstack-infra20:34
jeblairreed: that's what i was going for :)20:34
jeblairreed: (but the other bits were serious ;)20:34
mordred(I'm not sure why it's premature - you're supposed to be able to apt-get install anything in the ubuntu archive - I would expect git-review to work)20:34
reedi admit that I had to pause and think about it...20:34
fungiso yeah, i think it's necessary20:34
yolandachanging the ownership is taking long20:34
reedmordred, it's not released yet20:34
mordredreed: ah20:34
fungiyolanda: those git repos have... a lot of loose objects which are all separate files20:34
reedit will be in the next windows version20:34
mordredyolanda: yah. it' sgot to recurse the whole tree20:34
mordredwith the gits20:35
*** infra-re_ has joined #openstack-infra20:35
mordredfungi: are we managing the uid/gid of gerrit2 now?20:35
*** ddieterly is now known as ddieterly[away]20:35
clarkbmordred: no, but we shifted the base number20:35
fungimordred: we're managing the range of uids/gids for accounts created by puppet20:35
clarkbmordred: to give us 1k uids/gids for our users20:35
nibalizerwe can se the manage projects exec to be notrun. or add '1' to its acceptable output codes20:35
nibalizeri do think we want a clear puppet run before we attempt to start gerrit20:36
*** skolekonov has joined #openstack-infra20:36
clarkbnibalizer: it runs on refresh20:36
clarkbwhich it shouldn't refresh after the chown I hope20:36
*** kgiusti has left #openstack-infra20:36
*** dprince has quit IRC20:36
fungiclarkb: well, 1k is the former conflict zone. we manually pin uids/gids in the 2k range and puppet is creating non-pinned users/groups in the 3k and up range20:36
jeblairwell that's simple enough :)20:36
*** angdraug has joined #openstack-infra20:37
fungii guess we could have changed uid/gid in /etc/{group,gshadow,passwd,shadow} instead of the chown but this makes us redeployable more easily later20:37
*** skolekonov has quit IRC20:37
nibalizerclarkb: i agree20:37
jeblairare 'disable db backup', 'disable server backup', and 'disable github close pull requests cronjob' all on the old server things we should add to the list and do now?20:38
*** infra-red has quit IRC20:38
jeblairand openstackwatch20:38
yolandaok ownership change finished, going to run again20:39
*** baoli has joined #openstack-infra20:39
clarkbjeblair: with puppet disabled and volume moved I think we can go ahead iwt hthose20:39
clarkbwill let fungi and yolanda confirm though20:39
fungijeblair: yes, i think they should be safe to do now on the old server20:39
yolandayes, i think is safe20:39
fungimostly it's just more stuff to undo if we discover we need to roll this back, but i think the risk is minimal20:40
jeblairokay, all cron jobs deleted20:40
fungithanks jeblair20:40
jeblairfungi: and puppet will put them back for us20:40
jeblairif we want20:40
yolandaok puppet finished successfully now20:40
*** doug-fis_ has joined #openstack-infra20:40
fungiyolanda: excellent. that was just the chown we needed then?20:41
yolandayes, i didn't do anything else20:41
clarkbya what was happening was puppet was changing ownership on the files which triggered th eexec refresh20:41
yolandaso shall i restart the server now?20:41
fungiokay, so now we want to reboot and make sure everything comes back up correctly20:41
clarkbsince we did the chown out of band there was no exec refresh20:41
*** openstackgerrit has quit IRC20:41
jeblairpuppet did change some things back to root, which is good.  i don't see anything scary in the puppet log.20:42
jeblair(not everything in ~gerrit2 should be root)20:42
jeblairer should be gerrit220:42
fungisame (or i didn't, before i got diconnected for the reboot)20:42
anteayayolanda: can you add the chown command you ran to step 6 so the history is accurate?20:42
*** ddieterly[away] is now known as ddieterly20:42
*** hdd has joined #openstack-infra20:43
jeblairback up20:43
fungimainly i think we make sure the puppet-managed files are root-owned and gerrit2-readable so that it can't alter them for us, but we need to make sure that's encoded in puppet anywhere we find it not happening20:43
yolandaanteaya, added20:43
*** jamesmcarthur has joined #openstack-infra20:43
anteayayolanda: thank you20:44
fungigerrit seems to be running20:44
yolandaok, gerrit and gerritbot are running20:44
anteayagerrit renders for me20:44
fungiyay, almost done!20:44
jeblairshall i restart zuul?20:44
fungiyep, now's as good a time as any20:44
yolandaso there is a manual apply of a change pending, jeblair, are you dealing with that?20:45
jeblairyolanda: its done20:45
fungihe already staged it earlier20:45
*** doug-fis_ has quit IRC20:45
jeblairmy gertty seems happy20:45
yolandaso what's pending ? a matter of checking keys in zuul, git mirrors, etc?20:45
jeblairzuul is starting20:46
fungigerritbot looks unhappy20:46
*** maishsk has quit IRC20:46
anteayayolanda: yes I guess that is step 820:46
fungioh, it's getting data now20:46
yolandayep, gerritbot log shows fine20:46
mordredI have rechecked an orphaned o/r change20:47
mordredand it has been picked up20:47
fungii was seeing it spam tracebacks up until a moment ago20:47
*** abregman has quit IRC20:47
clarkb reports the heap size is set correctly20:47
fungiAttributeError: 'GerritBot' object has no attribute 'reconnect'20:47
clarkbyou have t oexpand the details area20:47
fungiwondering if we're on a different ircpython version20:47
mordredo good20:47
mordredwhat's the old review.o.o ip?20:48
*** hdd has quit IRC20:48
anteayaI logged into via chrome (which I was not logged in with before) and logging in worked for me20:48
jeblairmordred: it's in the etherpad
fungiyeah, old review.o.o has irc (13.1)20:48
pleia2mordred: listed at
anteayaclarkb: yay20:48
funginew review has irc (14.1)20:48
fungiwondering if something's different there20:48
*** maishsk has joined #openstack-infra20:48
mordredI think we should downgrade old-review to 13.1 for now20:48
yolandai tried a gerrit show-queue, shows 7762 tasks now20:48
nibalizero pleia2 beat me to it20:48
mordredtill we figure it out20:49
yolandagoing to check logs as well20:49
clarkbmordred: you mean new review to 13.1?20:49
yolandareplication tasks are shrinking so it looks ok20:49
fungiyolanda: i expect most of the gerrit task queue is replicating into /var/lib/git20:49
fungior /opt/git or wherever we ended up with that20:50
fungii expect it should go quickly since it's local20:50
yolandayep, there are tasks for local replication, and some others for mirrors20:50
mordred is the pip installed differences between the two20:51
*** esikachev has joined #openstack-infra20:51
*** jsavak has quit IRC20:51
fungiit seems to be listening to the event stream fine, but not connecting to freenode20:51
fungiso i strongly suspect it's something different in the irc module20:51
*** jsavak has joined #openstack-infra20:51
*** jamesmcarthur has quit IRC20:52
fungiwe were running a non-release version of gerritbot too20:52
mordredwow. something caused +d2to1==0.2.12.post120:52
fungion the old server20:52
*** doug-fish has joined #openstack-infra20:52
mordred to be installed20:52
* mordred needs to track that down20:52
fungimordred: i bet it's gerritbot 0.2.0 being ancient20:53
mordredI agree20:53
fungihow about i upgrade gerritbot to master for the short term and get ready to tag a 0.3.020:53
anteayafungi: I can live with that20:53
*** jamesmcarthur has joined #openstack-infra20:53
mordredfungi: ++20:53
fungiyeah, last commit on master is jhesketh's "Fix reconnection method"20:54
*** openstackgerrit has joined #openstack-infra20:55
fungiokay, gerritbot 0.2.1.dev11 is installed on new review, same as was on old, and i've restarted it20:55
fungiand it's joining channels now20:55
fungiso that seems to have been it20:55
anteayad2to1 looks like it is a packageing tool20:55
clarkbanteaya: ya its the precursor to pbr20:56
mordredit sohuldn't really show up anywhere we do anything20:56
*** bpokorny_ has joined #openstack-infra20:56
clarkbthe java melody data looks good so far20:56
anteayamakes sense20:56
clarkbwe have spiked to 9GB heap used20:56
anteayaclarkb: yay!20:56
fungiopenstackgerrit is in here now so i think we're good on that front20:57
yolandagerrit log shows some connection reset by peer errors20:57
clarkbI expect that to steadily climb past where we were at 12GB earlier20:57
*** doug-fish has quit IRC20:57
mordredjeblair: you did the nodepooling earlier, yeah?20:57
*** doug-fish has joined #openstack-infra20:57
mordredjeblair: we seem to not have new nodes yet20:57
fungisounded like he cleared all old nodes from it. may need a restart to find gearman again20:58
mordredah. maybe that's it?20:58
yolanda[2016-04-11 20:58:14,532] WARN : Exception caught20:58
yolandajava.lang.IllegalStateException: Unable to negotiate key exchange for kex algorithms (client:,diffie-hellman-group-exchange-sha256 / server: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1)20:58
fungiyolanda: i'm willing to bet that's benign. looks like someone tried to connect with a key exchange not supported by mina-sshd20:59
clarkbfungi: mordred you should be able to check the nodepool log to see if it is connected to gearman or not20:59
*** bpokorny has quit IRC20:59
yolandafungi yes, i tried clones, reviews, etc... and everything looks ok20:59
clarkbhrm ya it seems ot be allocated not much21:00
yolandahowever the same error repeats for several times, it may be same one retrying21:00
clarkbwhich should indicate no load from gearman21:00
*** Sukhdev has joined #openstack-infra21:00
*** trukise has joined #openstack-infra21:00
*** trukise has quit IRC21:01
anteayaare we out of maintenance yet?21:01
jeblairyeah, it probably died with zuul21:01
jeblairthe connection that is21:01
jeblairi'll restart nodepool21:01
anteayaor do we have more steps to complete?21:01
clarkbanteaya: need to get nodepool making nodes again but after that I think we are mostly good?21:02
fungiit's probably safe to clear the alert and let people start assuming gerrit's working (though nodepool's still on the blink for a bit maybe)21:02
clarkbya, everything should queue up and get served as able21:02
anteayayolanda: nice work on this21:02
*** ssancheztrujillo has joined #openstack-infra21:02
anteayayolanda: great prep21:02
*** jsavak has quit IRC21:03
yolandaanteaya, kudos to all team, the etherpad was greatly documented21:03
fungi#alert ok Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in
*** rossella_s has quit IRC21:03
anteayayolanda: agreed21:03
fungi#status ok Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in
openstackstatusfungi: sending ok21:03
*** eharney has quit IRC21:03
*** weshay has quit IRC21:03
* fungi grumbles at his inept fingers21:03
*** rossella_s has joined #openstack-infra21:03
yolandais great when things go so smoothly21:03
anteayaindeed it is21:03
anteayayolanda: you want to draft an update to the mailing list?21:04
fungithanks yolanda (and everyone who helped)! excellently planned and executed21:04
yolandaanteaya, shall we send that now or we wait for zuul/nodepool to complete?21:04
anteayayolanda: I think you could create a draft in an etherpad21:04
*** icey has quit IRC21:04
anteayaand then we can send once zuul/nodepool are happy again21:05
jeblairi think the mergers may be stuck21:05
fungi2016-04-11 21:02:02,494 DEBUG nodepool.NodePool: Adding gearman server zuul.openstack.org_473021:05
jeblairi think nodepool was probably fine21:05
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"21:05
-openstackstatus- NOTICE: Gerrit move maintenance completed successfully; note that DNS has been updated to new IP addresses as indicated in
yolandai'll create the etherpad and leave for some other infra-root to send21:05
anteayayolanda: sounds good21:05
clarkbjeblair: oh! that would explain why gearman jobs aren't ending up getting queued21:06
clarkbjeblair: need help restarting those?21:06
fungijeblair: oh, yep, no job work requests means nodepool sees no demand21:06
jeblairclarkb: i'm going to ansible them to double check and if so, restart them21:06
fungilooking at logs on the mergers21:06
clarkbjeblair: ok, question about nodepool restart, did you make sure the older version with working tripleo was installed?21:06
*** AJaeger has quit IRC21:07
fungilast log entry on zm01 is from 19:4421:07
jeblairclarkb: no21:07
clarkbwe will probably need to restart it again :( mordred jeblair can we revert that change and merge again when it is expected to work?21:07
jeblairfungi: yeah, they all say that; i will restart them all, but only after we ensure we're running the right nodepool21:07
jeblairclarkb: which change should we be running?21:07
*** AJaeger has joined #openstack-infra21:08
clarkbjeblair: afdd58c10a8b85ea1466d32c3a29c0d7a1b0aea921:08
openstackstatusfungi: finished sending ok21:08
clarkbjeblair: the one after that commit is the use shade for all interactions change21:09
*** regXboi has quit IRC21:09
jeblairclarkb: oh that, i thought we had another thing on top or something21:09
jeblairrestarting nodepool with that21:09
clarkbjeblair: it iddn't work because the assumption about using the first nic was not valid21:09
mordredclarkb: well, I'm hoping to be able to merge the fixes today. however, I'd also be fine with merging a revert and then doing a re-revert21:09
clarkbyou have to pick the nic that has an external gateway21:09
yolandainfra-root, anteaya,
mordredthe shade fixes did take longer than I was originally anticipating21:09
jeblairclarkb, mordred: yeah, let's either merge the fixes or a revert today21:09
*** fawadkhaliq has quit IRC21:10
mordredclarkb: you have to pick a nic on a network that has a subnet that has an external gateway21:10
mordredjeblair: ++21:10
*** fawadkhaliq has joined #openstack-infra21:10
jeblairrestarting mergers now21:10
jeblairand so many jobs running now21:10
mordredjeblair: fixes are ready - need to push a response to a Shrews review to the stack as soon as the o/r change lands21:10
anteayayolanda: looks great to me21:10
jeblairand so many nodes launching now21:10
anteayayolanda: thank you21:10
*** ddutta has quit IRC21:11
yolandajeblair, mordred, so if everything is ok, i can just send the email now while i'm connected21:11
jeblair291 nodes launching21:11
*** jamesmcarthur has quit IRC21:11
*** fawadkhaliq has quit IRC21:11
anteayayolanda: let's not tempt sending an okay before we are sure21:11
*** fawadkhaliq has joined #openstack-infra21:12
anteayayolanda: but thanks for all your work, and I appreciate it is late for you21:12
clarkbmelody continues to look good21:12
clarkbreally low http mean times21:12
yolandaanteaya, thanks. Yep, a bit late , but no problem, it's good to have flexible hours21:13
anteayayolanda: indeed21:13
yolandaok leaving now, it has been nice, thanks for all the help21:13
fungihave a good evening yolanda!21:13
mordredthanks yolanda !21:13
fungimergers definitely seem active again21:14
mordredclarkb, jeblair: however, I don't expect substantial changes to the stack between now and then - so if y'all want to start slogging through
*** ldnunes has quit IRC21:15
mordredand also
*** maishsk has quit IRC21:15
anteayattx: on the newton release schedule: it appears there is a one week hiatus between ptl self-nomination and the ptl election, is that intentional?21:15
mordredwhich should be safe to land now21:15
*** ajmiller has quit IRC21:16
*** ajmiller has joined #openstack-infra21:16
clarkbmordred: has WIP in title21:19
clarkbignore for now?21:19
*** maishsk has joined #openstack-infra21:19
mordredclarkb: yes21:20
openstackgerritMerged openstack-infra/system-config: Move statusbot and eavesdrop bot for kolla
mordredclarkb: it's there to remind me to actually do that21:20
mordredit's purely a debug log message noise issue - no substantive things21:20
clarkbyolanda: the etherpad message lgtm21:21
mordredbasically the debug logging gets REALLY happy to tell you that it's skipping network investigation because you told it to21:21
*** ssancheztrujillo has quit IRC21:21
fungilooks like all rare node types are registered in zuul's gearman, and not seeing any NOT_REGISTERED failures on the zuul status page21:22
fungigonna take a break to make dinner, then work up a gerritbot release tag21:23
clarkbmordred: comments in say done but that is the latest patchset21:24
clarkbwhere are the things being addressed?21:24
anteayafungi: enjoy dinner21:24
clarkbwe haven't sent the all clear email yet have we? I haven't seen it21:25
clarkbwas someone going to do that?21:25
anteayaclarkb: that was yolanda's email21:25
anteayaand no it hasnt' been sent21:25
mordredclarkb: I haven't pushed the next patchset yet because it'll fail the nodepool-shade-src test until the openstack requirements patch lands (it's in gate queue)21:25
anteayashe left it for someone to send once zuul/nodepool was happy21:25
anteaya for others to review21:26
mordredclarkb: but I have that fixed locally- will push as soon as we won't be wasting the 100 nodes21:26
jeblairoh, i think it's been happy for a while21:26
*** fawadkhaliq has quit IRC21:26
clarkbmordred: ok21:26
*** fawadkhaliq has joined #openstack-infra21:26
mordredclarkb: (also, it's possible you'll find something and I can bundle that in one set of updates)21:26
anteayawho would like to send yolanda's all clear email?21:26
tsufiev Folks, how I can draw zuul attention again to patches that were in gate queue when it was shut off?21:26
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** jamesmcarthur has joined #openstack-infra21:27
*** icey has joined #openstack-infra21:27
tsufievNamely and
*** |-paul-| has joined #openstack-infra21:28
*** ddutta has joined #openstack-infra21:28
mordredtsufiev: just leave a recheck comment21:28
fungitsufiev: leave a comment starting with the word "recheck" (no quotes) since they don't currently have any verified score, and as soon as they get passed by check jobs they'll be automatically enqueued into the gate pipeline21:28
*** kzaitsev_mb has quit IRC21:28
openstackgerritMerged openstack-infra/system-config: Tell os-client-config that tripleo doesn't have neutron
tsufievThanks, it helped!21:29
fungiunfortunately zuul was in a sad load situation just as we were starting our maintenance, and were unable to ssh into it to save its queue states21:29
fungianteaya: clarkb: jeblair: yep, i didn't know if anyone was still making sure things seemed stable, but happy to send the follow-up maintenance e-mail in a bit unless someone else is already queuing up what yolanda put in that pad21:30
anteayafungi: I was not, its all yours21:30
*** thiagop has quit IRC21:30
anteayafungi: thank you21:30
fungiit could probably stand an extra paragraph about the fact that we lost changes from the check/gate pipelines21:30
anteayagood point21:31
fungiso that people with tsufiev's problem don't all have to come ask us about it21:31
openstackgerritSpencer Krum proposed openstack-infra/system-config: Run less parallelism in ansible
*** kzaitsev_mb has joined #openstack-infra21:31
*** burgerk_ has joined #openstack-infra21:31
*** infra-re_ has quit IRC21:31
nibalizerso if we just run puppet apply in now, we can turn off all the apache/puppetmaster stuff ya?21:31
anteayado you think it should be a reply to either one of these posts? or
nibalizershould give us back some ram on the puppetmaster21:32
*** oanson has quit IRC21:32
clarkbnibalizer: yes, except that we still need the boot with volume + run puppet apply stuff to ahppen21:32
mordrednibalizer: yes. I htink we're safe to stop running puppetmaster on puppetmaster21:32
mordredclarkb: yah. but puppetmaster itself doesn't help with that21:32
clarkbbecause right now the only way to boot and attach a volume is with the master21:32
clarkbmordred: it does in that its the only way it works today21:32
mordredwait - what?21:32
clarkbmordred: because I wrote a change to make it work21:32
nibalizeroh we have two launch node paths right now?21:32
clarkbthen it got ignored while we merge conflicted it very badly21:32
mordredclarkb: I thought I'd updated to include your change21:33
clarkbmordred: oh? I never saw that change21:33
mordredyah. I'm pretty sure old no longer exists21:33
nibalizeri dont see 'apply' appearing anywhere in launch_node21:33
clarkbmordred: it doens't I check out my change from gerrit21:33
clarkbwhich was unmerged last time I looked21:33
mordredcan you point me to your change?21:33
*** burgerk has quit IRC21:34
mordrednibalizer: also:
*** burgerk_ has quit IRC21:35
mordredclarkb: I'll get that ported in to shade-launch-node21:36
clarkbmordred: ok21:36
mordredand then make patches to make the move final. I could have _sworn_ I already had that patchup, but I do not see it in gerrit, so clearly I'm wrong21:36
clarkbmordred: it does need to lose the ability to set the volume device because apparently that is going away in nova, except its the only way to get vexxhost to work21:38
clarkbso both options are bad21:38
*** mylu has quit IRC21:38
*** njohnston has left #openstack-infra21:39
*** ZZelle_ has quit IRC21:39
fungianteaya: i probably won't go to the trouble of digging out the message-id header for one of those but i'll follow up with an re: subject line for one of them21:39
anteayafungi: wonderful, thanks21:39
anteayahelps me when I go looking for it in archives later21:39
*** pradk has quit IRC21:40
clarkbactually its already gone away in nova I think, which is why it iddn't work on osic21:40
*** _ari_ has quit IRC21:40
*** vgridnev has quit IRC21:41
*** mylu has joined #openstack-infra21:43
*** icey has quit IRC21:43
*** doug-fish has quit IRC21:45
*** mrmartin has quit IRC21:45
*** bkeller` has quit IRC21:46
*** kzaitsev_mb has quit IRC21:46
*** kzaitsev_mb has joined #openstack-infra21:47
*** ddieterly is now known as ddieterly[away]21:48
*** matt-borland has quit IRC21:48
*** rockyg has joined #openstack-infra21:48
*** icey has joined #openstack-infra21:48
*** mriedem is now known as mriedem_away21:48
*** edmondsw has quit IRC21:48
*** tiswanso has quit IRC21:49
*** bkeller` has joined #openstack-infra21:50
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework
*** bpokorny_ has quit IRC21:51
*** amitgandhinz has quit IRC21:52
*** bpokorny has joined #openstack-infra21:52
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework
mordredclarkb: so, current shade uses block_device_mapping_v2 to do the attach at boot time21:56
*** amitgandhinz has joined #openstack-infra21:56
mordredclarkb: which I'm updating your patch to do21:56
clarkbyou can't reliably map them though21:56
clarkbthats the problem, on vexxhost it is the only way to avoid conflicting with the config drive, on osic it doens't work at all21:57
clarkband apparently that is because nova decided to stop supporting that21:57
*** amitgandhinz has quit IRC21:58
* morgan blinks.21:58
*** piet has quit IRC21:59
*** bkeller` has quit IRC22:00
morganclarkb: it makes me sad we have more implementation details leaking through22:00
mordredclarkb: so - lemme cover the non-vexxhost case first22:02
*** dimtruck is now known as zz_dimtruck22:02
mordredand then figure out the vexxhost case22:02
clarkbmordred: ok the non vexxhost case is just attach the volume without telling nova/cinder where to attach it22:02
clarkbmordred: also did find one thing about the nat networks change22:02
clarkbwill post that momentarily22:03
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add support to shade-launch-node for cinder attach
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add support to launch-node for cinder attach
mordredthere is no shade-launch-node anymore :)22:03
clarkbmordred: comments posted22:04
mordredclarkb: thanks!22:04
*** ociuhandu has joined #openstack-infra22:04
*** knikolla has quit IRC22:04
*** lakshmiS has quit IRC22:05
mordredclarkb: so - that issue you comment in gets fixed in a follow on patch22:05
mordredclarkb: the "if len(ports) AND not self._nat_destination" was really terrible logic22:05
mordredlemme point to it22:05
*** jamesmcarthur has quit IRC22:07
*** ddutta has quit IRC22:07
*** dtardivel has quit IRC22:08
*** markus_z has quit IRC22:08
* mordred is starting to think it might be easier to review this as one squashed change22:08
morganmordred: in rare cases...22:08
*** ihrachys has quit IRC22:08
*** aysyd has quit IRC22:09
*** mixos has quit IRC22:09
mordredclarkb: oh - you know what else? shade may not have a workaround for the vexxhost case because vexxhost didnt' have cinder back when I wrote most of that22:09
clarkbmordred: one thing on that got through rebase that shouldn't22:10
*** claudiub has quit IRC22:10
clarkbmordred: ya I think the fix for vexxhost is to figure out why libvirt and nova and cinder are not cooperating22:10
mordredclarkb: I'll take a todo list item to make sure shade works with vexxhost22:10
clarkbmordred: I filed a bug with nova and that seemed to dead end with "what version of libvirt are you running"22:10
mordredclarkb: yah, I mean- ultimately having the cloud not derp is important22:11
clarkbasked mnaser to update that but I don't think that ever happened22:11
morgan#TODO: Why...why...why can't we have interoperable clouds.22:11
* mordred hands morgan a potbellied pig22:11
morgannot that it's wrong to make a selection *if* the software supports it22:11
clarkbmordred: I think the possible alternative here is to use the /dev/by-id path22:11
morganmordred: oooh for roasting?!22:11
clarkbmordred: it uses some prefix of the cinder voilume uuid there iirc22:11
mordredclarkb: yes. although we need to see if that works at rax yet22:11
clarkbah yup22:11
clarkbit wasn't working on review.o.o earlier today :(22:11
mordredclarkb: you can actually do /dev/by-name22:11
mordredclarkb: on not-rax22:12
clarkbor to have vexxhost switch to using iso config drives22:12
clarkbso they show up under a different /dev prefix22:12
*** esikachev has quit IRC22:13
fungimaintenance conclusion e-mail has been sent22:13
clarkbmordred: I do think that can be squashed into the base change22:14
clarkbthats basically a new patchset22:14
clarkbnot a new logical change22:14
anteayafungi: thank you22:16
*** tpsilva has quit IRC22:17
*** maishsk has quit IRC22:17
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add support to launch-node for cinder attach
mordredclarkb: all fixed22:18
*** Guest5162 has quit IRC22:18
mordredclarkb: I'd also dropped the keep argument22:18
mordredclarkb: (this is what I get from trying to save typing by cutting and pasting between files)22:18
*** kzaitsev_mb has quit IRC22:19
*** maishsk has joined #openstack-infra22:20
mordredclarkb: k. just tried on the rebasing. unfortunately stuff moves in between - I could do it, but it might take a few hours22:21
mordredotoh - o/r change moved from check to gate - so only one more hour until we can recheck all of these22:21
clarkbjust changing the order and s/pick/squash/ is too conflicty?22:22
*** kzaitsev_mb has joined #openstack-infra22:22
mordredI tried a few different options for doing tat22:22
mordredjust out of curiosity - I'm going to see what the whole thing as one patch looks like22:22
clarkbmordred: also it might be nice to have 3 functions. One find external networks, one find internal networks, and one find nat networks22:23
*** jamesmcarthur has joined #openstack-infra22:23
clarkbto avoid the multipage function def22:23
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Switch gate-neutron-dsvm-api to be invoked via tempest plugin framework
openstackgerritMonty Taylor proposed openstack-infra/shade: SAKE OF ARGUMENT - Fix NAT processing
mordredclarkb: I'm not sure whether that ^^ is more readable22:25
mordredclarkb: but that's the whole thing squashed22:25
mordredzuul seems to be having a fun time processing its queue22:26
mordredQueue lengths: 3037 events, 138 results.22:27
*** gfidente has quit IRC22:27
clarkbmordred: queues have fallen, I think it just takes time to load up all the merge events and process them when we merge22:30
mordredah - nod22:31
*** mixos has joined #openstack-infra22:31
*** mtanino_ has quit IRC22:32
*** mixos has quit IRC22:32
*** thorst has quit IRC22:33
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Revert "Make gate-neutron-dsvm-api job non-voting"
*** thorst has joined #openstack-infra22:33
*** markvoelker has quit IRC22:33
openstackgerritMerged openstack/os-testr: Remove openstack-common.conf
*** mixos has joined #openstack-infra22:34
*** mixos has quit IRC22:34
armaxAJaeger: what did I miss on
armaxanteaya: ^22:36
*** asettle has joined #openstack-infra22:36
*** ddieterly[away] is now known as ddieterly22:37
*** thorst has quit IRC22:38
*** kzaitsev_mb has quit IRC22:38
*** kzaitsev_mb has joined #openstack-infra22:40
*** jamesmcarthur has quit IRC22:41
*** mixos has joined #openstack-infra22:41
*** mixos has quit IRC22:44
clarkbcacti and melody continue to look good22:46
*** jamesmcarthur has joined #openstack-infra22:46
*** mylu has quit IRC22:47
*** |-paul-| has quit IRC22:47
clarkbjhesketh: good morning22:49
*** mylu has joined #openstack-infra22:49
*** thorst has joined #openstack-infra22:50
*** fawadkhaliq has quit IRC22:51
jheskethHow'd the gerrit change go?22:51
clarkbpretty well, only had one hickup where the gerrit users uid changed between hosts, but a recursive chown sorted that out22:51
*** fawadkhaliq has joined #openstack-infra22:51
anteayajhesketh: morning22:52
mordredclarkb: do you have an opinion on one patch or patch series?22:52
mordredmorning jhesketh !22:52
clarkbmordred: I haven't gotten far enough in the stack to know yet22:52
jheskethNice stuff :-)22:53
*** sdake_ has joined #openstack-infra22:53
*** mylu has quit IRC22:53
anteayaarmax: what do you mean, what did you miss?22:54
armaxon the review22:55
anteayaarmax: what do you expect to see that you are not seeing?22:55
*** mylu has joined #openstack-infra22:55
clarkbmordred: I htink I do like having utility methods in first, then add in gateway subnet detection for auto port assignment22:55
anteayaarmax: yes, what about the review?22:55
clarkbmordred: mostly because then we can test the utiltiies independent of the new features22:55
*** sdake has quit IRC22:55
armaxAJaeger says “As discussed on IRC: Now is fine to merge"22:55
*** ddieterly is now known as ddieterly[away]22:55
armaxI didn’t recall having conversation on review
*** ddieterly[away] is now known as ddieterly22:55
anteayaarmax: he was reviewing two patches at the same time22:56
armaxanteaya: I mean, what was the discussion about? I didn’t join the discussion22:56
armaxanteaya: ok, I figured something like that happened22:56
anteayaarmax: he +A'd that one and the reverted his +A22:56
*** Nikolay_St has quit IRC22:56
anteayaas he +A'd the wrong patch22:56
fungijhesketh: other (minor) hiccup was that we were running a newer non-release install of gerritbot with your reconnect fix. i need to tag a proper new release of gerritbot22:56
anteayaso in irc he was discussing a different patch22:56
anteayaarmax: thanks for checking though22:57
armaxanteaya: ok, all good I guess22:57
anteayaall good22:57
anteayathanks for the follow up :)22:57
*** gordc has quit IRC22:57
armaxanteaya: sure22:57
jheskethfungi: which change was that? Did it cause problems ?22:58
*** sputnik13 has quit IRC22:58
*** rbrndt has quit IRC22:59
fungijhesketh: it's tip of master. your change was fine, we just haven't released gerritbot in a couple years23:00
openstackgerritZhipeng Huang proposed openstack-infra/project-config: Add new project Coupler to openstack repo
jheskethfungi: ah cool.23:03
jheskethAnd I believe the zuul connection change was applied too? Has that been going okay?23:03
*** kzaitsev_mb has quit IRC23:04
clarkbI haven't seen zuul anything odd, but not sure how to monitor that closely23:04
*** kzaitsev_mb has joined #openstack-infra23:04
*** maishsk has quit IRC23:04
*** maishsk_ has joined #openstack-infra23:04
anteayajhesketh: 281166 was manually applied to zuul before zuul was started23:05
jheskethOkay. Guess we just keep an eye on it then23:05
*** ddieterly is now known as ddieterly[away]23:05
*** ddieterly[away] has quit IRC23:06
anteayayeah, pretty early to tell23:06
fungithe memory leak has been taking days/weeks to surface, so only time will tell23:06
*** Sukhdev has quit IRC23:07
*** dingyichen has joined #openstack-infra23:08
asselindstufft, you around? We're seeing errrors on missing mosquitto 1.2.3
asselinpulled in from
anteayaasselin: I just realized I competely forgot about the third party meeting today23:08
dstufftwhat kind of errors23:08
anteayaasselin: I was focused on the gerrit maintenance23:08
anteayaand got up from my chair and realized what I had done23:09
clarkbanteaya: google says
anteayaasselin: sorry about that23:09
anteayaclarkb: thanks23:09
asselinanteaya, I missed it too23:10
*** kzaitsev_mb has quit IRC23:10
clarkber asselin google says ^23:10
dstufftoh, yea it looks like it got deleted today23:10
asselindstufft, ok...seems the package was deleted/renamed as clarkb said23:10
*** keedya has joined #openstack-infra23:10
anteayaasselin: great minds think alike apparently23:10
dstufftmosquitto |         | remove                                                                                | 2016-04-11 20:34:2223:10
dstufftI need to implement the thing that doesn't allow re-registering a name23:12
anteayaoh my23:13
anteayathat would be painful23:13
asselindstufft, i'm surprised deleting is allowed...23:13
*** jamesmcarthur has quit IRC23:16
dstufftasselin: PyPI is a balancing act between giving power to authors to manage their projects as best they see fit, and taking power away from authors to provide more consistency to end users23:17
anteayasounds familiar23:17
*** maishsk_ has quit IRC23:19
asselinyeah....cause issue like this:
asselinanyway, understood. I found it in a snapshot, so we'll be ok.23:19
*** maishsk has joined #openstack-infra23:21
*** salv-orlando has quit IRC23:21
*** furlongm has joined #openstack-infra23:21
*** tongli_ has joined #openstack-infra23:22
*** tongli_ has quit IRC23:22
*** jamielennox|away is now known as jamielennox23:22
*** tongli_ has joined #openstack-infra23:22
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
*** yamahata has joined #openstack-infra23:24
asselindstufft, but I got your hint finally: I can take over the namespace and re-upload it :p23:24
dstufftasselin: :)23:24
* anteaya wonders how long it will be before we see a commit fly by with the title "What are you looking at?"23:25
dstufftthat really shouldn't be allowed, but it is currently23:25
*** tongli has quit IRC23:25
clarkbit is an odd switch they did here, I wonder why they felt they should do it that way rather than handing over contrl of the existing package23:26
clarkbthats pretty simple on pypi iirc, you just add a new owner user23:26
clarkbthen they remove the old owner23:26
openstackgerritMerged openstack/requirements: Updated from generate-constraints
*** jamesmcarthur has joined #openstack-infra23:26
*** kzaitsev_mb has joined #openstack-infra23:27
*** tongli_ has quit IRC23:27
dstufftclarkb: yea it's not super hard, on the other hand PyPI has a UI straight from the depths of the 90's, so I don't blame anyone for getting confused about what is possible with it :]23:28
openstackgerritMonty Taylor proposed openstack-infra/shade: Consume config values for NAT destination
openstackgerritMonty Taylor proposed openstack-infra/shade: Search subnets for gateway_ip to discover NAT dest
openstackgerritMonty Taylor proposed openstack-infra/shade: WIP Start stamping the has_service debug messages
openstackgerritMonty Taylor proposed openstack-infra/shade: Refactor guts of _find_interesting_networks
openstackgerritMonty Taylor proposed openstack-infra/shade: Refactor the port search logic
openstackgerritMonty Taylor proposed openstack-infra/shade: Add nat_destination filter to floating IP creation
openstackgerritMonty Taylor proposed openstack-infra/shade: Allow passing nat_destination to get_active_server
*** baoli has quit IRC23:30
*** baoli has joined #openstack-infra23:31
*** sdague has quit IRC23:31
*** ddutta has joined #openstack-infra23:32
openstackgerritEmilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: copy audit.log if it does exist
*** markvoelker has joined #openstack-infra23:34
*** tosky has quit IRC23:35
*** kzaitsev_mb has quit IRC23:36
*** erlon has quit IRC23:37
*** kzaitsev_mb has joined #openstack-infra23:38
*** xarses has quit IRC23:38
*** markvoelker has quit IRC23:39
*** jamesmcarthur has quit IRC23:40
*** keedya has quit IRC23:41
*** markusz has joined #openstack-infra23:41
*** mylu has quit IRC23:42
*** Jeffrey4l__ has quit IRC23:42
*** maishsk has quit IRC23:43
*** Qiming has joined #openstack-infra23:44
*** mylu has joined #openstack-infra23:44
*** kzaitsev_mb has quit IRC23:45
clarkbdid anyone #success the gerrit upgrade?23:46
clarkbwiki says now23:46
clarkb#success infra team redeployed Gerrit on a new larger server. Should serve reviews with fewer 500 errors.23:46
openstackstatusclarkb: Added success to Success page23:46
*** tongli has joined #openstack-infra23:47
jlvillaltonyb: FYI: I removed the WIP comment for:  Use a boostrap grid to list all the meetings23:48
*** mylu has quit IRC23:48
*** mylu has joined #openstack-infra23:49
*** akshai has quit IRC23:49
*** mixos has joined #openstack-infra23:49
*** baoli has quit IRC23:50
JayFTo a wiki admin; has a dead link to a "How to use this wiki" page, which seems to only have been ever populated by spammers23:50
*** baoli has joined #openstack-infra23:51
*** tongli has quit IRC23:51
fungiJayF: good find. i'll clean that up23:52
*** sdake has joined #openstack-infra23:52
* JayF trying hard and failing hard to find any nova documentation for virt driver writers23:52
openstackgerritSean McGinnis proposed openstack/requirements: Update to os-brick 1.3.0
*** sdake_ has quit IRC23:53
mordredJayF: I think the docs are "copy one of the existing drivers and make changes until it works"23:54
JayFmordred: that's what I was afraid of23:54
*** angdraug has quit IRC23:54
fungilooks like we had some "how to use this wiki" contents when we were still using moin
*** keedya has joined #openstack-infra23:56
clarkbfungi: the joys of wiki migrations23:56
clarkbI am not getting anything done, I should stop starting at a computer screen23:56
fungibut yeah, apparently the page didn't exist (according to history) when spammers attempted to redirect it to a spam page back in february23:57
*** baoli has quit IRC23:57
*** kzaitsev_mb has joined #openstack-infra23:57

Generated by 2.14.0 by Marius Gedminas - find it at!