Saturday, 2015-04-18

clarkbarmax: we have seen it for a while but just nailed own today why it is likely happening00:00
clarkbarmax: let me show you the bug00:00
armaxwhich also leads me to beleive that a) the br-ex tunnel is solid00:00
*** dannywilson has quit IRC00:00
clarkbarmax: https://bugs.launchpad.net/nova/+bug/144556900:00
openstackLaunchpad bug 1445569 in OpenStack Compute (nova) "No dhcp lease after shelve unshelve" [Undecided,New]00:00
armaxand b) the tunnel established on top of it by Neutron is solid too00:00
armaxwe’ll have to isolate the issues one by one00:01
jeblairpuppet is running00:01
armaxand as much as we can be educated, the whole troubleshooting experience is a bit of a trial and error effort00:01
armaxclarkb: having said that I wonder how easy would be to get hold of a node post-run00:01
clarkbarmax: its relatively easy00:02
clarkbarmax: just need to select a run before it ends00:02
armaxclarkb: I guess the process still requires some manual intervention?00:03
clarkbarmax: yes00:03
armaxok00:03
*** annegentle has quit IRC00:06
*** serg_melikyan has joined #openstack-infra00:07
*** openstackgerrit has quit IRC00:07
clarkbarmax: randomly scanning a q-svc log on a tempest test showing fip doesn't come up as active shows it thinking it is active http://logs.openstack.org/14/172614/19/experimental/check-tempest-dsvm-neutron-multinode-full/6e988d4/logs/screen-q-svc.txt.gz#_2015-04-17_19_10_53_56100:07
*** openstackgerrit has joined #openstack-infra00:07
*** ddieterly has joined #openstack-infra00:07
serg_melikyanThanks! Looks like out commit merged successfully, and now we need to check that everywhere we have correct links :)00:08
clarkbhttp://logs.openstack.org/14/172614/19/experimental/check-tempest-dsvm-neutron-multinode-full/6e988d4/console.html#_2015-04-17_19_40_35_727 but tempest seems to disagree?00:08
clarkbarmax: that makes me think the underlying network stuff probably is good as you said00:08
fungiserg_melikyan: it's still in the process of being applied on the gerrit server via puppet, though in theory nothing should change at this point. i'm preparing .gitreview file updates for all the affected repos now00:09
serg_melikyanfungi: Thank you so much! Would we have global announcement about all renames?00:10
fungiserg_melikyan: there will be another e-mail to the mailing lists once everything's done00:10
*** marun has quit IRC00:11
*** Krinkle|detached is now known as Krinkle00:11
jeblairfungi, clarkb: git* has finished, and there's no stackforge/rally, so that's good.  :)00:11
fungiawesome00:11
clarkbthe q-l3 log for that IP doesn't have much info00:13
clarkbjust shows up in some iptable save outputs00:13
*** emagana has quit IRC00:17
jeblairfungi, clarkb: review is done00:18
clarkbthe congress specs repo scared me but I think thats a genuine new repo00:18
jeblairansible seems to have been happy00:18
jeblair2015-04-18 00:18:58,782 p=22307 u=root |  fatal: [design-summit-prep.openstack.org] => SSH Error: Host key verification failed.00:19
jeblairwhat's that?00:19
clarkbfungi: ^00:19
fungijeblair: it's a machine waiting on someone to approve the change which adds it to site.pp00:19
fungijeblair: it's a replacement for summit.openstack.org00:19
clarkbok I have to go now, renames and db update look good00:20
jeblairclarkb: thanks!00:20
fungijeblair: apparently there is a need to have what's hosted there served under a different dns name and not to have anything served from summit.openstack.org, but we can't remove summit.openstack.org from dns while puppet still expects it to resolve to that name00:20
fungiso the idea is to replace that machine with a different one under a silly enough name that it won't cause confusion, and then add dns records for it under whatever temporary names ttx wants when it's being used for summit prep things00:21
jeblairfungi: i'm looking for a change but i don't see it00:22
fungime too. maybe it got approved and i just need to accept the ssh key now, i'll know in a sec00:23
jeblairhttps://review.openstack.org/#/c/170132/2/manifests/site.pp00:23
jeblairfungi: that one?00:23
fungiahh, yep, guess i can just accept that host key now00:23
jeblairfungi: okay, i may need you to splain using small words -- why can't that be called summit.openstack.org?00:23
*** mtanino has quit IRC00:25
fungijeblair: because it was used in the past and people may have it bookmarked or similarly just go to that dns name from memory, and it needs to have a different hostname each summit prep season (or at least that's ttx's latest thinking on the matter)00:25
jeblairfungi: okay, i'll ask ttx to explain why it's having such an identity crisis :)00:25
fungii would recommend that, yes00:25
* jeblair feels like we're making something really simple really complicated00:25
jeblairwhich, i mean, we're good at, so...00:26
jeblairplay to your strengths and all00:26
fungihah00:26
clarkbjeblair did you see my terrible infra cloud idea?00:26
*** sabeen1 has joined #openstack-infra00:27
jeblairclarkb: no00:27
clarkbput fake nodepool info on the nodes then just rm -rf /opt && reboot then run my d-g + ansible vhange to deploy cloud00:28
*** sabeen1 has quit IRC00:28
fungiso the design-summit-prep machine is apparently going to be hosting both cheddar and odsreg as an interface sandbox for ttx to figure out what he wants them to do. a while back he had us remove summit.openstack.org from dns because it's a misleading name since it's not actually hosting content for the summit itself. he wanted to continue to use the machine but we won't be able to puppet it under a00:29
funginame which isn't in dns00:29
*** tqtran has quit IRC00:29
fungii don't think he actually sees a need for it to have a different name every cycle, just it needs one more more appropriate names in dns and summit.openstack.org needs to _not_ be in dns (unless we eventually have a site for the summit under that dns name)00:30
jeblairclarkb: i like your test==deploy thinking.  :)00:30
jeblairfungi: okay, that makes sense00:30
fungijeblair: and it was easier to add a new machine and change the name in puppet than to try to rename the machine itself00:31
clarkbjeblair ya then we can treat our upstream testing as our cloud testing too00:33
clarkbwould need to make d-g accomodate that use case a bit more though00:34
clarkbnot sure its worth doing but it came up as a straightforward way we could deploy using known tools00:34
*** tiswanso has joined #openstack-infra00:35
*** flaper87 has joined #openstack-infra00:35
*** tiswanso has quit IRC00:39
*** fedexo has joined #openstack-infra00:40
*** tiswanso has joined #openstack-infra00:40
*** dboik has joined #openstack-infra00:40
*** serg_melikyan has quit IRC00:45
*** Sukhdev has quit IRC00:51
fungiokay, i'm going to send the end-of-maintenance e-mail now that i've got the .gitreview changes proposed, unless anyone else is already working on one00:53
*** Sukhdev has joined #openstack-infra00:54
*** salv-orlando has quit IRC00:58
jeblairfungi: i was not, thanks01:03
fungion it01:03
*** davideagnello has joined #openstack-infra01:03
*** rwsu has quit IRC01:05
*** davideagnello has quit IRC01:09
*** r-daneel has quit IRC01:10
*** woodster_ has quit IRC01:10
*** Swami has quit IRC01:11
*** tsg_ has quit IRC01:15
*** vilobhmm1 has quit IRC01:16
*** armax has quit IRC01:19
*** armax has joined #openstack-infra01:19
*** yamahata has joined #openstack-infra01:20
*** Sukhdev has quit IRC01:20
openstackgerritStefano Maffulli 'reed' proposed openstack-infra/system-config: mailing list for Analyst Relations Working Group  https://review.openstack.org/17399101:20
*** tonytan4ever has quit IRC01:26
*** tonytan4ever has joined #openstack-infra01:26
*** ZZelle_ has quit IRC01:27
*** achanda has quit IRC01:28
*** tonytan4ever has quit IRC01:31
*** achanda has joined #openstack-infra01:32
*** unicell has quit IRC01:34
*** yamahata has quit IRC01:36
*** yamahata has joined #openstack-infra01:36
*** Longgeek has joined #openstack-infra01:37
*** Longgeek has quit IRC01:39
*** claudiub has quit IRC01:41
*** esker has joined #openstack-infra01:45
*** achanda has quit IRC01:45
*** sdake_ has joined #openstack-infra01:46
*** sdake has quit IRC01:47
*** Longgeek has joined #openstack-infra01:48
*** Longgeek has quit IRC01:48
*** Longgeek has joined #openstack-infra01:51
*** sdake_ has quit IRC01:51
*** salv-orlando has joined #openstack-infra01:59
*** achanda has joined #openstack-infra01:59
*** salv-orlando has quit IRC02:06
*** ajmiller has joined #openstack-infra02:08
openstackgerritMerged openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding  https://review.openstack.org/16426402:09
openstackgerritMerged openstack-infra/system-config: Update production gerrit to 2.8.4.19  https://review.openstack.org/17506802:10
*** eharney has quit IRC02:11
*** MaxV has joined #openstack-infra02:11
*** MaxV has quit IRC02:15
*** zhiwei has joined #openstack-infra02:16
*** harlowja is now known as harlowja_away02:17
*** Kennan has quit IRC02:18
*** emagana has joined #openstack-infra02:22
*** rkukura has joined #openstack-infra02:23
*** luqas has joined #openstack-infra02:23
ajmillerHi there -- I'm getting 503 from Gerrit.  I've been doing reviews, was working fine, and now all of a sudden 503 errors are happening...02:23
ajmillerAnyone else who thinks this is an exciting thing to do on Friday night?02:24
fungioh, that is because i approved a change which is restarting gerrit02:24
fungisorry. it should come back up shortly02:24
fungii forgot the module would "conveniently" restart it for us02:25
ajmillerAh, cool, no problem....  Just wanted to make sure someone knew.02:25
fungiajmiller: i'll let you know when it's back up02:25
ajmillerfungi thanks02:25
fungiand i'll send a quick irc announcement02:25
fungithanks for the heads up02:25
*** zhiwei has left #openstack-infra02:26
*** stevemar has joined #openstack-infra02:26
fungi#status notice gerrit is undergoing a quick-ish restart to implement a debugging patch. should be back up in ~10 minutes. apologies for any inconvenience02:27
openstackstatusfungi: sending notice02:27
*** luqas has quit IRC02:27
-openstackstatus- NOTICE: gerrit is undergoing a quick-ish restart to implement a debugging patch. should be back up in ~10 minutes. apologies for any inconvenience02:28
openstackstatusfungi: finished sending notice02:29
*** achanda has quit IRC02:29
*** rkukura has quit IRC02:32
*** mtanino has joined #openstack-infra02:33
*** rkukura has joined #openstack-infra02:34
*** stevelle has quit IRC02:35
*** Kennan has joined #openstack-infra02:35
*** stevelle has joined #openstack-infra02:36
*** achanda has joined #openstack-infra02:36
fungiajmiller: it should be back now02:42
ajmillerfungi yup, thanks!02:42
*** woodster_ has joined #openstack-infra02:43
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Update production gerrit to 2.8.4.19"  https://review.openstack.org/17509802:45
*** davideagnello has joined #openstack-infra02:46
fungijeblair: that ^ is currently already applied and puppet agent is disabled on review.o.o and everything is working again. i'm self-approving it now so i can reenable puppet agent safely there02:46
*** ivar-laz_ has joined #openstack-infra02:46
*** ivar-lazzaro has quit IRC02:49
*** ddieterly has quit IRC02:49
*** ivar-laz_ has quit IRC02:50
*** cody-somerville has quit IRC02:50
fungibasically the reindex seems to have worked fine after puppet stopped the service and applied the upgrade, but gerrit wouldn't start once the reindex completed. once i rolled back that patch, it was able to start again02:50
fungierror message is quoted in the commit message of the revert, but the full traceback can be seen starting at 2015-04-18 02:26:24 in ~gerrit2/review_site/logs/error_log02:51
mordredclarkb: I'm in the middle of writing a rage reply to a thread about nova/neutron02:51
*** rkukura has quit IRC02:52
mordredclarkb: it was spurred by someone suggesting that people who want an option that isn't openvswitch and instead just want something like linuxbridge need to "get over it" because linuxbridge is a "nonoption"02:54
*** tiswanso has quit IRC02:59
openstackgerritMerged openstack-infra/system-config: Revert "Update production gerrit to 2.8.4.19"  https://review.openstack.org/17509803:00
*** emagana has quit IRC03:02
*** cody-somerville has joined #openstack-infra03:03
*** emagana has joined #openstack-infra03:03
*** emagana has quit IRC03:07
*** davideagnello has quit IRC03:08
*** zhiwei has joined #openstack-infra03:08
*** Kennan has quit IRC03:09
dougwigmordred: ping, you around? i wanted to chat about your ML post, and here would be quicker.03:13
*** achanda has quit IRC03:15
*** sdake_ has joined #openstack-infra03:17
*** dimsum__ has quit IRC03:18
*** baoli has joined #openstack-infra03:18
*** dhritishikhar has joined #openstack-infra03:21
dougwigmordred: nvm, sent email.  ping me, i've got the model you're asking for in production for an internal cloud.03:23
*** Kennan has joined #openstack-infra03:24
*** salv-orlando has joined #openstack-infra03:32
*** rfolco has quit IRC03:35
*** stevelle has quit IRC03:35
*** stevelle has joined #openstack-infra03:36
*** cody-somerville has quit IRC03:50
*** salv-orlando has quit IRC03:50
*** stevelle has quit IRC03:52
*** stevelle has joined #openstack-infra03:53
*** dhritishikhar_ has joined #openstack-infra03:55
*** dhritishikhar has quit IRC03:56
*** stevemar has quit IRC03:58
*** harlowja_at_home has joined #openstack-infra04:01
*** fedexo has quit IRC04:01
*** fedexo has joined #openstack-infra04:05
*** ajmiller has quit IRC04:07
*** otter768 has quit IRC04:08
*** harlowja_at_home has quit IRC04:12
rakhmerovis there anyone here?04:15
rakhmerovfrom infra team04:15
*** achanda has joined #openstack-infra04:16
*** stevemar has joined #openstack-infra04:16
*** ajmiller has joined #openstack-infra04:18
*** Krinkle is now known as Krinkle|detached04:20
*** ddieterly has joined #openstack-infra04:20
*** achanda has quit IRC04:21
*** achanda has joined #openstack-infra04:21
*** ddieterly has quit IRC04:24
*** armax has quit IRC04:25
*** sabeen1 has joined #openstack-infra04:27
*** armax has joined #openstack-infra04:31
*** Longgeek_ has joined #openstack-infra04:31
*** unicell has joined #openstack-infra04:32
*** Longgeek has quit IRC04:33
*** ajmiller_ has joined #openstack-infra04:34
*** stevelle has left #openstack-infra04:35
*** ajmiller has quit IRC04:35
*** ajmiller_ is now known as ajmiller04:36
*** prad has quit IRC04:39
*** esker has quit IRC04:42
*** vilobhmm1 has joined #openstack-infra04:59
*** mtanino has quit IRC05:02
*** pal has joined #openstack-infra05:02
*** adrian_otto has joined #openstack-infra05:05
adrian_ottogood evening. I'm perplexed by this review: https://review.openstack.org/17453805:05
adrian_ottojenkins voted -1 on it, but all tests are showing as SUCCESS05:05
adrian_ottoand it's not merging, even after 5 rechecks?05:05
adrian_ottowhat am I missing?05:06
clarkbincomoatible requirements05:07
clarkbthe comments from jenkins should explain it05:08
*** pal has quit IRC05:16
*** ddieterly has joined #openstack-infra05:21
*** ivar-lazzaro has joined #openstack-infra05:22
*** emagana has joined #openstack-infra05:25
*** ddieterly has quit IRC05:25
*** ivar-lazzaro has quit IRC05:26
*** luqas has joined #openstack-infra05:28
rakhmerovclarkb: our project (Mistral) was accepted into OpenStack space this week, what should we do to move it into openstack from stackforge? Where can we read about it?05:30
*** emagana has quit IRC05:30
clarkbyou will want to put it on the infra meeting agenda so that we know to schedule renames05:32
*** luqas has quit IRC05:32
clarkbbut also propose a change to openstack-infra/project-config to move it05:32
clarkbthat we will apply as part of the rename process05:33
rakhmerovclarkb: ok, got it. Is there a description somewhere what exactly should I propose in the change to project-config?05:34
clarkbI dont think so, but we just merged a bunch for rally and murano so you can look at those05:35
rakhmerovooh, ok05:35
rakhmerovthanks a lot05:35
*** Sukhdev has joined #openstack-infra05:52
*** dhritishikhar_ has quit IRC05:52
*** mmedvede has quit IRC05:57
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/17471606:01
*** mmedvede has joined #openstack-infra06:06
*** otter768 has joined #openstack-infra06:09
*** otter768 has quit IRC06:14
*** Sukhdev has quit IRC06:16
*** miqui_ has quit IRC06:17
*** salv-orlando has joined #openstack-infra06:18
*** esker has joined #openstack-infra06:24
*** subscope has joined #openstack-infra06:25
*** dhritishikhar has joined #openstack-infra06:25
*** yamahata has quit IRC06:25
*** esker has quit IRC06:28
*** salv-orlando has quit IRC06:29
*** teran has quit IRC06:30
*** teran has joined #openstack-infra06:30
*** vilobhmm1 has quit IRC06:32
*** salv-orlando has joined #openstack-infra06:33
*** achanda has quit IRC06:39
*** salv-orlando has quit IRC06:40
*** woodster_ has quit IRC06:40
*** hashar has joined #openstack-infra06:42
*** achanda has joined #openstack-infra06:45
*** dhritishikhar has quit IRC06:45
*** fedexo has quit IRC06:50
*** hashar has quit IRC07:06
*** msuriar has quit IRC07:12
*** msuriar has joined #openstack-infra07:12
*** salv-orlando has joined #openstack-infra07:14
*** subscope has quit IRC07:19
*** salv-orlando has quit IRC07:22
*** ddieterly has joined #openstack-infra07:22
*** ddieterly has quit IRC07:27
*** Kennan2 has joined #openstack-infra07:28
*** Kennan has quit IRC07:29
*** dhritishikhar has joined #openstack-infra07:29
*** sabeen1 has quit IRC07:31
*** Kennan has joined #openstack-infra07:33
*** Kennan2 has quit IRC07:33
*** _nadya_ has joined #openstack-infra07:33
*** dhritishikhar has quit IRC07:35
*** yamahata has joined #openstack-infra07:38
*** soren has joined #openstack-infra07:40
*** yamahata has quit IRC07:43
*** MaxV has joined #openstack-infra07:44
*** dhritishikhar has joined #openstack-infra07:45
*** zhiwei has left #openstack-infra07:45
*** tnovacik has joined #openstack-infra07:48
*** michchap has joined #openstack-infra07:53
*** dhritishikhar has quit IRC07:53
*** soren has quit IRC07:55
*** tnovacik has quit IRC07:59
*** MaxV has quit IRC08:04
*** otter768 has joined #openstack-infra08:10
*** esker has joined #openstack-infra08:13
*** otter768 has quit IRC08:14
*** _nadya_ has quit IRC08:15
*** mrmartin has joined #openstack-infra08:17
*** esker has quit IRC08:18
*** mrmartin has quit IRC08:19
*** ddieterly has joined #openstack-infra08:23
*** ddieterly has quit IRC08:28
openstackgerritMarton Kiss proposed openstack-infra/groups: Display Suite module security update  https://review.openstack.org/17511908:31
openstackgerritMerged openstack-infra/groups: Display Suite module security update  https://review.openstack.org/17511908:37
*** achanda has quit IRC08:37
*** dhritishikhar has joined #openstack-infra08:40
*** achanda has joined #openstack-infra08:41
*** luqas has joined #openstack-infra08:47
*** andreykurilin__ has joined #openstack-infra08:49
*** luqas has quit IRC08:52
*** achanda has quit IRC08:53
*** stevemar has quit IRC08:55
*** mpaolino has joined #openstack-infra08:59
openstackgerritMarton Kiss proposed openstack-infra/groups: Fix csv export notice messages on missing emails  https://review.openstack.org/17512009:00
*** teran has quit IRC09:03
*** mpaolino has quit IRC09:07
*** _nadya_ has joined #openstack-infra09:12
*** dhritishikhar has quit IRC09:19
*** dhritishikhar has joined #openstack-infra09:19
*** tnovacik has joined #openstack-infra09:22
*** hdd has joined #openstack-infra09:23
*** _nadya_ has quit IRC09:23
*** ddieterly has joined #openstack-infra09:24
*** ddieterly has quit IRC09:29
*** adrian_otto has quit IRC09:30
*** andreykurilin__ has quit IRC09:31
*** salv-orlando has joined #openstack-infra09:42
*** dhritishikhar has quit IRC09:46
*** dhritishikhar has joined #openstack-infra09:48
*** hdd has quit IRC09:49
*** salv-orlando has quit IRC10:01
*** teran has joined #openstack-infra10:04
*** salv-orlando has joined #openstack-infra10:04
*** ajmiller_ has joined #openstack-infra10:08
*** teran has quit IRC10:10
*** otter768 has joined #openstack-infra10:10
*** ajmiller has quit IRC10:12
*** otter768 has quit IRC10:15
*** ildikov has quit IRC10:22
*** hashar has joined #openstack-infra10:32
*** hashar has quit IRC10:32
*** Shrews has quit IRC10:39
*** ildikov has joined #openstack-infra10:40
*** Shrews has joined #openstack-infra10:41
*** pcaruana has joined #openstack-infra10:48
*** salv-orlando has quit IRC11:05
*** teran has joined #openstack-infra11:06
*** teran has quit IRC11:11
*** xyang1 has quit IRC11:12
*** dhritishikhar_ has joined #openstack-infra11:14
*** dhritishikhar has quit IRC11:15
*** dhritishikhar__ has joined #openstack-infra11:18
*** ddieterly has joined #openstack-infra11:20
*** dhritishikhar_ has quit IRC11:21
*** dhritishikhar__ has quit IRC11:24
*** ddieterly has quit IRC11:24
*** ddieterl_ has joined #openstack-infra11:24
*** dhritishikhar__ has joined #openstack-infra11:37
openstackgerritThomas Bechtold proposed openstack/os-testr: Fix ValueError in subunit_trace  https://review.openstack.org/17512611:40
*** esker has joined #openstack-infra11:50
*** esker has quit IRC11:55
*** dhritishikhar_ has joined #openstack-infra11:57
*** dhritishikhar__ has quit IRC12:01
*** salv-orlando has joined #openstack-infra12:05
*** teran has joined #openstack-infra12:06
*** teran has quit IRC12:11
*** otter768 has joined #openstack-infra12:11
*** hdd has joined #openstack-infra12:13
*** otter768 has quit IRC12:16
*** salv-orlando has quit IRC12:20
*** salv-orlando has joined #openstack-infra12:24
*** luqas has joined #openstack-infra12:25
*** woodster_ has joined #openstack-infra12:26
*** salv-orlando has quit IRC12:29
*** luqas has quit IRC12:29
*** mgagne has quit IRC12:30
*** e0ne has joined #openstack-infra12:31
*** ttx has quit IRC12:31
*** ttx has joined #openstack-infra12:31
*** ttx has quit IRC12:31
*** ttx has joined #openstack-infra12:32
*** Longgeek_ has quit IRC12:33
*** mgagne has joined #openstack-infra12:36
*** mgagne is now known as Guest3630412:36
*** e0ne has quit IRC12:41
*** e0ne has joined #openstack-infra12:44
*** e0ne has quit IRC12:46
*** e0ne has joined #openstack-infra12:50
*** esker has joined #openstack-infra12:51
*** baoli has quit IRC12:52
*** e0ne has quit IRC12:54
*** esker has quit IRC12:56
*** e0ne has joined #openstack-infra12:59
*** e0ne has quit IRC13:01
*** e0ne has joined #openstack-infra13:05
*** teran has joined #openstack-infra13:07
*** Longgeek has joined #openstack-infra13:08
*** e0ne has quit IRC13:09
*** teran has quit IRC13:12
*** esker has joined #openstack-infra13:12
*** e0ne has joined #openstack-infra13:13
*** e0ne has quit IRC13:15
*** tnovacik has quit IRC13:16
fungijeblair: clarkb: mordred: i screwed up. congress-specs was in the project-config change for the move but i missed it when building the commands for the maintenance. i'm going to prep a fix to clean that up13:16
*** prad has joined #openstack-infra13:16
*** e0ne has joined #openstack-infra13:19
*** ddieterl_ has quit IRC13:20
*** e0ne has quit IRC13:22
*** e0ne has joined #openstack-infra13:26
*** hdd has quit IRC13:30
*** e0ne has quit IRC13:30
*** e0ne has joined #openstack-infra13:34
fungii think https://etherpad.openstack.org/p/repo-renaming-2015-04-18-cleanup ought to do it13:36
*** ddieterly has joined #openstack-infra13:36
fungizuul is mostly idle, so i'll start quiescing it shortly and send an e-mail announcement and irc warning, shooting to do those steps from ~14:00-14:15 utc13:37
*** e0ne has quit IRC13:37
fungier, make that 15:00-15:15 utc13:38
*** e0ne has joined #openstack-infra13:42
*** e0ne has quit IRC13:43
*** yamahata has joined #openstack-infra13:45
*** cdent has joined #openstack-infra13:46
*** e0ne has joined #openstack-infra13:48
*** e0ne has quit IRC13:50
*** davideagnello has joined #openstack-infra13:53
*** e0ne has joined #openstack-infra13:54
*** e0ne has quit IRC13:57
*** salv-orlando has joined #openstack-infra13:59
fungi#status notice Gerrit will be offline between 15:00-15:30 UTC today for emergency maintenance (starting an hour from now)14:00
openstackstatusfungi: sending notice14:00
-openstackstatus- NOTICE: Gerrit will be offline between 15:00-15:30 UTC today for emergency maintenance (starting an hour from now)14:00
*** e0ne has joined #openstack-infra14:00
openstackstatusfungi: finished sending notice14:02
*** e0ne has quit IRC14:03
*** e0ne has joined #openstack-infra14:07
*** teran has joined #openstack-infra14:08
ttxfungi: ohai!14:08
fungittx: hi there14:09
fungiam i screwing up rc work? :/14:09
*** salv-orlando has quit IRC14:09
ttxno no14:09
fungiwhew14:09
ttxI just happen to be around to continue the thawing14:09
fungioh, great! well, once i get gerrit cleaned up from my screwup last night, i'm happy to help with that14:10
ttxfungi: I think we missed a few libs in the "disable requirements check job for libs"14:10
*** tnovacik has joined #openstack-infra14:10
ttxsee https://review.openstack.org/#/c/174538/ & https://review.openstack.org/#/c/174537/14:10
fungiprobably. it looked like dhellmann basically just did oslo libs and client libs14:10
ttxnot sure we actually need those, but meh14:11
*** BharatK has quit IRC14:12
*** otter768 has joined #openstack-infra14:12
fungidoesn't hurt to unbklock them. i'll work on a couple patches for that14:12
*** teran has quit IRC14:13
ttxthx!14:13
ttxI'd like to unblock as many things as possibel so that we can move fast on Monday14:14
*** salv-orlando has joined #openstack-infra14:14
fungiabsolutely14:16
*** e0ne has quit IRC14:16
*** salv-orlando has quit IRC14:17
*** otter768 has quit IRC14:17
*** e0ne has joined #openstack-infra14:20
*** e0ne has quit IRC14:22
*** bknudson has joined #openstack-infra14:24
*** dboik has quit IRC14:24
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Temporarily ignore requirements on master libs  https://review.openstack.org/17514314:25
fungittx: ^14:25
*** e0ne has joined #openstack-infra14:26
ttx+114:27
*** davideagnello has quit IRC14:28
fungithey don't seem to be failing any integration test jobs, so that ought to be sufficient to un-block them14:29
mordreddougwig: woot!14:29
fungi#status notice Gerrit will be offline between 15:00-15:30 UTC today for emergency maintenance (starting half an hour from now)14:30
openstackstatusfungi: sending notice14:30
mordredfungi: I just woke up - do you need any help?14:30
*** woodster_ has quit IRC14:30
-openstackstatus- NOTICE: Gerrit will be offline between 15:00-15:30 UTC today for emergency maintenance (starting half an hour from now)14:31
*** e0ne has quit IRC14:31
fungimordred: if you want to review the steps in https://etherpad.openstack.org/p/repo-renaming-2015-04-18-cleanup that would help14:31
fungii think i have it well in hand, but a second set of eyes can't hurt14:31
*** dboik has joined #openstack-infra14:31
openstackstatusfungi: finished sending notice14:32
mordredfungi: looking14:35
mordredfungi: yeah. looks solid to me14:36
fungiremaining jobs are on track to complete right on schedule, so as soon as zuul exits i'll stop gerrit and proceed with the steps there14:36
fungishould go quickly. it's just one project14:37
fungipretty much the duration of the lucene reindex +/- a few minutes14:37
fungii should be able to complete the other steps while it's going14:38
fungiluckily there are no changes proposed to the errant openstack/congres-specs yet14:39
ttxfungi: Can I send changes to the gate ? Or should I hold ?14:39
fungittx: in theory you can. zuul's supposed to be saving its event queue and restoring it when it starts back. but you might want to hold unless you need to jet and won't be around to do it in an hour14:40
*** ddieterly has quit IRC14:40
fungii expect it to work, but bugs do happen14:40
ttxok, holding, no hurry14:41
*** salv-orlando has joined #openstack-infra14:44
*** dhritishikhar_ has quit IRC14:46
*** e0ne has joined #openstack-infra14:51
*** e0ne has quit IRC14:53
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet OpenStack: enable Puppet 4.0 as non voting  https://review.openstack.org/17514514:53
*** salv-orlando has quit IRC14:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet OpenStack: enable Puppet 4.0 as non voting  https://review.openstack.org/17514514:55
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Puppet OpenStack: enable Puppet 4.0 as non voting  https://review.openstack.org/17514514:55
*** e0ne has joined #openstack-infra14:58
*** e0ne has quit IRC15:00
*** e0ne has joined #openstack-infra15:01
*** mpaolino has joined #openstack-infra15:02
*** e0ne has quit IRC15:04
jeblairfungi: i have just arrived15:04
jeblairfungi: also, fwiw, we all screwed up :|15:04
fungiwe have a couple straggler jobs in the post pipeline, but i'll probably just kill them15:04
*** dhritishikhar has joined #openstack-infra15:05
fungistatus Gerrit is offline for emergency maintenance, ETA 15:30 UTC to completion15:05
fungijeblair: ^ look good?15:05
jeblairfungi: yeah15:06
jeblairfungi: i'm double checking things in the etherpad15:06
fungithanks!15:06
fungi#status Gerrit is offline for emergency maintenance, ETA 15:30 UTC to completion15:07
openstackstatusfungi: unknown command15:07
fungiokay, zuul is offline, stopping gerrit now15:07
fungi#status alert Gerrit is offline for emergency maintenance, ETA 15:30 UTC to completion15:07
openstackstatusfungi: sending alert15:07
jeblairfungi, mordred: confirmed there are no watches or changes for openstack/congress-specs, so the 2 sql commands should be fine.15:07
fungithanks for double-checking15:07
fungiseems we caught it in time15:07
*** e0ne has joined #openstack-infra15:08
fungiokay, gerrit is down. starting on db edits now15:08
jeblairfungi: okay if i steal git* moves?15:09
-openstackstatus- NOTICE: Gerrit is offline for emergency maintenance, ETA 15:30 UTC to completion15:09
*** ChanServ changes topic to "Gerrit is offline for emergency maintenance, ETA 15:30 UTC to completion"15:09
*** teran has joined #openstack-infra15:09
fungifeel free--thanks!15:09
jeblairack15:09
fungilucene reindex is going now15:09
fungii'll tackle github while it runs15:10
openstackstatusfungi: finished sending alert15:11
jeblairgit* done15:11
*** andreykurilin__ has joined #openstack-infra15:11
*** e0ne has quit IRC15:12
fungigithub's done15:12
fungii'll clean up the jenkins workspaces while there's time15:12
fungiokay, as soon as the reindex completes, i'll start gerrit, zuul, and reenable puppet on the puppetmaster15:13
*** teran has quit IRC15:14
*** ajmiller_ is now known as ajmiller15:17
*** dizquierdo has joined #openstack-infra15:18
fungii've also got the .gitreview change prepped to push for it as soon as things have started back up15:18
fungiand the completion e-mail pre-written ready to send15:18
fungireindex just finished! starting gerrit now15:18
fungigerrit is up, starting zuul15:19
fungilooks like it restored its event queue correctly15:20
fungiit's starting to unwind that now15:20
fungipuppet reenabled15:21
fungi.gitreview change pushed15:21
fungi#status ok15:21
openstackstatusfungi: sending ok15:21
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/manual/ http://ci.openstack.org/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/"15:22
*** ihrachyshka has joined #openstack-infra15:23
openstackstatusfungi: finished sending ok15:24
jeblairfungi, clarkb, ttx: tomorrow i leave for nearly 2 weeks of constant travel, so my schedule is going to be sort of random.  i will probably be afk a lot.15:24
fungijeblair: thanks for the heads up--safe travelling15:24
jeblairactually, be 'nearly' i think i mean 'more than'.15:25
fungijeblair: if you haven't seen scrollback, i briefly tried out your stream-events gerrit patch last night but had to revert it. something about the signature of bouncy-castle in the war didn't validate15:25
*** ZZelle has quit IRC15:25
jeblairoh yeah i saw that15:26
*** ZZelle has joined #openstack-infra15:26
*** cody-somerville has joined #openstack-infra15:26
jeblairapparently we were able to build a working war a week ago, but for some reason, not now15:26
fungiassuming you're not around, i'll see if zaro has ideas about how to deal with that on monday15:26
jeblairyeah, i have not gotten very far on that yet, other than finding the suggestion that there may be multiple bouncycastles at play15:27
fungithat gives me an idea for how to spend the rest of my saturday15:28
jeblairbut i would not expect us to bundle any bouncycastle stuff in the war.  but maybe we are and that's the problem?  <shrug>15:28
fungii wonder if anyone around here rents actual bouncy castles?15:28
jeblairfungi: i think that's what we should use instead15:28
*** dizquierdo has quit IRC15:28
*** yamahata has quit IRC15:30
fungizuul tested and reported on the openstack/congress-specs .gitreview change, so looks like everything's in working order15:35
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add an ansible playbook for project renames  https://review.openstack.org/17514715:35
mordredfungi, jeblair: slightly late ^^15:35
fungiit's plenty early for next time!15:35
jeblairclarkb: ^15:36
jeblairneato!15:36
mordredjeblair, clarkb, fungi: that's untested as of right now - next steps are figure out the best way to test it :)15:37
mordredbut i wanted to get it up for comment/feedback15:37
*** dhritishikhar has quit IRC15:37
*** adrian_otto has joined #openstack-infra15:38
fungii would say next rename maintenance, we set the maintenance plan up to run that along with a thorough double-check of all the things we expect it to change15:38
mordredit's possible to make it a little more efficient by reordering a few of the steps and making the lucene reindex run as an async task that we check back on later15:38
mordredfungi: ++15:38
mordredfungi: I'm also going to run it in no-op mode and get the list of what ansible thinks it's going to run15:38
fungiif we pre-write our validation steps, it should go about as quickly as a usual rename maintenance15:39
fungiwe can even do that for a couple maintenances if we want additional comfort that we can rely on it to just work in the future15:39
mordredyah - I hope it winds up with renames not being a team event, but really just a boring maint task15:40
fungithough at the moment, the real time sink with these is the lucene reindex anyway15:40
fungias far as how much we could realistically shrink the maintenance window itself15:40
openstackgerritMerged openstack-infra/project-config: Temporarily ignore requirements on master libs  https://review.openstack.org/17514315:40
mordredin fact, the input vars file could be committed to git as a code review - so that it's really just running a single command and not much double-checking needed during the window15:41
fungibut yeah making it a mostly one-person operation would be great15:41
mordredfungi: yah15:41
*** ddieterly has joined #openstack-infra15:41
*** ddieterly has quit IRC15:47
*** dizquierdo has joined #openstack-infra15:48
*** dhritishikhar has joined #openstack-infra15:48
*** cody-somerville has quit IRC15:49
clarkblooks like all the fun is over15:51
clarkbmordred your example yaml needs its item lists indented more15:52
clarkbplaubook has the same problem15:52
mordredclarkb: I mean, not for functionality15:53
mordredif you prefer more indentation I can do that15:53
mordredI've been injecting less indentation recently just because - but happy to change it15:53
clarkbmordred I thought pyyaml cared about that15:53
mordrednope15:53
clarkbiirc jjb explodes on it at least15:53
mordredmaybe it used to and has been since fixed?15:54
clarkbbut jjb does crazyness in the loader too so maybe thats our bug15:54
mordrednod15:54
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add an ansible playbook for project renames  https://review.openstack.org/17514715:55
mordredk. couple of syntax checks ... that runs fine15:56
*** dhritishikhar_ has joined #openstack-infra16:01
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add an ansible playbook for project renames  https://review.openstack.org/17514716:01
mordredclarkb: http://paste.openstack.org/show/204549 is running that through ansible-playbook -C16:01
mordredfungi: ^^16:02
anteayashould a python app have a circular import? as in should file a import file b and file b import file a?16:02
anteayamy gut feeling says no16:02
clarkbanteaya your gut is correct16:03
anteayaokay thanks16:03
anteayanow I have to read both to see if I can unwind16:03
*** adrian_otto has quit IRC16:03
clarkbusually you can make a file c that both a and b import16:04
clarkbc would contain the common set of things from a and b that both need16:04
anteayathat would make sense to me16:04
*** andreykurilin__ has quit IRC16:04
*** dhritishikhar has quit IRC16:04
anteayait is an astrology app, open source in python, so yay but the first app the author wrote so hardcoded everything everywhere :(16:05
*** teran has joined #openstack-infra16:10
*** mpaolino has quit IRC16:10
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Fix metadata.json file  https://review.openstack.org/17493516:12
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Fix .gitreview file  https://review.openstack.org/17450516:12
*** otter768 has joined #openstack-infra16:13
*** teran has quit IRC16:15
*** woodster_ has joined #openstack-infra16:16
*** otter768 has quit IRC16:18
*** baoli has joined #openstack-infra16:18
*** nelsnelson has joined #openstack-infra16:22
*** baoli has quit IRC16:22
*** baoli has joined #openstack-infra16:23
*** ddieterly has joined #openstack-infra16:23
*** dhritishikhar_ has quit IRC16:32
*** hdd has joined #openstack-infra16:32
*** baoli has quit IRC16:33
*** yfried__ has joined #openstack-infra16:44
*** dizquierdo has quit IRC16:50
*** nelsnelson has quit IRC16:50
openstackgerritGal Sagie proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow  https://review.openstack.org/17515216:50
openstackgerritGal Sagie proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow  https://review.openstack.org/17515216:52
openstackgerritGal Sagie proposed openstack-infra/project-config: Enable a check job for stackforge/dragonflow  https://review.openstack.org/17515216:53
*** salv-orlando has joined #openstack-infra16:54
*** ArxCruz has joined #openstack-infra16:58
*** dhritishikhar_ has joined #openstack-infra17:00
*** salv-orlando has quit IRC17:01
*** salv-orlando has joined #openstack-infra17:06
*** teran has joined #openstack-infra17:10
*** teran has quit IRC17:15
*** dteselkin has quit IRC17:18
*** rakhmerov has quit IRC17:18
*** salv-orlando has quit IRC17:18
*** HeOS__ has quit IRC17:19
*** katyafervent has quit IRC17:19
*** tsufiev has quit IRC17:19
*** hdd has quit IRC17:20
*** asilenkov has quit IRC17:20
*** Sukhdev has joined #openstack-infra17:20
*** ArxCruz has quit IRC17:22
*** dhritishikhar_ has quit IRC17:28
*** armax has quit IRC17:31
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add utility for inspecting puppet module  https://review.openstack.org/17515617:36
openstackgerritSpencer Krum proposed openstack-infra/system-config: Use envassert for integration tests  https://review.openstack.org/17359217:37
*** _nadya_ has joined #openstack-infra17:38
*** hdd has joined #openstack-infra17:39
*** Krinkle|detached is now known as Krinkle17:43
*** hdd_ has joined #openstack-infra17:46
*** hdd has quit IRC17:46
*** Sukhdev has quit IRC17:52
*** wenlock has joined #openstack-infra17:53
*** dhritishikhar_ has joined #openstack-infra17:57
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove extra tarball lines  https://review.openstack.org/16815917:58
*** baoli has joined #openstack-infra18:06
*** yolanda has quit IRC18:06
*** dhritishikhar_ has quit IRC18:07
*** baoli has quit IRC18:09
*** teran has joined #openstack-infra18:11
*** otter768 has joined #openstack-infra18:14
*** _nadya_ has quit IRC18:15
*** teran has quit IRC18:16
*** otter768 has quit IRC18:19
*** cdent has quit IRC18:26
*** woodster_ has quit IRC18:30
*** dhritishikhar_ has joined #openstack-infra18:33
*** ddieterly has quit IRC18:35
*** sputnik13 has joined #openstack-infra18:40
*** ihrachyshka has quit IRC18:44
*** sputnik13 has quit IRC18:46
*** vilobhmm1 has joined #openstack-infra18:57
*** dhritishikhar_ has quit IRC18:57
*** baoli has joined #openstack-infra18:58
*** baoli has quit IRC19:01
*** baoli has joined #openstack-infra19:02
*** baoli has quit IRC19:02
*** sputnik13 has joined #openstack-infra19:03
openstackgerritAndreas Jaeger proposed openstack-infra/release-tools: Add script to cleanup of translations  https://review.openstack.org/17282719:04
*** achanda has joined #openstack-infra19:05
*** andreykurilin__ has joined #openstack-infra19:10
*** sc` has joined #openstack-infra19:11
*** teran has joined #openstack-infra19:12
*** dhritishikhar_ has joined #openstack-infra19:15
*** teran has quit IRC19:17
*** ArxCruz has joined #openstack-infra19:20
*** andreykurilin__ has quit IRC19:20
*** yamahata has joined #openstack-infra19:21
*** ArxCruz has quit IRC19:22
*** andreykurilin__ has joined #openstack-infra19:24
*** isviridov_away is now known as isviridov19:24
*** isviridov is now known as isviridov_away19:31
*** ddieterly has joined #openstack-infra19:36
*** vilobhmm1 has quit IRC19:37
*** salv-orlando has joined #openstack-infra19:39
*** ddieterly has quit IRC19:40
*** wenlock has quit IRC19:41
*** Longgeek has quit IRC19:43
*** achanda has quit IRC19:45
*** andreykurilin__ has quit IRC19:48
*** salv-orlando has quit IRC19:53
*** dboik has quit IRC19:57
*** dhritishikhar_ has quit IRC19:57
*** salv-orlando has joined #openstack-infra20:03
*** Krinkle is now known as Krinkle|detached20:06
*** teran has joined #openstack-infra20:13
*** woodster_ has joined #openstack-infra20:13
*** otter768 has joined #openstack-infra20:15
*** teran has quit IRC20:17
*** ddieterly has joined #openstack-infra20:19
*** otter768 has quit IRC20:20
*** salv-orlando has quit IRC20:22
*** achanda has joined #openstack-infra20:24
*** ddieterly has quit IRC20:26
*** ddieterly has joined #openstack-infra20:27
*** hdd_ has quit IRC20:42
*** hdd_ has joined #openstack-infra20:48
*** sdake_ has quit IRC20:50
*** sdake has joined #openstack-infra20:53
*** adrian_otto has joined #openstack-infra20:53
*** achanda has quit IRC21:03
*** achanda has joined #openstack-infra21:05
*** andreykurilin__ has joined #openstack-infra21:07
*** soren_ has joined #openstack-infra21:10
*** achanda has quit IRC21:12
*** teran has joined #openstack-infra21:13
*** soren_ has quit IRC21:14
*** teran has quit IRC21:18
*** hdd_ has quit IRC21:22
*** soren_ has joined #openstack-infra21:30
*** salv-orlando has joined #openstack-infra21:40
*** salv-orlando has quit IRC21:46
*** ddieterly has quit IRC21:48
*** ddieterly has joined #openstack-infra21:50
*** soren_ has quit IRC21:58
*** achanda has joined #openstack-infra22:05
*** Sukhdev has joined #openstack-infra22:08
*** stevemar has joined #openstack-infra22:13
*** teran has joined #openstack-infra22:14
*** dboik has joined #openstack-infra22:15
*** otter768 has joined #openstack-infra22:16
*** teran has quit IRC22:19
*** otter768 has quit IRC22:21
*** Sukhdev has quit IRC22:21
*** luqas has joined #openstack-infra22:29
*** luqas has quit IRC22:34
*** achanda has quit IRC22:35
*** dboik_ has joined #openstack-infra22:38
*** dboik has quit IRC22:38
*** stevemar has quit IRC22:45
*** Darkwan has quit IRC22:47
*** ajmiller has quit IRC23:14
*** teran has joined #openstack-infra23:15
*** teran has quit IRC23:20
*** andreykurilin__ has quit IRC23:28
*** achanda has joined #openstack-infra23:39
*** jeblair has quit IRC23:50
*** jeblair has joined #openstack-infra23:50
*** adrian_otto has quit IRC23:54
*** mriedem has joined #openstack-infra23:55
*** jeblair has quit IRC23:55
*** jeblair has joined #openstack-infra23:57
*** ajmiller has joined #openstack-infra23:58
*** salv-orlando has joined #openstack-infra23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!