Tuesday, 2016-03-29

*** JRobinson__ has quit IRC00:02
*** asingh has quit IRC00:02
*** asettle has joined #osic00:10
*** mdfuller has joined #osic00:13
*** rmevans has joined #osic00:22
*** rmevans has quit IRC00:38
*** ankurgupta has left #osic00:38
*** bluex has quit IRC00:38
*** bluex has joined #osic00:39
*** asingh has joined #osic00:48
*** b3rnard0 is now known as b3rnard0_away00:50
*** bluex has quit IRC00:52
*** bluex has joined #osic00:53
*** asettle has quit IRC00:54
*** jorge_munoz has quit IRC00:54
*** JRobinson__ has joined #osic00:55
*** adrian_otto has quit IRC00:55
*** bluex has quit IRC01:02
*** asettle has joined #osic01:04
*** bluex has joined #osic01:05
*** bluex has quit IRC01:08
*** asettle has quit IRC01:08
*** bluex has joined #osic01:09
*** bluex has quit IRC01:12
*** bluex has joined #osic01:12
*** bluex has quit IRC01:16
*** asingh has quit IRC01:18
*** bluex has joined #osic01:18
*** asettle has joined #osic01:23
*** karthikp has quit IRC01:25
*** bluex has quit IRC01:33
*** bluex has joined #osic01:33
*** bluex has quit IRC01:34
*** mdfuller has quit IRC01:44
*** mdfuller has joined #osic01:46
*** woodster_ has quit IRC01:57
*** karthikp has joined #osic02:07
*** ediardo has quit IRC02:18
*** muralia_ has joined #osic02:22
*** muralia has quit IRC02:22
*** karthikp has quit IRC02:55
*** sgundur has left #osic02:57
*** adrian_otto has joined #osic03:22
*** asingh has joined #osic03:22
*** pushkaru has joined #osic03:24
*** asingh has quit IRC03:48
*** JRobinson__ is now known as JRobinson__afk03:55
*** asettle has quit IRC03:58
*** adrian_otto has quit IRC04:05
*** adrian_otto has joined #osic04:05
*** JRobinson__afk is now known as JRobinson__04:35
*** ediardo has joined #osic04:42
*** adrian_otto has quit IRC04:50
*** adrian_otto has joined #osic04:51
*** adrian_otto has quit IRC04:58
*** mdfuller has quit IRC04:59
*** ediardo has quit IRC05:10
*** asettle has joined #osic05:12
*** adrian_otto has joined #osic05:14
*** adrian_otto has quit IRC05:17
*** adrian_otto has joined #osic05:20
*** asettle has quit IRC05:39
*** adrian_otto has quit IRC05:49
*** JRobinson__ has quit IRC05:52
*** pushkaru has quit IRC05:54
*** asettle has joined #osic05:54
*** asettle has quit IRC05:59
*** adrian_otto has joined #osic06:17
*** adrian_otto has quit IRC07:18
*** asettle has joined #osic08:26
*** JRobinson__ has joined #osic08:50
*** castulo has quit IRC09:08
*** toanster has quit IRC09:09
*** spotz_zzz has quit IRC09:09
*** electrocucaracha has quit IRC09:09
*** pumaranikar has quit IRC09:09
*** toan has joined #osic09:11
*** spotz_zzz has joined #osic09:14
*** pumaranikar has joined #osic09:14
*** castulo has joined #osic09:14
*** asettle has quit IRC09:27
*** asettle has joined #osic09:52
*** asettle has quit IRC09:57
*** JRobinson__ has quit IRC10:07
*** electrocucaracha has joined #osic10:24
*** asettle has joined #osic10:56
*** asettle has joined #osic11:56
*** mdfuller has joined #osic12:06
*** sfinucan has joined #osic12:48
electrocucarachagood morning13:13
*** pushkaru has joined #osic13:18
*** ametts has joined #osic13:22
sarafrajmorning13:25
*** jthorne has joined #osic13:30
*** b3rnard0_away is now known as b3rnard013:30
*** woodster_ has joined #osic13:32
anushgood morning13:33
pdardeaugood morning13:35
*** KrishR has joined #osic13:40
*** asettle has quit IRC13:49
*** asettle has joined #osic13:55
*** inc0 has joined #osic13:55
*** sigmavirus24_awa is now known as sigmavirus2413:57
*** asettle has quit IRC14:00
*** woodburn has quit IRC14:02
*** mdfuller has quit IRC14:03
jrollmornin'14:07
hemanthmo/14:07
*** sigmavirus24 has left #osic14:08
*** muralia has joined #osic14:12
*** sgundur has joined #osic14:12
snetigood morning!!14:12
*** muralia_ has quit IRC14:14
pumaranikargood morning14:20
*** spotz_zzz is now known as spotz14:20
*** Mudpuppy has joined #osic14:24
*** asingh has joined #osic14:25
*** homerp_ has joined #osic14:37
*** jorge_munoz has joined #osic14:45
rosmaitahemanthm: meet for standup in osic-7 room14:46
*** rderose has joined #osic14:46
hemanthmrosmaita: ack14:46
*** cloudtrainme has joined #osic14:48
*** karthikp has joined #osic14:49
anushsarafraj: nova standup!!14:49
*** karthikp has quit IRC14:53
*** karthikp has joined #osic14:53
sarafrajanush: yes14:53
sarafrajanush: oh we might be in different room14:54
sarafrajanush: let me send you that14:54
anushSarafraj don't see anyone in Vidyo. Don't know if I'm having problems14:54
anushHa. Ok14:54
*** mdfuller has joined #osic14:55
sarafrajanush: johnthetubaguy is giving us deep dive on nova if you want to join14:57
rosmaitahttps://vc.rackspace.com/flex.html?roomdirect.html&key=MG48bWbR6mEVUNqQVK3dLG0AUR814:57
anushI will take 5-10 minutes.14:58
*** cloudtrainme has quit IRC14:59
*** cloudtrainme has joined #osic15:01
*** siva_krishnan has left #osic15:16
*** siva_krishnan has joined #osic15:21
*** lin_yang has quit IRC15:43
*** lin_yang has joined #osic15:43
dasmspotz: last time we were talking about Polish Easter Monday tradition.15:46
dasmspotz: http://wivb.com/2016/03/26/we-asked-you-answered-what-is-dyngus-day/15:46
hockeynutdasm I grew up in Buffalo NY so I know all about Dyngus Day15:46
dasmhockeynut: yay. Dyngus FTW!15:47
hockeynutin fact as a kid I participated in chasing girls around trying to tap them with pussy willows15:47
spotzdasm That talks about eating not buckets of water:)15:47
spotzockeynut that's a whole new view of you:)15:47
*** karthikp has quit IRC15:47
dasmspotz: see what hockeynut wrote :D15:48
spotzhockeynut ^^15:48
hockeynut<snicker>15:48
dasmspotz: it's mutated. from just a little bit of water, to buckets of water ;)15:48
dasmspotz: https://en.wikipedia.org/wiki/Easter_Monday#Eastern_and_Central_Europe15:49
dasm" All countries practice a unique custom on this day. In Poland, Hungary, Slovakia, and the Czech Republic[3] traditionally, early in the morning boys awake girls by pouring a bucket of water on their head and striking them about the legs with long thin twigs or switches made from willow, birch or decorated tree branches"15:50
dasm*boys wake girls by pouring a bucket of water on their head*15:50
dasm\o/15:50
dasm*awake15:50
spotzdasm Uh huh and where's the girls' revenge holiday?:)15:50
dasmspotz: usually it's Tuesday.15:50
dasmso... today ;)15:51
spotzhehehe15:51
*** adrian_otto has joined #osic15:55
adrian_ottoo/15:59
*** ametts has quit IRC15:59
*** dolphm_bot has joined #osic16:00
dolphm_botjorge_munoz, rderose, xek: Keystone standup!16:00
dolphm_botWhat are you working on today, and what do you need help with?16:00
lbragstadtoday I am working on capstone things and reviewing a couple keystone patches during my free cycles. I'll be out this afternoon.16:00
rderoseback in SA today, getting ready to present in Keystone IRC and working on shadow user (newton) patch16:01
pumaranikarhello johnthetubaguy , Can you please have a look at the debug patch for live migration bug https://review.openstack.org/#/c/298099/16:02
snetihi johnthetubaguy, can you please review this patch when you have time: https://review.openstack.org/#/c/290156/16:08
*** jthorne has quit IRC16:08
*** jthorne has joined #osic16:09
*** karthikp has joined #osic16:12
dolphm_botThank you, lbragstad, rderose!16:15
*** dolphm_bot has quit IRC16:15
*** onthecly has joined #osic16:16
*** jthorne has quit IRC16:16
*** jthorne has joined #osic16:17
*** nadeem has joined #osic16:18
*** nadeem has quit IRC16:19
*** nadeem has joined #osic16:19
*** jthorne has quit IRC16:22
snetihemanthm, sfinucan I have submitted couple of patches. Can you please review them.16:22
hemanthmsneti: sure, links?16:23
sfinucanSure - stick me in the reviewers list and I'll do it tomorrow afternoon16:23
snetihemanthm, sfinucan: https://review.openstack.org/#/c/297381/, https://review.openstack.org/#/c/294261/, https://review.openstack.org/#/c/297842/16:23
sfinucan(Home time for me now + sprint retro/planning tomorrow morning :()16:24
snetisure sfinucan..whenever you have time16:24
snetihemanthm, thanks!16:25
*** cloudtrainme has quit IRC16:25
*** jthorne has joined #osic16:27
*** KrishR1 has joined #osic16:32
*** KrishR has quit IRC16:33
*** raddaoui has quit IRC16:36
*** cloudtrainme has joined #osic16:40
*** jthorne has quit IRC16:41
*** jthorne has joined #osic16:42
*** homerp_ has quit IRC16:44
*** sfinucan has quit IRC16:45
*** raddaoui has joined #osic16:51
*** mdfuller has quit IRC16:52
hemanthmaunnam_: heads up, I have another patch coming up that's going to conflict with yours :)16:55
*** ametts has joined #osic16:57
*** mdfuller has joined #osic16:58
*** palendae_ has joined #osic17:06
*** spotz has quit IRC17:07
*** dragondm has quit IRC17:07
*** lin_yang has quit IRC17:07
*** blogan has quit IRC17:07
*** ametts has quit IRC17:07
*** hockeynut has quit IRC17:07
*** neillc has quit IRC17:07
*** darrenc has quit IRC17:07
*** kprabhuv has quit IRC17:07
*** aunnam_ has quit IRC17:07
*** mmotiani has quit IRC17:07
*** cloudnull has quit IRC17:07
*** palendae has quit IRC17:07
*** b3rnard0 has quit IRC17:07
*** sgundur has quit IRC17:07
*** saisriki has quit IRC17:07
*** leifz has quit IRC17:07
*** mcarden has quit IRC17:07
*** claco has quit IRC17:07
*** b3rnard0 has joined #osic17:07
*** spotz has joined #osic17:07
*** neillc has joined #osic17:07
*** dragondm has joined #osic17:07
*** mcarden_ has joined #osic17:07
*** hockeynut_afk has joined #osic17:07
*** saisrikiran has joined #osic17:07
*** kprabhuv- has joined #osic17:07
*** ametts_ has joined #osic17:07
*** saisrikiran is now known as saisriki17:07
*** ankurgupta has joined #osic17:07
*** hockeynut_afk is now known as hockeynut17:07
*** aunnam has joined #osic17:08
*** palendae_ is now known as palendae17:08
*** sgundur has joined #osic17:08
*** blogan has joined #osic17:08
*** lin_yang has joined #osic17:08
*** leifz has joined #osic17:08
*** cloudnull has joined #osic17:08
*** darrenc has joined #osic17:08
*** mmotiani has joined #osic17:08
*** rderose has quit IRC17:09
*** claco has joined #osic17:11
sarafrajjthorne: Bunch of folks are hitting issues connecting to VM's in osic cloud. Any idea?17:11
jthornesarafraj: i'ma ware17:12
jthornei'm aware*17:12
jthornefloating ips aren't working after i spun up 600 instances in Cloud 117:12
inc0damn you neutron!17:12
jthornetrying to figure out what's up17:12
jthornethe werid thing is that every service is up17:12
sarafrajjthorne: ok thx17:12
inc0jthorne, try checking iptables17:13
*** onthecly has quit IRC17:13
inc0maybe iptables got so many rules that it got overloaded17:13
johnthetubaguypumaranikar: looking at your patch, what have you found from running that?17:15
aunnamhemanthm: haha again,lets see who wins now:) can you send me the link to your patch17:15
hemanthmaunnam: mind17:17
hemanthmbah17:17
hemanthmhttps://review.openstack.org/29887517:17
hemanthmaunnam ^17:17
pumaranikarjohnthetubaguy, test_resource_tracker.InstanceClaimTestCase.test_deleted_instances_with_migrations is failing17:19
pumaranikarjohnthetubaguy, I am not sure if it is related. If not, any other way to test this bug ?17:19
johnthetubaguypumaranikar: you have added a new DB call thats not allowed, thats what I would expect I think17:19
johnthetubaguyor not expected by the test, I guess17:19
pumaranikarjohnthetubaguy, yes falis when tries _db_instance_get_by_uuid from instance.py17:21
*** sgundur has left #osic17:21
johnthetubaguypumaranikar: yeah, that should be fine, if you hit that still17:22
pumaranikarjohnthetubaguy, matt suggested me to compare the instance.host to CONF.host17:22
pumaranikarjohnthetubaguy, any other way to check/debug ?17:23
johnthetubaguypumaranikar: unsure, I will have to go dig into that more, sorry, will try get back to you about that17:25
pumaranikarjohnthetubaguy, ok.17:26
pumaranikarjohnthetubaguy, thanks.17:27
*** homerp_ has joined #osic17:28
*** inc0 has quit IRC17:32
*** inc0 has joined #osic17:33
*** sgundur has joined #osic17:39
*** onthecly has joined #osic17:48
*** homerp_ has quit IRC17:51
*** rderose has joined #osic17:55
*** KLevenstein has joined #osic17:57
*** mdfuller has quit IRC17:59
*** asettle has joined #osic18:00
*** mdfuller has joined #osic18:00
*** cloudtrainme has quit IRC18:02
*** onthecly has quit IRC18:05
*** onthecly has joined #osic18:06
*** asettle has quit IRC18:10
*** homerp_ has joined #osic18:21
dasmjthorne: idk what you did, but ssh (at least for this network 172.99.106.9) works18:23
jthornei don't know what i did either! effing neutron18:24
jthornei simmply deleted all 600 of my instances18:24
dasmxD18:24
*** homerp_ has joined #osic18:27
*** mdfuller has quit IRC18:27
kencjohnstonjohnthetubaguy quick question. Do you know the status of cinder volume migrate in Nova's Newton Priorities?18:28
*** onthecly has quit IRC18:37
*** cloudtrainme has joined #osic18:41
*** onthecly has joined #osic18:42
*** onthecly has quit IRC18:47
*** sigmavirus24 has joined #osic18:49
*** dwalleck has joined #osic18:50
*** mdfuller has joined #osic18:54
*** rderose has quit IRC18:55
*** woodburn has joined #osic18:58
*** woodburn has quit IRC18:58
*** david-lyle has joined #osic18:58
*** woodburn has joined #osic18:58
*** dwalleck has quit IRC19:05
*** ankurgupta has quit IRC19:12
*** mdfuller has quit IRC19:16
* hockeynut wants to run Rally against the OSIC dev environment - where is that documented (topology, ids/pws, etc) ?19:17
*** mdfuller has joined #osic19:21
*** ankurgupta has joined #osic19:23
*** ediardo has joined #osic19:57
*** rmevans has joined #osic19:58
*** mdfuller has quit IRC19:58
*** jlwhite has joined #osic19:59
*** mdfuller has joined #osic20:02
*** rderose has joined #osic20:07
*** mdfuller has quit IRC20:20
*** mdfuller has joined #osic20:21
ntatajthorne, some of us lost connections to osic cloud again. is it still something to do with your 600 instances ?20:26
jthorneno. rabbit is a being a pain in the ass20:26
*** mcarden_ is now known as mcarden20:26
jthornecan't get the services to reconnect reliably20:27
clacoyw20:27
*** sigmavirus24 is now known as sigmavirus24_awa20:37
*** sigmavirus24_awa is now known as sigmavirus2420:37
*** asettle has joined #osic20:43
jthornefinally20:49
jthornecluster is stable. sorry folks.20:50
sarafrajjthorne: thx20:51
jthornenow i can finally spin up 600 instances again to see if our "tuning" fixes things20:53
dasmjthorne: did you manage to find root cause?20:54
jthornedasm so the first time is pun up 600 instances neutron networking flaked out20:54
jthorneit seems rabbit was overloaded or something20:54
dasmjthorne: correct me, if i'm wrong but spinning instances shouldn't be a problem for cloud20:54
dasmmhm. so mq20:55
jthorneonce i removed all my instances networking came back, which is very odd20:55
dasmjthorne: not for everyone. few of osicers had still problems with vms.20:55
dasmbut in general, it was better ;)20:55
jthorneright20:56
jthornethere was rabbit problems after that20:56
jthornewhen cloudnull and i were trying to apply some rabbit tuning20:56
dasmmhm.20:56
jthornethe docs weren't very clear with how to do it and it was causing problems20:56
jthornewe have finally applied it properly, and i'm spinning up 600 instances again20:56
dasm+120:56
*** JRobinson__ has joined #osic20:57
*** mdfuller has quit IRC20:57
jthornewho's having issues connecting to floats?20:57
dasmworks for me.20:59
*** cloudtrainme has quit IRC21:09
*** rmevans has quit IRC21:10
*** rmevans has joined #osic21:14
ntatajthorne, I still can't connect :(21:16
ntata172.99.106.10721:17
jthornewhich float?21:17
jthornethanks21:17
jthornentata: only ssh listening?21:17
ntatayes21:17
jthornecool. i'll use this to test a theory21:18
*** Mudpuppy has quit IRC21:20
ntatajthorne, actually I couldn't ping my instance either.. I guess ICMP has issues too21:22
jthornentata: if it was earlier, then that makes sense. everything was borked about 30 mins ago21:22
*** mdfuller has joined #osic21:25
*** inc0 has quit IRC21:26
cloudnulldasm: to clarify 600 instances is not a problem the larger issue was related to the messaging queue filling up faster than the neutron agent service could consume the instructures which caused rabbit instabillty, jthorne and i tihnk we've corrected that with elang vm tuning. now we need to see about making th agent work faster.21:27
sarafrajjthorne: I just spawned few instances and now able to reach to those. Even ping from another machine in osic1 cloud does not reach these new instances.21:27
sarafrajs/now/not21:28
jthorneit's probably because of the backlog21:29
jthornei'm deleting my 600 instances one by one to see if it clears it up21:29
cloudnull^ agreeded21:29
sarafrajjthorne: ok make sense21:29
jthorneduring the first time, once i deleted all my instances, things starting working21:29
jthorneduring the first test*21:29
jthornebecause the backlog was cleared up21:29
jthornethat's my working theory21:29
sarafrajjthorne: so you are creating 600 instances for testing some use case?21:30
*** homerp_ has left #osic21:30
jthorneRackspace has had customers request their openstack environments to spin up X instances in X time. we've been able to get 600 instances spun up in about 6 minutes, but there are always other issues such as networking not functioning or something.21:31
jthornei wanted to test that against Cloud 1 which has more than just one set of users using it21:31
sarafrajjthorne: ok21:31
jthorneand its a stat in the OSIC Exposed White Paper that will be available at the Summit. i want to make sure that stat is true21:31
*** nadeem has quit IRC21:34
cloudnullin terms of speed jthorne was able to do the original test in ~20 min which was before the intiial round of tuning. After tuning rabbitmq he's gotten it down to ~6 min . which IMO is a fantastic improvement in API performance. now we need to see if we can get the workers up to that level too.21:34
*** cloudtrainme has joined #osic21:35
*** ankurgupta has quit IRC21:36
jthornentata: are you able to communicate with your float?21:38
jthornentata: i haven't been able to. not sure if you have restricted security groups21:38
sarafrajjthorne: Also we ran out of floating ip's.21:39
ntatajthorne, not yet.. I don't have any specific restrictions.. I just have the default sec group set21:39
jthornesarafraj: there's only so much to give21:39
jthornesarafraj: how many more do you need?21:40
sarafrajjthorne: well I am just using one. need 2-3 more. I terminated bunch of instances earlier21:40
sarafrajWill those floating ip's get recycled back to pool?21:41
jthorneonly if you release it from your Project21:41
jthorneif you detach it from an instance it simply goes back into your Project's floating pool21:41
jthorneand you can reuse it elsewhere21:41
sarafrajso I terminated instances without dettaching floating ip21:42
jthornethat will detach it for you21:42
jthorneso you can reuse it again on another instance21:42
sarafrajok, so it shows me all 25 floating ip's are being used.21:42
sarafrajI think these floating ip's are for all osic folks and not just me21:42
sarafrajI am just using 121:43
*** KrishR1 has quit IRC21:43
*** sgundur has left #osic21:43
*** KrishR has joined #osic21:43
jthornesarafraj: go to Compute > Access and Security > Floating IPs21:44
jthorneanything not attached to an instance can be used21:44
jthornethe pie graph is stating that you all have allocated 25 floating IPs to your project, not that 25 floating IPs have been allocated to instances21:44
sarafrajjthorne: ok got it. thx21:46
dani_castellanosare you guys still testing stuff?21:47
dani_castellanosI can't connect to my instance21:47
*** sigmavirus24 is now known as sigmavirus24_awa21:47
sarafraj:)21:47
jthornedani_castellanos: yes and no. we're trying to figure out why some floats are having issues and others are not21:48
jthornefor example, osic.org points to a float and it remained working.21:48
dani_castellanosok. Thanks!21:49
*** adrian_otto has quit IRC21:49
*** rmevans has quit IRC21:57
*** rmevans has joined #osic21:58
jthornentata: can you detach and reattach your floating ip?22:06
jthornei want to see if it suddenly works22:06
ntatajthorne, ok, let me try22:07
ntatajthorne, I dettached and attached the same float.. still couldn't connect back22:08
jthorneblast22:08
jthorne172.66.106.107, right?22:08
jthorneerr22:08
jthorne172.99.106.10722:08
ntatayes22:08
jthornevery pecuiliar22:09
jthornei was able to spin up and instance, attach a float, and have it work22:09
jthornewhy do only some of the existing floats not work22:09
*** cloudtrainme has quit IRC22:09
ntatajthorne, actually are you trying to ssh to the instance?22:10
jthorneno just ping22:10
ntatasorry, my bad.. obviously you can't ssh without the pvt key22:11
jthorneit will still connect22:11
jthornei will just be denied22:11
*** ametts_ has quit IRC22:15
ntatagot it22:21
*** Mudpuppy has joined #osic22:29
*** mdfuller has quit IRC22:29
*** spotz is now known as spotz_zzz22:31
*** mdfuller has joined #osic22:31
raddaouiI detached and then reattached my floating ip now and it works again22:41
*** rmevans has quit IRC22:51
jthornentata: should be good now22:53
jthornedani_castellanos: should be good now22:53
jthornentata and dani_castellanos thatnks for your patience. i'm done for today22:53
ntatajthorne, yay! I'm able to login22:53
ntatathanks jthorne22:53
*** inc0 has joined #osic22:55
*** karthikp has quit IRC22:56
*** KLevenstein has quit IRC22:57
*** rderose has quit IRC23:00
*** cloudtrainme has joined #osic23:07
*** KrishR has quit IRC23:07
*** cloudtrainme has quit IRC23:10
*** asingh has quit IRC23:10
*** jorge_munoz has quit IRC23:15
*** adrian_otto has joined #osic23:16
*** JRobinson__ is now known as JRobinson__afk23:21
*** inc0 has quit IRC23:22
*** JRobinson__afk is now known as JRobinson__23:41
*** jlwhite has quit IRC23:41
*** pushkaru has quit IRC23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!