Thursday, 2017-01-05

*** marst has quit IRC00:04
*** ducttape_ has quit IRC00:11
*** ducttape_ has joined #openstack-operators00:11
mnaserdmsimard wouldn't be the first time openstack gets redone in go00:20
* mnaser hides00:20
*** ducttape_ has quit IRC00:25
*** chyka has quit IRC00:29
*** chyka has joined #openstack-operators00:30
*** chyka has quit IRC00:34
mdormananybody out there using RabbitMQ 3.6.6 or 3.6.x yet?  What’s been the experience?  Any gottchas?00:37
*** mriedem has quit IRC00:41
*** ducttape_ has joined #openstack-operators00:51
*** sterdnotshaken has quit IRC00:51
*** ducttape_ has quit IRC00:52
*** ducttape_ has joined #openstack-operators00:52
*** VW has joined #openstack-operators01:11
*** VW has quit IRC01:15
*** sdake has quit IRC01:21
*** mriedem has joined #openstack-operators01:31
*** liuyulong has joined #openstack-operators01:32
*** furlongm has quit IRC01:36
*** furlongm has joined #openstack-operators01:36
*** ducttape_ has quit IRC01:37
*** ducttape_ has joined #openstack-operators01:45
*** ducttape_ has quit IRC02:21
*** ducttape_ has joined #openstack-operators02:23
*** harlowja has quit IRC03:05
*** aluria has quit IRC03:05
*** cloudnull has quit IRC03:05
*** Oku_OS-away has quit IRC03:05
*** yantarou has quit IRC03:05
*** yantarou has joined #openstack-operators03:05
*** aluria has joined #openstack-operators03:05
*** Oku_OSz has joined #openstack-operators03:05
*** cloudnull has joined #openstack-operators03:05
*** mriedem has quit IRC03:16
*** sudipto_ has joined #openstack-operators03:24
*** sudipto has joined #openstack-operators03:24
*** udesale has joined #openstack-operators03:31
*** marst has joined #openstack-operators03:34
*** VW has joined #openstack-operators03:45
*** ducttape_ has quit IRC03:55
*** haplo37 has quit IRC04:12
*** haplo37 has joined #openstack-operators04:21
*** furlongm has quit IRC04:35
*** furlongm has joined #openstack-operators04:36
*** ducttape_ has joined #openstack-operators04:46
*** VW has quit IRC04:52
*** sudipto has quit IRC05:02
*** sudipto_ has quit IRC05:02
*** ducttape_ has quit IRC05:15
*** sudipto has joined #openstack-operators05:27
*** sudipto_ has joined #openstack-operators05:27
*** dfflanders has quit IRC05:41
*** VW has joined #openstack-operators06:06
*** ducttape_ has joined #openstack-operators06:15
*** ducttape_ has quit IRC06:20
*** bjolo has joined #openstack-operators06:38
*** sdake has joined #openstack-operators06:51
*** pcaruana has joined #openstack-operators07:05
*** tesseract has joined #openstack-operators07:09
*** simon-AS559 has joined #openstack-operators07:14
*** rcernin has joined #openstack-operators07:17
*** simon-AS559 has quit IRC07:19
*** sdake_ has joined #openstack-operators07:28
*** armax has joined #openstack-operators07:30
*** sdake has quit IRC07:30
*** VW has quit IRC07:33
*** ducttape_ has joined #openstack-operators07:46
*** ducttape_ has quit IRC07:51
*** armax has quit IRC08:13
*** aluria has quit IRC08:17
*** aluria has joined #openstack-operators08:17
*** simon-AS559 has joined #openstack-operators08:17
*** simon-AS5591 has joined #openstack-operators08:19
*** simon-AS559 has quit IRC08:21
*** armax has joined #openstack-operators08:22
*** jsheeren has joined #openstack-operators08:26
*** racedo has joined #openstack-operators08:31
*** saneax-_-|AFK is now known as saneax08:46
*** racedo has quit IRC08:47
*** saneax is now known as saneax-_-|AFK08:47
*** racedo has joined #openstack-operators08:49
*** saneax-_-|AFK is now known as saneax08:50
*** racedo has quit IRC08:51
*** sdake_ is now known as sdake08:54
*** racedo has joined #openstack-operators08:55
*** zioproto has joined #openstack-operators09:07
*** armax has quit IRC09:14
*** bvandenh has joined #openstack-operators09:15
*** ducttape_ has joined #openstack-operators09:17
*** ducttape_ has quit IRC09:21
*** VW has joined #openstack-operators09:33
*** VW has quit IRC09:38
*** sdake_ has joined #openstack-operators09:42
*** saneax is now known as saneax-_-|AFK09:43
*** sdake has quit IRC09:44
*** dbecker has joined #openstack-operators09:45
*** derekh has joined #openstack-operators09:47
*** saneax-_-|AFK is now known as saneax09:47
*** rmart04 has joined #openstack-operators09:52
*** rmart04 has quit IRC09:52
*** rmart04 has joined #openstack-operators10:01
*** paramite has joined #openstack-operators10:05
*** sdake_ has quit IRC10:06
*** openstackgerrit has joined #openstack-operators10:12
openstackgerritSaverio Proto proposed openstack/osops-tools-contrib: nova-libvirt-compare.py: compare Nova state to hypervisor state  https://review.openstack.org/41691810:12
*** openstack has joined #openstack-operators10:15
*** sudipto_ has quit IRC10:16
*** sudipto has quit IRC10:16
*** udesale has quit IRC10:28
*** rmart04 has quit IRC10:31
*** rmart04 has joined #openstack-operators10:33
*** udesale has joined #openstack-operators10:36
*** ducttape_ has joined #openstack-operators10:47
*** furlongm has quit IRC10:48
*** furlongm_ has joined #openstack-operators10:48
*** ducttape_ has quit IRC10:52
*** sudipto has joined #openstack-operators11:00
*** sudipto_ has joined #openstack-operators11:01
*** armax has joined #openstack-operators11:05
*** udesale has quit IRC11:12
*** furlongm has joined #openstack-operators11:12
*** furlongm_ has quit IRC11:13
*** sudipto_ has quit IRC11:16
*** sudipto has quit IRC11:16
*** armax has quit IRC11:26
*** Miouge has joined #openstack-operators11:34
*** rmart04 has quit IRC11:40
*** rmart04 has joined #openstack-operators11:41
*** rmart04 has quit IRC11:41
*** rmart04 has joined #openstack-operators11:42
*** derekh has quit IRC11:55
*** tdurakov has quit IRC11:55
*** esmiurium has quit IRC11:55
*** emerson has quit IRC11:55
*** dansmith has quit IRC11:55
*** Tridde has quit IRC11:55
*** topol has quit IRC11:55
*** haukebruno has quit IRC11:55
*** zeus has quit IRC11:55
*** Anticimex has quit IRC11:55
*** zeus- has joined #openstack-operators11:55
*** haukebruno has joined #openstack-operators11:55
*** topol_ has joined #openstack-operators11:55
*** Trident has joined #openstack-operators11:55
*** dansmith has joined #openstack-operators11:56
*** derekh has joined #openstack-operators11:56
*** zeus- is now known as zeus11:56
*** esmiurium has joined #openstack-operators11:56
*** dansmith is now known as Guest4110011:56
*** armax has joined #openstack-operators11:56
*** tdurakov has joined #openstack-operators11:56
*** zeus is now known as Guest7429011:56
*** serverascode has quit IRC11:59
*** serverascode has joined #openstack-operators12:01
*** racedo has quit IRC12:02
*** racedo has joined #openstack-operators12:03
*** saneax is now known as saneax-_-|AFK12:11
beddariwe're getting requests from customers wanting shelve-by-default ... anyone dealt with that?12:12
zioprotobeddari: never12:14
zioprotohello all12:14
*** haplo37 has quit IRC12:14
zioprotoI just figured out that Horizon when listing the images, tries also to show glance images that are not public anymore12:15
zioprotoI get my log files for glance poisoned with http://paste.openstack.org/show/593986/12:15
beddarihi Saviero /wave ... shelve by default kind of makes sense, so I get why they request it12:15
zioprotobeddari: I meant I never dealt with shelve by default :)12:15
*** ducttape_ has joined #openstack-operators12:18
beddarire the glance log, why is it doing a GET there? I'm guessing a race of some sorts, so that image is actually just gone?12:18
beddariah "not public anymore", so they used to be?12:19
*** ducttape_ has quit IRC12:22
*** haplo37 has joined #openstack-operators12:24
zioprotook I give a bit of background12:26
zioprotowe generate new glance images everymonth12:27
zioprotoso you have a default ubuntu image for example12:27
zioprotothe uuid changes everymonth12:27
zioprotothe old image is set as private12:27
zioprotoand users dont see it anymore12:27
zioprotobut12:27
zioprotoopenstack server show uuid12:27
zioprotowill show the uuid of the image of the time when the instance was started12:27
zioprotoso we see horizon12:27
zioprotoasks informations about this glance uuids that are not public anymore12:28
zioprotothis generates tons of 404 and backtraces in the glance api12:28
*** catintheroof has quit IRC12:38
*** catintheroof has joined #openstack-operators12:44
*** vinsh_ has quit IRC12:51
*** armax has quit IRC12:54
*** catintheroof has quit IRC12:57
*** catintheroof has joined #openstack-operators12:58
*** catintheroof has quit IRC13:02
*** catintheroof has joined #openstack-operators13:04
*** sudipto_ has joined #openstack-operators13:04
*** sudipto has joined #openstack-operators13:04
*** ducttape_ has joined #openstack-operators13:09
*** catintheroof has quit IRC13:11
*** simon-AS559 has joined #openstack-operators13:12
*** simon-AS5591 has quit IRC13:15
*** simon-AS5591 has joined #openstack-operators13:17
*** simon-AS559 has quit IRC13:20
beddariah hm I see, my answer is just (boring) "we don't do that" ...13:23
*** Anticimex has joined #openstack-operators13:25
*** udesale has joined #openstack-operators13:31
*** d0ugal has quit IRC13:34
*** d0ugal has joined #openstack-operators13:35
*** catintheroof has joined #openstack-operators13:47
*** ducttape_ has quit IRC13:48
*** liverpooler has quit IRC13:51
*** jamesdenton has joined #openstack-operators13:52
*** cdelatte has joined #openstack-operators13:53
erhudymdorman: i've been upgrading clusters to rabbit 3.6.5 and it has been unremarkable13:56
erhudyactually, we've been on 3.6.x for a while, older clusters are on 3.6.013:57
*** marst has quit IRC13:57
erhudyupgrading from 3.5.x to 3.6.0 was slightly janky because i think they changed the password hashing mechanism and it invalidated the account passwords, but was easy to fix13:57
*** dminer has joined #openstack-operators14:05
*** marst has joined #openstack-operators14:05
*** marst has quit IRC14:10
*** marst has joined #openstack-operators14:11
*** pkoniszewski has quit IRC14:15
*** pkoniszewski has joined #openstack-operators14:15
*** ducttape_ has joined #openstack-operators14:16
*** Guest74290 is now known as zeus`14:18
*** zeus` is now known as zeus14:18
*** zeus has quit IRC14:18
*** zeus has joined #openstack-operators14:18
*** zioproto1 has joined #openstack-operators14:31
*** simon-AS559 has joined #openstack-operators14:32
*** simon-AS5591 has quit IRC14:32
*** zioproto has quit IRC14:32
*** VW has joined #openstack-operators14:38
*** liverpooler has joined #openstack-operators14:39
*** ducttape_ has quit IRC14:43
*** mriedem has joined #openstack-operators14:45
*** zioproto1 has quit IRC14:54
*** clickboom has joined #openstack-operators15:00
*** bjolo has quit IRC15:00
*** jsheeren has quit IRC15:01
openstackgerritMaciej Jozefczyk proposed openstack/osops-tools-contrib: Add cold start, cold stop, cold restart to script  https://review.openstack.org/41700315:04
*** chlong has joined #openstack-operators15:05
openstackgerritMaciej Jozefczyk proposed openstack/osops-tools-contrib: Add cold stop, cold start, cold restart to dhcp_agents_balancer.py Cold stop: remove all network to agent associations Cold start: add network to agent associations Cold restart: do stop and start  https://review.openstack.org/41700315:07
*** udesale has quit IRC15:07
*** ducttape_ has joined #openstack-operators15:14
*** Guest41100 is now known as dansmith15:15
*** cnf has quit IRC15:15
*** dansmith is now known as Guest5735515:15
*** cnf has joined #openstack-operators15:16
*** Guest57355 is now known as dansmith15:16
*** sdake has joined #openstack-operators15:20
*** tdasilva- is now known as tdasilva15:23
*** sdake_ has joined #openstack-operators15:23
*** sdake has quit IRC15:25
*** catintheroof has quit IRC15:32
*** clickboom has quit IRC15:34
*** ducttape_ has quit IRC15:37
*** d0ugal has quit IRC15:38
*** d0ugal has joined #openstack-operators15:41
mriedemwe got one reply on http://lists.openstack.org/pipermail/openstack-operators/2016-November/012085.html - can everyone else take a look at this and let me know if they are using this plugin mechanism for image downloads before we yank it in ocata?15:43
mdormanerhudy:  great, thanks for the data point15:46
*** vinsh has joined #openstack-operators15:49
*** VW has quit IRC15:49
*** ducttape_ has joined #openstack-operators15:49
*** VW has joined #openstack-operators15:54
* mnaser sighs15:58
mnaserwe don't get discount codes for "the forum" anymore even if we've contributed code?15:58
mnaserthe only way we can is by attending the PTG event as well?15:58
*** sdake_ has quit IRC16:02
*** sdake has joined #openstack-operators16:03
*** VW_ has joined #openstack-operators16:03
*** VW_ has quit IRC16:05
*** VW_ has joined #openstack-operators16:05
*** VW has quit IRC16:06
*** rcernin has quit IRC16:07
mdormanthere’s all the AUC criteria, too.  I did not necessarily take it to mean that attending the PTG is the _only_ way to get a discount code.  Just that if you do attend PTG, you would get one.16:09
*** TxGirlGeek has joined #openstack-operators16:10
*** rmart04 has quit IRC16:13
*** VW_ has quit IRC16:14
*** VW has joined #openstack-operators16:15
*** sdake has quit IRC16:16
*** tesseract has quit IRC16:21
*** VW_ has joined #openstack-operators16:22
*** VW has quit IRC16:23
mnasermdorman: but, as a contributor, that was a nice thing to have which is gone now.. unless i attend the PTG16:30
mnaseressentially paying twice16:30
*** catintheroof has joined #openstack-operators16:32
erhudymnaser: reading your convo in #ceph about colocated OSDs with VMs16:33
erhudyall i can say is run away, don't walk16:33
mnasererhudy story time?16:33
erhudynot really a specific story, just a warning that it's fraught with peril16:33
mnaseroh i've heard that part a lot16:33
mnaserbut also i hear that redhat has been pushing that architecture latelyw16:34
mnaserwith some use cases16:34
erhudyreally? they always tell us the exact opposite16:34
erhudyour larger clusters are a constant stream of blocked request whining16:34
*** chyka has joined #openstack-operators16:34
erhudywe also effed up by having the ceph public/private networks be the same network, so mon traffic competes with OSD replication traffic and sometimes the mons start unexpectedly forcing elections16:34
erhudyif i were doing it over i'd have them on separate VLANs and QoS the public storage VLAN up at the switch16:35
erhudyplus the operational annoyance of disks failing and then the controller or HBA won't recognize the replacement disk until a reboot, or it creates an entirely new device node, so you have to get all the instances off or tell people their instances are going down for a bit16:38
erhudywe're still on kilo on the larger clusters so live migration can be a crapshoot16:39
erhudyi'd like to deconverge our block storage off the compute nodes so that we could buy purpose-built nodes for each role and also ensure we're able to tune each role properly for what it's doing16:40
mnasererhudy the public/private network thing would cause a lot of problems regardless16:42
mnaserand that operational thing... we've worked around it by just doing live migrations, works well for us (but we're on mitaka)16:42
mnaseri have considered deconverge-ing but so far... it works®16:43
erhudywe recently had an ops issue with some newer machines and we had to do around 800 live migrations, probably <10 actually failed, so it wasn't awful16:43
erhudyit just sucks having to put on the shoulder-length glove and reach up into libvirt to clean up a busted migrate16:43
mnasermitaka live migrations have been *really* good now16:45
*** saneax-_-|AFK is now known as saneax16:49
*** VW_ has quit IRC16:51
logan-erhudy: are you running a new enough libvirt version where you can enable some of the newer live migrate flags like memory delta compression and auto convergence? we saw a lot of migration failures/timeouts, especially windows instances, but the memory compression basically changed it from a crapshoot to a very low migration failure rate16:52
*** VW has joined #openstack-operators16:52
erhudyyeah, windows instances were guaranteed failures16:52
erhudywe source all our stuff from canonical's cloud archive, so on the kilo clusters libvirt's at 1.2.12-0ubuntu14.3~cloud0, and on the liberty ones 1.2.16-2ubuntu11.15.10.4~cloud016:53
*** sdake has joined #openstack-operators16:54
logan-you might check if VIR_MIGRATE_COMPRESSED is supported.. iirc the kilo cloud archive had compression but not auto converge16:55
*** liverpooler has quit IRC16:55
logan-good preso about the newer migration flags available https://01.org/sites/default/files/dive_into_vm_live_migration_2.pdf16:55
erhudyman, using that picture of the hoboken PATH elevator flooding in their presentation16:55
erhudytoo soon!16:55
logan-haha16:57
erhudyi expect to revisit live migration more closely once we get to mitaka16:57
*** sterdnotshaken has joined #openstack-operators16:57
*** pcaruana has quit IRC16:58
*** liverpooler has joined #openstack-operators17:00
*** racedo has quit IRC17:07
*** kittyceramic has joined #openstack-operators17:09
*** kittyceramic has left #openstack-operators17:09
*** derekh has quit IRC17:12
*** sterdnotshaken has quit IRC17:16
*** dbecker has quit IRC17:18
*** Long_yanG has quit IRC17:20
*** LongyanG has joined #openstack-operators17:20
*** sdake has quit IRC17:24
*** sdake has joined #openstack-operators17:28
*** saneax is now known as saneax-_-|AFK17:53
*** liverpooler has quit IRC17:56
*** liverpooler has joined #openstack-operators17:58
*** VW_ has joined #openstack-operators17:59
*** VW_ has quit IRC17:59
*** VW_ has joined #openstack-operators18:00
*** VW_ has quit IRC18:00
*** VW has quit IRC18:02
*** sterdnotshaken has joined #openstack-operators18:10
*** LiberalCarrot has joined #openstack-operators18:14
*** LiberalCarrot has left #openstack-operators18:14
*** simon-AS559 has quit IRC18:20
*** VW has joined #openstack-operators18:35
*** VW has quit IRC18:40
*** simon-AS559 has joined #openstack-operators18:53
*** VW has joined #openstack-operators19:00
*** simon-AS559 has quit IRC19:01
*** paramite has quit IRC19:10
*** rcernin has joined #openstack-operators19:11
*** delaf has joined #openstack-operators19:15
*** harlowja has joined #openstack-operators19:16
*** sudipto_ has quit IRC19:21
*** sudipto has quit IRC19:21
*** racedo has joined #openstack-operators19:32
*** emerson has joined #openstack-operators19:34
*** sterdnotshaken has quit IRC19:36
*** racedo has quit IRC19:45
*** sdake has quit IRC19:52
*** simon-AS559 has joined #openstack-operators19:57
*** markvoelker has quit IRC19:59
*** bjolo has joined #openstack-operators19:59
*** markvoelker has joined #openstack-operators19:59
*** marst_ has joined #openstack-operators20:01
*** chlong has quit IRC20:03
*** marst has quit IRC20:05
*** TxGirlGeek has quit IRC20:06
*** TxGirlGeek has joined #openstack-operators20:06
*** VW has quit IRC20:21
*** VW has joined #openstack-operators20:23
*** VW has quit IRC20:26
*** sterdnotshaken has joined #openstack-operators20:27
*** VW has joined #openstack-operators20:27
*** bvandenh has quit IRC20:27
*** ducttape_ has quit IRC20:28
*** ducttape_ has joined #openstack-operators20:28
*** TxGirlGeek has quit IRC20:30
*** ducttape_ has quit IRC20:34
*** ducttape_ has joined #openstack-operators20:34
*** sdake has joined #openstack-operators20:42
*** dbecker has joined #openstack-operators20:42
*** bvandenh has joined #openstack-operators20:43
*** catintheroof has quit IRC20:45
*** bvandenh has quit IRC20:48
*** ddyer has quit IRC20:53
*** dbecker has quit IRC20:55
*** ddyer has joined #openstack-operators21:00
*** TxGirlGeek has joined #openstack-operators21:05
*** bvandenh has joined #openstack-operators21:07
*** bjolo has quit IRC21:11
*** cdelatte has quit IRC21:19
*** _ducttape_ has joined #openstack-operators21:21
*** armax has joined #openstack-operators21:22
*** ducttape_ has quit IRC21:22
*** Miouge has quit IRC21:29
*** sterdnotshaken has quit IRC21:31
*** Miouge has joined #openstack-operators21:33
*** sterdnotshaken has joined #openstack-operators21:33
*** sterdnotshaken has quit IRC21:33
*** sterdnotshaken has joined #openstack-operators21:37
*** bvandenh has quit IRC21:37
*** sterdnotshaken has joined #openstack-operators21:40
*** sterdnotshaken has quit IRC21:40
*** liverpooler has quit IRC21:51
openstackgerritMerged openstack/osops-tools-contrib: Add cold stop, cold start, cold restart to dhcp_agents_balancer.py Cold stop: remove all network to agent associations Cold start: add network to agent associations Cold restart: do stop and start  https://review.openstack.org/41700321:51
*** Miouge has quit IRC21:53
*** dbecker has joined #openstack-operators22:03
*** sterdnotshaken has joined #openstack-operators22:06
*** darrenc is now known as darrenc_afk22:07
*** simon-AS559 has quit IRC22:09
*** saneax-_-|AFK is now known as saneax22:09
*** sterdnotshaken has quit IRC22:10
*** sterdnotshaken has joined #openstack-operators22:14
*** sterdnotshaken has quit IRC22:14
*** sterdnotshaken has joined #openstack-operators22:15
*** sterdnotshaken has quit IRC22:15
*** darrenc_afk is now known as darrenc22:25
*** saneax is now known as saneax-_-|AFK22:26
*** sterdnotshaken has joined #openstack-operators22:27
*** saneax-_-|AFK is now known as saneax22:34
*** sterdnotshaken1 has joined #openstack-operators22:35
*** jamesdenton has quit IRC22:37
*** sterdnotshaken has quit IRC22:37
*** sdake has quit IRC22:39
*** _ducttape_ has quit IRC22:46
*** mriedem has quit IRC22:54
*** saneax is now known as saneax-_-|AFK22:54
*** marst_ has quit IRC22:59
*** sdake has joined #openstack-operators23:03
*** TxGirlGeek has quit IRC23:04
*** saneax-_-|AFK is now known as saneax23:05
*** sterdnotshaken has joined #openstack-operators23:16
*** saneax is now known as saneax-_-|AFK23:16
*** sterdnotshaken1 has quit IRC23:18
*** saneax-_-|AFK is now known as saneax23:26
*** saneax is now known as saneax-_-|AFK23:43
*** dminer has quit IRC23:46
*** ducttape_ has joined #openstack-operators23:48
*** VW has quit IRC23:48
*** ducttape_ has quit IRC23:53
*** saneax-_-|AFK is now known as saneax23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!