Monday, 2015-10-19

*** yuntongjin has joined #openstack-ceilometer00:17
*** larainema has joined #openstack-ceilometer00:34
*** robot has joined #openstack-ceilometer00:49
*** robot is now known as zqfan00:49
*** zqfan has quit IRC01:19
*** nadya has joined #openstack-ceilometer01:26
*** nadya has quit IRC01:31
*** liusheng has joined #openstack-ceilometer01:35
*** yfujioka has joined #openstack-ceilometer01:36
*** julim has quit IRC01:39
*** ljxiash has quit IRC02:10
*** zqfan has joined #openstack-ceilometer03:01
*** larainema has quit IRC03:02
*** ddieterly has quit IRC03:10
*** nadya has joined #openstack-ceilometer03:27
*** nadya has quit IRC03:31
*** eglynn_ has quit IRC03:34
*** links has joined #openstack-ceilometer03:41
*** larainema has joined #openstack-ceilometer03:47
tonybstable team and @cores: Please look at: that should unblock the stable/liberty branch for ceilometer.04:06
*** ddieterly has joined #openstack-ceilometer04:10
*** ddieterly has quit IRC04:15
*** jwcroppe_ has quit IRC04:20
*** jwcroppe has joined #openstack-ceilometer04:26
*** yuntongjin has quit IRC04:26
*** yprokule has joined #openstack-ceilometer04:33
*** yfujioka_ has joined #openstack-ceilometer04:41
*** yfujioka has quit IRC04:44
*** yfujioka_ has quit IRC04:44
*** yfujioka has joined #openstack-ceilometer04:44
*** yuntongjin has joined #openstack-ceilometer05:10
*** ddieterly has joined #openstack-ceilometer05:11
*** yuntongjin1 has joined #openstack-ceilometer05:15
*** ddieterly has quit IRC05:16
*** yuntongjin has quit IRC05:17
openstackgerritLianhao Lu proposed openstack/gnocchi: Use ListOpt for default_aggregation_methods option
*** nadya has joined #openstack-ceilometer05:27
*** nadya has quit IRC05:32
*** shardy_a1k has joined #openstack-ceilometer05:52
*** shardy_afk has quit IRC05:54
*** shardy_a1k has quit IRC05:57
*** shardy_afk has joined #openstack-ceilometer05:58
*** r-mibu has joined #openstack-ceilometer06:02
*** ddieterly has joined #openstack-ceilometer06:12
*** lsmola_ has joined #openstack-ceilometer06:16
*** ddieterly has quit IRC06:17
*** boris-42 has joined #openstack-ceilometer06:28
*** yasu has joined #openstack-ceilometer06:45
*** yasu has quit IRC06:58
*** yasu__ has joined #openstack-ceilometer06:58
*** nadya has joined #openstack-ceilometer07:12
*** ddieterly has joined #openstack-ceilometer07:13
*** ddieterly has quit IRC07:18
*** safchain has joined #openstack-ceilometer07:26
*** shardy_afk is now known as shardy07:36
*** nadya has quit IRC07:42
openstackgerritZhiQiang Fan proposed openstack/python-ceilometerclient: fix typos in docstring
*** Ala has joined #openstack-ceilometer08:02
*** deepthi has joined #openstack-ceilometer08:11
*** ddieterly has joined #openstack-ceilometer08:14
*** yassine has joined #openstack-ceilometer08:17
*** yassine has quit IRC08:17
*** yassine has joined #openstack-ceilometer08:17
*** ddieterly has quit IRC08:18
*** larainema has quit IRC08:22
*** shardy_ has joined #openstack-ceilometer08:25
*** Liam has joined #openstack-ceilometer08:25
*** shardy has quit IRC08:26
*** shardy_ has quit IRC08:30
*** akakios has joined #openstack-ceilometer08:31
*** shardy has joined #openstack-ceilometer08:31
*** nadya has joined #openstack-ceilometer08:34
openstackgerritMerged openstack/ceilometer: Updated from global requirements
*** yasu__ has quit IRC08:41
*** akakios has quit IRC08:43
*** eglynn_ has joined #openstack-ceilometer08:45
*** eglynn_ has quit IRC08:46
*** atty has joined #openstack-ceilometer08:47
zqfanis it possible to notify in this room when a new bug is created?08:53
*** cdent has joined #openstack-ceilometer08:57
*** Liam has quit IRC09:02
*** Liam has joined #openstack-ceilometer09:04
*** openstack has joined #openstack-ceilometer09:17
*** ddieterly has quit IRC09:19
*** iberezovskiy has joined #openstack-ceilometer09:21
openstackgerritZhiQiang Fan proposed openstack/ceilometer: reject post sample via direct while gnocchi is enabled
openstackgerritZhiQiang Fan proposed openstack/ceilometer: reject post sample via direct while gnocchi is enabled
*** Liam has quit IRC09:43
*** liusheng has quit IRC09:44
*** liusheng has joined #openstack-ceilometer09:44
*** nadya has quit IRC09:53
*** r-mibu has quit IRC09:53
*** yassine has quit IRC09:54
*** larainema has joined #openstack-ceilometer09:56
*** r-mibu has joined #openstack-ceilometer09:59
*** yasu has joined #openstack-ceilometer10:13
*** ddieterly has joined #openstack-ceilometer10:16
*** ddieterly has quit IRC10:21
openstackgerritZhiQiang Fan proposed openstack/ceilometer: Reject posting sample with direct=true if Gnocchi is enabled
*** cdent has quit IRC10:24
*** akakios has joined #openstack-ceilometer10:27
*** akakios has quit IRC10:38
*** nadya has joined #openstack-ceilometer10:39
*** weihan has joined #openstack-ceilometer10:39
*** eglynn has joined #openstack-ceilometer10:41
*** cdent has joined #openstack-ceilometer10:52
*** khushbu_ has joined #openstack-ceilometer11:03
*** akakios has joined #openstack-ceilometer11:06
*** ddieterly has joined #openstack-ceilometer11:17
*** ddieterl_ has joined #openstack-ceilometer11:20
*** ddieterly has quit IRC11:21
*** gordc has joined #openstack-ceilometer11:21
cdentgordc: are you still ill because I sure as hell am and it is your fault.11:30
gordccdent: mind or matter11:30
* cdent surrenders11:31
gordci'm not stronger than ever.11:31
gordcalso lazier than ever.11:31
cdentI think you were right the first time11:32
* cdent puts up his dukes11:33
gordci'm staying clear. i'm not getting sick again11:33
* cdent will be bringing some unique uk germs to tokyo, for sharing11:34
*** Ala has quit IRC11:35
gordccdent: i think they will force you to wear a mask11:35
gordcyou can pretend you're a doctor.11:35
cdentgee that's swell11:35
akakiosHi all, I am having a problem getting ceilometer to retrieve ceph/rados metrics. I have followed the steps in the Ceph Object Storage section ( and adjusted the ceilometer.conf & pipeline.yaml ( I have also restarted the ceilometer agent (and other ceilometer services in the controller) but the rados metrics st11:38
akakiosill won't show up. Nova & glance metrics are collected and rados has been setup as the object store. Querying the swift API also works correctly. Any ideas on what step/s I may have missed? I don't see any rados or object store related errors in the logs. Environment is running on Openstack Kilo and Ceph 0.93.11:38
gordcakakios: you see any errors in your central agent logs?11:46
akakiosgordc: the errors I've seen are related to authentication leading to the nova client11:48
*** krotscheck_ is now known as krotscheck11:48
akakiosgordc: unfortunately, environment is being reinstalled so I can't provide the exact errors11:48
gordcakakios: hmm. you can try opening a bug but i'm not sure we have enough info to go on (maybe someone else has same issue and has more detail or a fix)11:50
akakiosgordc: okay, going to give it another shot and see if it will get through this time11:52
akakiosgordc: but so far, I haven't missed any steps?11:52
gordcakakios: i don't use ceph personally but it's entirely possible there's a gap in the documentation. feel free to open a bug to track it11:55
gordcmight take a while to get to it since a lot of the devs are prepping for summit11:55
akakiosgordc: okay, thanks for the help11:55
*** thorst has joined #openstack-ceilometer12:01
*** larainema has quit IRC12:02
gordccdent: i think we need to equivalent for ceilometermiddleware12:03
gordcshould we be adding it to ceilometer repo or somewhere else?12:03
* cdent looks12:04
gordcwe have similar 'not installed correctly' errors as ceilometerclient
cdentat the moment in devstack, all ceilo-middleware stuff is (and should be) in lib/swift12:05
cdentso I'm guessing that's an oversight12:06
gordchmmm. seems to be there
gordcmaybe that's not the issue.12:09
*** yasu has quit IRC12:11
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements
*** ljxiash has joined #openstack-ceilometer12:25
cdentjd__: I haven't got any brilliant insights on what do about this issue in the gnocchi gate12:25
*** weihan_ has joined #openstack-ceilometer12:25
cdentthe only thing I can think of is adding some more swift jobs (with other indexers) as experimental and try them to see if the results are any different12:25
*** weihan_ has quit IRC12:26
*** weihan has quit IRC12:26
*** weihan has joined #openstack-ceilometer12:26
*** weihan has quit IRC12:27
cdentI diddled around with trying to replicate things locally but haven't had much success12:27
*** yasu has joined #openstack-ceilometer12:27
*** weihan has joined #openstack-ceilometer12:27
*** edmondsw has joined #openstack-ceilometer12:29
*** weihan has quit IRC12:30
jd__cdent: :/12:31
*** weihan has joined #openstack-ceilometer12:31
*** weihan has quit IRC12:34
* cdent is trying one more local investigation12:36
*** atty has quit IRC12:36
*** weihan has joined #openstack-ceilometer12:37
*** weihan has quit IRC12:37
*** weihan has joined #openstack-ceilometer12:38
cdentjd__: got it12:41
cdentnot a solution12:41
*** dikonoor has joined #openstack-ceilometer12:41
cdentbut I've replicated the failure12:41
cdent"die 387 'gnocchi-api did not start'" is the problem, yeah?12:41
*** yasu has quit IRC12:42
dikonoorgordc: hi gordc12:43
dikonoorgordc: could you take a look at when you get a chance12:43
openstackLaunchpad bug 1506717 in Ceilometer "Ceilometer GET events throws deadlock or timeout errors" [Undecided,New]12:43
*** khushbu_ has quit IRC12:46
*** weihan has quit IRC12:46
*** weihan has joined #openstack-ceilometer12:47
*** trown|outttypeww is now known as trown12:47
gordcdikonoor: isolation_level is causing issues in db2?12:52
dikonoorgordc: works fine if no isolation level is set12:54
dikonoorgordc: which was the case before12:55
*** ljxiash has quit IRC12:55
gordcdikonoor: what's the isolation level normally in db212:56
gordcthe issue is there are two queries to build event list12:56
gordcif you don't set isolation_level to repeatable read, both queries (may) work against different datasets12:57
*** ljxiash has joined #openstack-ceilometer12:57
*** yassine__ has joined #openstack-ceilometer12:58
gordcthe alternative is to leave default isolation levels for all db, and just toss any differences from second query and assume it's all good.12:58
*** yprokule has quit IRC13:00
*** yprokule has joined #openstack-ceilometer13:01
*** larainema has joined #openstack-ceilometer13:01
openstackgerritMerged openstack/ceilometer: make script under tools directory executable
openstackgerritMerged openstack/ceilometer: Reject posting sample with direct=true if Gnocchi is enabled
openstackgerritMerged openstack/ceilometer: Added the README.rst in devstack folder
*** ddieterl_ has quit IRC13:03
*** yuntongjin has joined #openstack-ceilometer13:04
dikonoorgordc: Not sure of the default isolation level in db2 . Also, I don't know if this problem occurs with other databases (but the documentation implies that it could). why do we do two queries to get events?13:16
*** Ala has joined #openstack-ceilometer13:17
gordcone to get events, the other to get traits.13:17
gordclast i tried, i couldn't come up with one single query to do that (as i couldn't handle scenario with events with no traits)13:18
gordcalso, i refuse to use orm relationships becasue it's slow as molasses.13:19
*** yuntongjin1 has joined #openstack-ceilometer13:23
jd__cdent: yeah13:24
jd__cdent: it times-out because of swift, etc…13:24
jd__cdent: you have something interesting?13:24
cdentjd__: well I have a local.conf that causes the same issue, so it iought to become a bit easier to debug13:24
*** yuntongjin has quit IRC13:25
jd__lucky you13:25
cdenthowever I'm not yet any closer to understanding what's going on13:25
cdentI agree with you that it seems likely an eventlet related thing13:26
dikonoorgordc: ok..I don't see this problem all the time..but once I see this..I have to wait for few minutes for this to be resolved.. :-|13:26
*** julim has joined #openstack-ceilometer13:27
gordcdikonoor: i'm not against dropping it. mysql already has repeatable read as default. the only thing is that if we don't set isolation level, we need to make sure the results are consistent still.13:28
*** akakios has quit IRC13:28
dikonoorgordc: hmm..agreed13:28
*** dan-t has joined #openstack-ceilometer13:29
*** ddieterly has joined #openstack-ceilometer13:32
cdentjd__: can you help me understand the flow of things a bit better?13:35
cdentIf I make any request to the gnocchi-api server after it starts up it appears to do something with the keystone client service_catalog and then does a PUT /v1/AUTH_<some key> to swift. What is that PUT? Why is it happening?13:36
*** lawrancejing has joined #openstack-ceilometer13:38
*** ddieterl_ has joined #openstack-ceilometer13:40
zqfanhi, good evening, I have a patch need your reviews: , this patch aims at improving performance for query-sample API based on mysql backend, previously it will time out (10 minutes) even specify limit on a test data set which only contains 500K samples, with this patch, it can response in around 10 seconds13:43
*** links has quit IRC13:44
*** ddieterly has quit IRC13:44
jd__cdent: it's the initial container creation to store new measures13:44
jd__cdent: that's the call that is blocking gnocchi entirely, the API and metricd13:44
jd__it never returns :(13:44
cdentyeah, I see what13:44
cdentIt just wasnt' clear what its purpose was13:44
jd__is it reproducible with a small Python script?13:45
cdentI'm narrowing my local.conf down to its bare minimum so I can share it with you13:45
jd__which would indicate Gnocchi has nothing to do with it13:45
jd__ok cool13:45
*** flwang has joined #openstack-ceilometer13:46
flwanggordc: ping13:46
*** exploreshaifali has joined #openstack-ceilometer13:47
cdentjd__: here's a relatively fast and small local.conf that shows the problem:
cdentjd__: with that setup the swift client blocks when trying to use it without gnocchi being involved at all13:51
* cdent tries again without gnocchi running anywhere13:52
jd__pretty cool13:52
jd__I hope we can say we have a so good test coverage we find bugs people miss -_-13:52
jd__I'm trying your local.conf13:52
*** ityaptin has joined #openstack-ceilometer13:53
gordcflwang: hey, whatsup?13:54
flwanggordc: jd__: did you see this ?13:55
gordcflwang: i do now13:55
cdentjd__: :( If I use that same local.conf but with no gnocchi around, swift works just fine...13:55
*** weihan has quit IRC13:55
gordcflwang: it's at the same time as ceilometer work session though13:56
flwangzaqar team and horizon team are working on a real time table update topic13:56
*** yprokule has quit IRC13:56
flwanggordc: i will talk with mrunge to see if the time can be changed13:56
flwanggordc: in a short, zaqar team would like to have a zaqar publisher, just like kafka13:57
gordcflwang: ah, the stuff vkmc was playing with13:57
*** yprokule has joined #openstack-ceilometer13:58
gordcflwang: not sure what exactly is being planned but i can try joining if it doesn't overlap with ceilometer sessions13:59
*** flwang1 has joined #openstack-ceilometer14:01
vkmcgordc, that incredible piece of code you meant, right?14:02
flwang1gordc: just wanna get your comments about adding a publisher for zaqar in ceilometer14:02
flwang1vkmc: hey14:02
gordcvkmc: lol. sure.14:02
vkmcflwang, gordc, hey :)14:02
*** flwang has quit IRC14:02
jd__cdent: interesting!14:05
gordcflwang1: i'm not super familiar with zaqar but how is it different from the other oslo.messaging drivers? or they completely unrelated?14:05
jd__cdent: a diff in pip freeze maybe? I don't think gnocchi configures swift in a different way14:06
flwang1gordc: they're different14:06
cdentjd__: something like that yeah14:06
flwang1the oslo.messaging drivers, like rabbitmq, are most like a lib, but zaqar is multi tenancy messaging system14:07
* cdent makes a diff14:07
flwang1gordc: for example, when publish the message to zaqar, you need to know the project id to post the messages14:07
gordcflwang1: the project_id of the target?14:09
flwang1gordc: yep14:09
*** KrishR has joined #openstack-ceilometer14:09
gordcflwang1: i see.14:10
gordcflwang1: i don't really have any issue with having it internally in ceilometer repo or externally.14:10
flwang1gordc: the current problem of horizon is it has to poll forever to get the latest object status14:11
flwang1for example, when user is viewing the instances page14:11
cdentjd__: no diff14:11
flwang1horizon has to poll nova to get the latest status for all the instnaces per 2.5 sec14:12
*** yuntongjin1 has quit IRC14:12
flwang1it's really bad14:12
flwang1so horizon team is trying to leverage ceilometer and zaqar to figure out a good way to fix it14:12
gordcflwang1: you mean the the pie graphs on the main page?14:13
flwang1gordc: the instance ovewview table14:13
flwang1on horizon menu  /project/instances14:13
jd__cdent: hihihihi O_O14:14
flwang1gordc: and if there is no ceilometer or zaqar deployed, then horizon will use current way to poll14:14
gordcflwang1: this page?
jd__cdent: i'm just getting crazy14:15
cdentdarn, I thought you had uncovered the secret...14:15
flwang1gordc: no, Instances14:15
flwang1not overview14:15
jd__cdent: I can reproduce it now, same as you14:16
*** rbak has joined #openstack-ceilometer14:16
cdentI guess that's progress14:16
jd__cdent: I'm gonna try to comment part of gnocchi devstack to see what part makes that14:16
gordcflwang1: got it14:16
gordcyeah, i wasn't aware that page was polling constantly.14:16
cdentI suppose a next step is to instrument the swift-proxy server to see what it is doing when it is blocking14:16
* cdent tries that14:17
flwang1gordc: if there is no objection, i will post a spec in ceilometer-spec for you guys review14:17
flwang1gordc: and i will talk with horizon guys to see if we can change the time14:17
flwang1i mean the session time14:17
gordcflwang1: sure sounds fine.14:18
gordcflwang1: i should point out we also have a ceilometer+horizon slot
flwang1for visualizing data, IIRC?14:19
flwang1gordc: cool, that's an option14:20
gordcnot exactly related to what you're trying to accomplish but basically we're trying to discuss how to present ceilometer data in a useful manner14:20
gordc(because based on feedback, it's not useful currently)14:20
*** shardy_ has joined #openstack-ceilometer14:20
flwang1gordc: yep, i think it's good topic14:21
flwang1gordc: i do have some suggestions for that14:21
*** shardy has quit IRC14:22
flwang1gordc: thank you so much for your comments14:22
gordcflwang1: cool cool. feel free to add notes to etherpad14:23
openstackgerritWen Zhi Yu proposed openstack/ceilometer: Unify the timestamp of polled data
flwang1gordc: will do :)14:24
*** alexpilotti has quit IRC14:26
*** shardy_ has quit IRC14:26
*** rakhi_ has joined #openstack-ceilometer14:26
*** links has joined #openstack-ceilometer14:26
*** shardy has joined #openstack-ceilometer14:27
*** links has quit IRC14:35
*** weihan has joined #openstack-ceilometer14:37
*** weihan has quit IRC14:40
*** weihan has joined #openstack-ceilometer14:41
jd__gordc: do we know if anyone from Horizon is going to show up?14:45
cdentjd__: slightly more data (you may already have this datapoint): If you kill the swift-proxy server and start it back up again subsequent queries do not block14:45
jd__gordc: because I'm sure we all have good ideas but if nobody cares to implement them, it's just a waste of time14:45
jd__cdent: oh? no I did not notice that, very strange14:46
jd__cdent: me I just learnt that the problem lies in init_gnocchi14:46
gordcjd__: i believe david-lyle has made his team aware?14:46
cdentjd__: how'd you determine that?14:47
gordcjd__: i don't know if the exact right people will come but we can always re-raise in ML14:47
jd__cdent: I enabled Gnocchi plugin, then # out everything – swift worked14:48
cdentoh I see what you mean14:49
jd__cdent: then I uncommented line by line and waited to find the culprit = swift does not work anymore14:49
jd__init_gnocchi does very few things14:49
jd__I'll comment line by line and see14:49
jd__dat suspense14:50
* cdent votes for auth cache dir14:50
jd__cdent: I guess you lost14:53
cdentokay, but if you've found it we all win and cookies all round14:53
cdentoh hey, I just had a successful run14:54
cdentI changed that curl call14:54
cdentto behave slightly differently14:54
cdentand now it works14:54
cdentone moment please14:56
jd__recreate_database gnocchi utf814:56
jd__that's the line apparently14:56
* jd__ rechecks14:56
* cdent rechecks his thing14:56
jd__I really can't see how this line can make things not work so might be something else14:57
cdentmy theory is that this line isn't good enough:
cdentI'm adding a --max-time 514:58
cdentso that it will fail waiting for a response14:58
cdentotherwise it never actually retries14:58
* cdent waits15:00
jd__hmmm so it's just a slowness issue?15:00
cdentit doesn't seem to want to work this time, still waiting15:00
cdentyeah, didn't work that time, not sure what's different15:00
* cdent sighs15:00
* cdent tries again15:02
jd__epic debug…15:03
*** deepthi has quit IRC15:04
*** rakhi_ has quit IRC15:05
*** larainema_ has joined #openstack-ceilometer15:05
cdentjd__: is there any chance the database recreate is throwing postgres into confusion which would then block the world?15:06
jd__cdent: I don't know, I'm retrying to confirm this is init_gnocchi failing and where15:07
jd__one of my test got mangled because of a bad unstack I think15:07
* jd__ wishes devstack was fast15:07
cdentyeah, unstack is not super reliable15:07
cdentpatches accepted! ;)15:07
jd__why is my devstack trying to run keystone-all15:09
jd__is this on the gate too?15:09
cdentgate has mod wsgi false for gnocchi but true for keystone if I recall correctly15:10
jd__no gnocchi uses wsgi too actually15:11
cdentwe turned if off a while ago didn't we?15:11
cdentor was it that we just set the port?15:11
openstackgerritZhiQiang Fan proposed openstack/ceilometer: avoid generate temporary table when query samples
* cdent prepares to try some other permutations15:13
cdent(I got distracted from that, that was actually my original goal)15:13
jd__so my current problem is the following15:14
jd__2015-10-19 15:12:44.397 TRACE keystone error: [Errno 98] Address already in use15:14
jd__keystone starts somewhere in background15:14
*** ityaptin has quit IRC15:14
jd__and screen shows that in the one that is ran in screen15:14
jd__so there's like 2 keystone-all running15:14
jd__I don't get why15:14
cdentthat sounds like a bad unstack, or a left over keystone in /etc/{httpd,apache2}/conf.d15:15
jd__apache conf is clean, and a I unstacked and killed everything15:15
jd__and I was able to reproduce it15:15
* jd__ investigates15:15
* cdent doesn't seem to have that problem15:16
jd__Lost in Bash15:20
jd__an OpenStack movie15:20
*** dikonoor has quit IRC15:22
vkmcjd__, want to see that15:22
* cdent keeps on stackin'15:24
* jd__ too15:24
jd__that could be a song15:24
cdentjd__: you probably already guess this, but postgres/mysql makes no difference, same problem in both15:24
jd__♭ these scripts are made for stackin' ᖱ15:25
cdentand that's what I'm gonna do15:25
cdentone of these days these scripts are gonna stack all over you15:25
cdentyep, that'll do15:25
cdentjd__: try this:
cdentfrom what I can tell gnocchi-metricd is starting up to talk to swift and making a bunch of bad connections, causing the proxy server to run out of available stuff15:33
cdentmoving it to the end prevents that15:33
cdentbut is not enough15:33
cdentthe swift proxy server isn't quite ready when the curl is done so we have to wait for it but the previous retry was not the right kind: it would only timeout once because it was blocking15:34
cdentthe --max-time change makes it so the curl will fail and then properly retry15:34
cdentI've made that work twice in a row now so it is at least a move in the right direction15:34
cdentseem like there may be an issue in metricd's retry logic or connection pool handling? or swift is just lame at startup time15:35
cdenttry that, lemme know, I'll brb15:35
openstackgerritJulien Danjou proposed openstack/gnocchi: devstack: remove unused utf8 argument
jd__cdent: trying!15:38
*** lawrancejing has quit IRC15:45
jd__cdent: first run worked15:49
jd__I really don't understand what your patch changes15:49
jd__it should not change anything to have metricd started first15:49
cdentif you look at the gnocchi-metricd logs in failed runs you may see that it has a lot of failed auth errors15:50
cdentor just failed connections, can't remember15:50
cdentwhatever it is, there are a lot of  them15:50
cdentdoes metricd rememer to do the same initial container creation that the api server does?15:51
* jd__ goes watch logs15:53
jd__and what does it fail if swift is running anyway15:53
cdentMy guess is that there is an initial slowness in s-proxy that metricd is not dealing with, and is instead just doing more work, somehow, this then aggrevates swift even more so that by the time we get to testing if the api server is up things are hosed15:54
cdentit's quite hard to figure out the how/why of it all :(15:54
jd__yeah, sounds like a race condition on Swift side15:58
jd__cdent: you send a patch?15:58
cdentI will yeah, sure, just a sec15:58
jd__my second devstack run succeeded15:58
*** safchain has quit IRC16:00
openstackgerritChris Dent proposed openstack/gnocchi: Make sure that swift doesn't block gnocchi startup
cdentwe'll see how that goes16:03
jd__ah Gerrit still 500 with Unicode in comments, sigh16:06
* cdent checks what year it is16:07
*** shardy_ has joined #openstack-ceilometer16:08
*** shardy has quit IRC16:10
openstackgerritDong Ma proposed openstack/ceilometer: Added CORS support to Ceilometer
*** shardy_ has quit IRC16:14
*** shardy has joined #openstack-ceilometer16:15
*** vishwana_ has joined #openstack-ceilometer16:26
*** trown|brb has joined #openstack-ceilometer16:28
*** Ala has quit IRC16:28
*** vishwanathj has quit IRC16:29
*** chmouel has quit IRC16:29
*** trown has quit IRC16:29
cdentjd__: it worked in the gate too. have a good flight!16:32
*** weihan has quit IRC16:33
*** chmouel has joined #openstack-ceilometer16:34
*** trown|brb is now known as trown16:34
*** davidlenwell_ is now known as davidlenwell16:36
jd__cdent: thanks!16:36
*** exploreshaifali has quit IRC16:45
*** yprokule has quit IRC16:49
*** Ephur has joined #openstack-ceilometer16:50
*** yassine__ has quit IRC16:51
*** openstackgerrit has quit IRC17:01
*** openstackgerrit has joined #openstack-ceilometer17:01
*** lsmola_ has quit IRC17:04
*** jwcroppe_ has joined #openstack-ceilometer17:23
*** jwcroppe has quit IRC17:24
*** iberezovskiy has quit IRC17:26
*** dan-t has quit IRC17:26
*** jwcroppe has joined #openstack-ceilometer17:41
*** jwcroppe_ has quit IRC17:44
openstackgerritMerged openstack/gnocchi: Make sure that swift doesn't block gnocchi startup
*** cdent has quit IRC18:21
*** nadya has quit IRC18:21
*** devlaps has joined #openstack-ceilometer18:22
*** dan-t has joined #openstack-ceilometer18:22
*** shardy is now known as shardy_afk18:30
*** eglynn has quit IRC18:32
*** eglynn has joined #openstack-ceilometer18:32
ildikovgordc: hey, are you around?18:37
gordcildikov: yep18:39
*** khushbu_ has joined #openstack-ceilometer18:43
ildikovgordc: maybe I asked it already, but maybe not :)18:44
ildikovgordc: I have a patch for Aodh which contains the config options18:45
ildikovbut it has additional items added by the script which generates the docs18:45
*** Guest3686 is now known as mgagne18:45
*** mgagne has joined #openstack-ceilometer18:45
ildikovthe items are coming from oslomiddleware18:45
ildikovdo you have any idea what dependency can bring them in, which is not present when we generate the config file?18:46
ildikovgordc: ^^18:46
gordcaodh has additional items?18:47
ildikovgordc: so my issue is that the config file I generate with tox contains less config options than the list what is generated by the doctools script18:47
ildikovgordc: which would not be a problem if we would not generate the config ref with scripts, which means that if I delete it now by hands it will not solve my issue long term18:48
ildikovgordc: I'm just out of ideas regarding where the dependenies can go wrong and where the oslomiddleware options can come18:49
ildikovgordc: sorry for the dumb question/topic :)18:49
*** larainema_ has quit IRC18:50
gordcildikov: did you add it here?
*** larainema_ has joined #openstack-ceilometer18:50
gordcildikov: to be honest, my mind is shut down so i'm not really sure i know what you're asking.lol18:50
ildikovgordc: lol :)18:51
openstackgerritMerged openstack/python-gnocchiclient: Creates better exceptions for http code 409
ildikovgordc: I have this patch:
ildikovgordc: which is not merged, because the list contains additional items compared to the actual config file of Aodh18:52
ildikovgordc: but the option list is generated by a script, that reads the code and of course does some magic...18:52
ildikovgordc: and I ran out of ideas that which part of the magic went wrong :)18:53
gordcwhere's the script?18:53
*** julim has quit IRC18:53
*** nadya has joined #openstack-ceilometer18:53
ildikovgordc: script:
ildikovgordc: hook for Aodh:
*** julim has joined #openstack-ceilometer18:56
gordchmm.. so we can't do the exact same thing as we did for ceilometer?18:56
gordci have no idea what that script does.18:57
*** boris-42 has quit IRC18:58
gordcildikov: i think it's pretty obvious i have no idea how that works18:59
ildikovgordc: it differs from Ceilometer because the script uses the global CONF object, which we removed from Aodh :(19:00
gordci see.19:01
ildikovgordc: I wonder how much that would hurt to leave the very few extra options in the docco19:01
ildikovgordc: I'm really not sure how many people reads it as a novel, I mean from the first line to the last19:01
ildikovgordc: I hate the idea, but don't have any better :S19:02
ildikovgordc: and I already asked the person, who wrote that script, which I only slightly modified...19:02
gordcildikov: i think it's ok to have more (if they're still valid)19:04
ildikovgordc: maybe I will ask the guys to merge the patch and report a bug about the issue19:05
ildikovgordc: the options are valid, just not for Aodh or at least we don't use them, so I assume they don't do any harm19:06
gordcildikov:  yeah that migh tbe easiest.19:06
gordci've no idea how that script works.19:06
ildikovgordc: yeah, in this detail me neither :(19:08
ildikovgordc: thanks for your time!19:10
gordcildikov: is it just the CORS stuff?19:13
ildikovgordc: oslo_middleware/max_request_body_size and oslo_middleware/secure_proxy_ssl_header IIRC19:16
ildikovgordc: I mean these are not in the sample config file we generate19:16
gordcseems like keystone has it in their's as well19:22
gordcildikov: maybe it's just a limitation of tool19:22
gordci'd probably just leave it19:22
ildikovyou mean the docco or the keystone config file itself where they have it?19:26
*** eglynn has quit IRC19:29
gordcit's in both for keystone.. but i don't think max_request_body_size is actually used anywhere19:32
*** KrishR has quit IRC19:37
*** eglynn has joined #openstack-ceilometer19:41
*** KrishR has joined #openstack-ceilometer19:41
*** khushbu_ has quit IRC19:49
*** exploreshaifali has joined #openstack-ceilometer19:50
*** eglynn has quit IRC20:07
*** eglynn has joined #openstack-ceilometer20:07
*** nadya has quit IRC20:14
ildikovgordc: I'm convinced :)20:14
openstackgerritChris Dent proposed openstack/gnocchi: rest: deserialize directly with file descriptor
*** larainema_ has quit IRC20:28
*** larainema_ has joined #openstack-ceilometer20:29
openstackgerritChris Dent proposed openstack/gnocchi: TEST: Try a longer timeout on waiting for swift
*** Ephur has quit IRC20:31
*** gordc has quit IRC20:38
*** thorst has quit IRC20:40
*** Ephur has joined #openstack-ceilometer20:49
*** julim has quit IRC21:04
*** trown is now known as trown|outttypeww21:06
*** eglynn has quit IRC21:15
*** jwcroppe has quit IRC21:21
*** larainema_ has quit IRC21:31
*** larainema has quit IRC21:32
*** dan-t has quit IRC21:47
*** exploreshaifali has quit IRC22:14
*** ddieterl_ has quit IRC22:16
*** flwang1 has quit IRC22:29
*** thorst has joined #openstack-ceilometer22:34
*** ddieterly has joined #openstack-ceilometer22:45
*** KrishR has quit IRC22:53
*** rbak has quit IRC22:59
openstackgerritZhiQiang Fan proposed openstack/ceilometer: avoid generate temporary table when query samples
*** lawrancejing has joined #openstack-ceilometer23:16
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometer: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/ceilometermiddleware: Updated from global requirements
*** ddieterly has quit IRC23:35
*** ddieterly has joined #openstack-ceilometer23:38
*** ddieterly has quit IRC23:42
*** ddieterly has joined #openstack-ceilometer23:43
*** ddieterl_ has joined #openstack-ceilometer23:49
*** ddieterly has quit IRC23:52

Generated by 2.14.0 by Marius Gedminas - find it at!