Friday, 2013-11-01

*** nati_uen_ has quit IRC00:00
*** nati_uen_ has joined #openstack-metering00:00
*** nati_uen_ has quit IRC00:01
*** nati_uen_ has joined #openstack-metering00:02
openstackgerritVictor Stinner proposed a change to openstack/ceilometer: Replace mox with mocks in tests.compute.pollsters
*** nati_ueno has quit IRC00:03
*** sandywalsh has quit IRC00:11
*** matsuhashi has joined #openstack-metering00:25
*** nosnos has joined #openstack-metering01:09
*** bingbu has joined #openstack-metering01:23
*** nati_uen_ has quit IRC01:24
*** nati_ueno has joined #openstack-metering01:24
*** nati_ueno has quit IRC01:30
sphoorti_sileht, are you around?01:43
*** nati_ueno has joined #openstack-metering01:53
*** SergeyLukjanov is now known as _SergeyLukjanov02:01
*** _SergeyLukjanov has quit IRC02:02
*** suo has joined #openstack-metering02:09
*** krtaylor has joined #openstack-metering02:10
*** llu has quit IRC02:17
*** sdake has joined #openstack-metering02:27
*** sdake has joined #openstack-metering02:27
*** SergeyLukjanov has joined #openstack-metering02:39
*** matsuhashi has quit IRC02:43
*** matsuhashi has joined #openstack-metering02:43
*** matsuhas_ has joined #openstack-metering02:44
*** matsuhashi has quit IRC02:44
*** nati_ueno has quit IRC03:02
*** nati_ueno has joined #openstack-metering03:03
*** nati_ueno has quit IRC03:11
*** SergeyLukjanov has quit IRC03:11
*** SergeyLukjanov has joined #openstack-metering03:12
*** nati_ueno has joined #openstack-metering03:15
*** matsuhas_ has quit IRC03:20
*** matsuhashi has joined #openstack-metering03:21
*** sdake has quit IRC03:21
*** matsuhashi has quit IRC03:26
*** sdake has joined #openstack-metering03:29
*** suo has quit IRC03:49
*** suo has joined #openstack-metering03:49
*** SergeyLukjanov has quit IRC03:49
*** sphoorti_ has quit IRC03:52
*** SergeyLukjanov has joined #openstack-metering04:01
openstackgerritChenZheng proposed a change to openstack/ceilometer: Add i18n warpping for all LOG messages
*** SergeyLukjanov has quit IRC04:25
*** matsuhashi has joined #openstack-metering04:28
*** sphoorti has joined #openstack-metering04:48
*** sdake has quit IRC04:49
*** bingbu has quit IRC04:55
*** shakayumi has joined #openstack-metering05:09
openstackgerritChenZheng proposed a change to openstack/ceilometer: Add i18n warpping for all LOG messages
openstackgerritJenkins proposed a change to openstack/ceilometer: Imported Translations from Transifex
*** SergeyLukjanov has joined #openstack-metering06:05
*** sphoorti has quit IRC06:26
*** nati_ueno has quit IRC06:27
*** nati_ueno has joined #openstack-metering06:27
*** sphoorti has joined #openstack-metering06:28
*** nati_ueno has quit IRC06:32
*** nati_ueno has joined #openstack-metering06:33
*** evanjfraser has joined #openstack-metering06:41
*** evanjfraser has quit IRC06:49
*** sayali has joined #openstack-metering07:07
*** eglynn has joined #openstack-metering07:25
*** nati_ueno has quit IRC07:26
*** nati_ueno has joined #openstack-metering07:27
sphoortieglynn, hello :)07:29
*** nati_ueno has quit IRC07:31
*** lsmola has joined #openstack-metering07:44
*** sayali has quit IRC07:44
*** eglynn has quit IRC07:45
*** matsuhashi has quit IRC08:08
*** matsuhashi has joined #openstack-metering08:08
*** matsuhashi has quit IRC08:12
*** sayali has joined #openstack-metering08:13
*** SergeyLukjanov has quit IRC08:14
*** matsuhashi has joined #openstack-metering08:16
*** matsuhashi has quit IRC08:34
*** eglynn has joined #openstack-metering08:45
sphoortieglynn, hi :)08:53
*** tasdomas` has joined #openstack-metering08:57
*** tasdomas has quit IRC08:58
*** matsuhashi has joined #openstack-metering09:02
*** sayali has quit IRC09:02
*** sphoorti has quit IRC09:02
*** sayali has joined #openstack-metering09:02
*** sphoorti has joined #openstack-metering09:03
*** Ruetobas has quit IRC09:04
*** Ruetobas has joined #openstack-metering09:06
*** giroro_ has joined #openstack-metering09:09
*** matsuhashi has quit IRC09:10
*** Ruetobas has quit IRC09:11
*** matsuhashi has joined #openstack-metering09:11
*** matsuhashi has quit IRC09:15
eglynnsphoorti: hi09:16
sphoortihey eglynn , i am working on this bug
sphoortii am failing to understand the exact bug. As to what exactly i should be doing09:18
sphoortimy mentors told me to look into ceilometer/networks/ file09:19
sphoortihow should i approach this problem?09:19
sphoortieglynn, ?09:19
*** matsuhashi has joined #openstack-metering09:20
*** nprivalova has joined #openstack-metering09:21
eglynnsphoorti: /me looking ...09:21
sphoortithank you eglynn :)09:21
eglynnsphoorti: OK so the basic problem is a mismatch between the structure of actual notification payload emitted by neutron09:26
eglynnsphoorti: ... and the expectations that ceilometer has around that content09:26
eglynnsphoorti: in particular, ceilo expects the notification message to look like this:09:26
eglynn{'payload': {'tenant_id': PROJECT_ID, 'label_id': LABEL_ID, ...}}09:26
eglynnwhereas traceback in the bug report is telling you that it doesn't look like that09:27
eglynnin fact it's missing a mapping for the 'tenant_id' key in the payload dict09:27
sphoortitraceback says its in this format  project_id=message['payload']['tenant_id'] right eglynn ?09:27
sphoortiso is it a syntactical error which i need to resolve?09:28
eglynnsphoorti: no, the traceback says that's what ceilometer expects09:28
eglynnsphoorti: whereas the structure looks somehow different09:28
eglynnsphoorti: this is not a syntax error in the ceilo code09:28
eglynnsphoorti: it's a mismatch between the expected and actual notification structure09:29
eglynnsphoorti: so how are you going to debug this?09:29
sphoortibut eglynn the file has the same line as that in traceback09:29
eglynnsphoorti ... eh, yeah, all python tracebacks look like that09:30
sphoortiokay eglynn , so in communication between neutron and ceilometer, the format in which neutron is sending is not matching with ceilometer's?09:30
eglynnsphoorti: ... i.e. include the problematic line from the code that cuased the exception to be raised09:30
sphoortiso the eglynn should i be opening some other file for this error?09:31
eglynnsphoorti: ... think about it for a second, before I hand you the answer09:31
sphoortisure eglynn09:31
eglynnsphoorti: ... where does the problematic notification originate from?09:31
eglynnsphoorti: ... how could you get visibility onto the actual notification structure?09:32
eglynnsphoorti: ... have you looked at the ceilometer logs?09:32
eglynnsphoorti: ... with debug=True09:32
eglynnsphoorti: ... have you looked at the neutron code that produces this notification?09:33
sphoortieglynn, i ll first check the ceilometer logs09:33
eglynnsphoorti: ... facts you need to establish09:33
eglynn1. what the notification actually looks like in reality09:33
eglynn2. whether the structure of the notification is variable according to the scenario at hand, or fixed09:34
eglynn3. whether it's an error in the neutron notification logic in certain cases, or just a mismatch between the actual notification structure and ceilometer's expectation of same09:35
eglynnsphoorti: the answer to question #3 will tell you where the fix needs to be applied09:35
eglynnsphoorti: back to the traceback, do you understand what the traceback is telling you?09:36
eglynnsphoorti: ... it's not saying that the notification is in the format project_id=message['payload']['tenant_id']09:36
eglynnsphoorti: ... rather its saying the code handling the notification is blowing up when it tries to access message['payload']['tenant_id']09:38
sphoortieglynn, is the flow from to and then from the to_sample to
eglynnsphoorti: the notification handlers are loaded as stevedove extensions and then sit there within the service process passively waiting for notifications to consume09:41
sphoortieglynn, i just opened the extensions.py09:44
eglynnsphoorti: why?09:44
eglynnsphoorti: seriuusly, close it right up again ... that module will tell nothing about this bug09:45
sphoorticlosed eglynn . sorry :(09:46
eglynnsphoorti: stevedore is just a library we use to dynamically load extensions09:46
eglynnsphoorti: it's completely generic, there's nothing to see there WRT this particular bug09:46
eglynnsphoorti: you need to look at ...(a)09:47
eglynn(a) the ceilometer logs showing the actual notification structure09:47
eglynn(b) the neutron code that emits that notification09:47
eglynnsphoorti: first things first ... figure out what the AMQP notification actually looks like09:48
sphoortiokay eglynn09:48
eglynnsphoorti: do you have a devstack running with both ceilometer & neutron?09:50
eglynnsphoorti: do you know which ceilometer service loads the problematic notification handler?09:51
sphoortieglynn, i have ceilometer09:51
eglynnsphoorti: ^^^ the answer to that question will tell which ceilometer log file to look in09:51
eglynnsphoorti: "i have ceilometer" == "i have ceilometer but not neutron"?09:52
sphoortiyes eglynn . sorry :(09:52
eglynnsphoorti: no problem!09:52
eglynnsphoorti: ... BTW I'm just trying to prompt your own independent thought process09:53
eglynnsphoorti: ... (hope I don't appear difficult or narky :) )09:53
eglynnsphoorti: so why is neutron important in this case?09:54
eglynnsphoorti: (for background, there are two separate network services in openstack, the original nova-network and the newer neutron service)09:54
eglynnsphoorti: devstack by default still uses nova-network09:55
eglynnsphoorti: no in this bug, it doesn't explicitly state that the issue arises with neutron09:55
sphoortino eglynn . That will help me actually :).09:55
eglynnsphoorti: s/no in this bug/now in this bug/09:56
eglynnsphoorti: but we can see that it's neutron from the log file URL that identifies the CI gate as: gate-tempest-devstack-vm-neutron-large-ops09:56
eglynnsphoorti: so in order to reproduce the issue, you'll need a neutron based devstack09:57
sphoortieglynn, should i parallely start following these instructions?
eglynnsphoorti: yes, spin up a fresh devstack with localrc based on that wiki page PLUS the ceilometer related setting you already have10:00
sphoortieglynn, i ll simulateneously do that now.10:02
openstackgerritChenZheng proposed a change to openstack/ceilometer: Add i18n warpping for all LOG messages
eglynnsphoorti: cool :)10:02
sphoortieglynn, i am working on neutron installtion. will ping you in a bit10:11
*** nijaba_ has quit IRC10:19
*** nprivalova has quit IRC10:21
*** matsuhashi has quit IRC10:25
sphoortieglynn, sorry for pinging late, was fixing network connectivity issue on virtual box. Restarted the virtual box.10:29
sphoortithe script is running now eglynn :)10:29
*** nosnos has quit IRC10:32
sphoortieglynn, the collectorservice in ceilometer handles the notifications right?10:34
eglynnsphoorti: yep, exactly10:47
*** nprivalova has joined #openstack-metering10:47
eglynnsphoorti: (ensure debug=True in /etc/ceilometer/ceilometer.conf to dial logging up to the max)10:48
*** suo has quit IRC10:48
sphoortieglynn, neutron installed :) i can see it in my system resources :)10:50
sphoortieglynn, now checking /etc/ceilometer/ceilometer.conf10:52
sphoortieglynn, debug=True added in ceilometer.conf10:54
*** mwagner has joined #openstack-metering11:28
*** sphoorti has quit IRC11:28
*** sayali has quit IRC11:29
*** sphoorti has joined #openstack-metering11:29
*** sayali has joined #openstack-metering11:29
*** ruhe has joined #openstack-metering11:35
*** nprivalova has quit IRC11:47
*** ruhe has quit IRC12:14
sphoortieglynn, are you around?12:30
*** sphoorti has quit IRC12:39
*** nprivalova has joined #openstack-metering12:43
*** thomasem has joined #openstack-metering12:47
*** sphoorti has joined #openstack-metering12:48
*** julim has joined #openstack-metering13:05
thomasemHey all13:09
sphoortihi thomasem :)13:10
thomasemHey, sphoorti! How's the day treating you?13:11
sphoortithomasem, eglynn was helping me understand me the bug.13:11
thomasemOh, awesome.13:11
sphoortithomasem, i installed neutron on devstack13:11
sphoortiand i was looking into the file13:12
sphoortithomasem, how i should approach this bug?13:12
thomasemCan you link the bug?13:13
sphoortisure :)13:13
eglynnsphoorti: the key code to look at is not the (again that's just the foundation)13:19
eglynnsphoorti: instead look at the ceilometer/network/notifications.py13:19
eglynnsphoorti: and the collector logs, so the incoming AMQP notifications from neutron13:20
sphoortieglynn, i am searching for the collector logs13:23
eglynnsphoorti: did you set LOGFILE in your localrc?13:24
eglynnsphoorti: see for details of configuring logging13:25
eglynnsphoorti:otherwise just run:13:26
eglynnsceen -r13:26
eglynn*screen -r13:27
eglynnchoose the ceilo collector screen13:27
sphoortiokay eglynn will do that13:28
eglynnand you can search back thru' the collector logs13:28
sphoortieglynn, i had copied the sample localrc file. terriyu had suggested me13:28
eglynnsphoorti: OK, forget about changing the LOGFILE setting now13:28
sphoortieglynn, i have LOGFILE in my localrc file13:28
eglynnsphoorti: just look at the screen -r13:28
*** ruhe has joined #openstack-metering13:28
sphoortiit has aegthe destination is set to logs/
lsmolaeglynn, hello13:31
eglynnlsmola: yo13:31
*** ruhe has quit IRC13:32
lsmolaeglynn, we are talking on #openstack-horizon about sayali 's patch13:33
*** sayali has quit IRC13:33
lsmolaeglynn, that we will probably accept the 10, then the UX guys have to figure out the best approach to show more entities in one chart13:33
*** sphoorti has quit IRC13:34
eglynnlsmola: meh, no channel logging?13:34
*** sayali has joined #openstack-metering13:35
eglynnlsmola: who are the UX guys? the horizon core team?13:35
lsmolaeglynn, the jcoufal is the UX guy :-) and yeah the Horizon team too :-)13:35
*** sphoorti has joined #openstack-metering13:36
lsmolaeglynn, the rest of the conversation is on #openstack-opw13:36
lsmolaeglynn, seems you are not there :-)13:36
*** ruhe has joined #openstack-metering13:38
*** ruhe has quit IRC13:57
eglynnjd__: you around?14:06
eglynnjd__: you happy enough for me to go ahead and stick those slides up on
*** eglynn is now known as eglynn-lunch14:09
*** giroro_ has quit IRC14:10
sayalii will be back later too :)14:11
*** ruhe has joined #openstack-metering14:11
*** Ruetobas has joined #openstack-metering14:11
*** sayali has quit IRC14:11
*** sphoorti has quit IRC14:12
*** Ruetobas has quit IRC14:16
*** Ruetobas has joined #openstack-metering14:28
*** nprivalova has quit IRC14:29
*** Ruetobas has quit IRC14:32
*** ruhe has quit IRC14:34
*** Ruetobas has joined #openstack-metering14:36
*** eglynn-lunch is now known as eglynn14:49
*** sphoorti has joined #openstack-metering14:52
*** lsmola has quit IRC15:12
*** nealph has left #openstack-metering15:17
*** ArcTanSusan has joined #openstack-metering15:46
*** lsmola has joined #openstack-metering15:55
*** Ruetobas has quit IRC16:01
*** sdake has joined #openstack-metering16:01
*** jsergent has joined #openstack-metering16:02
*** Ruetobas has joined #openstack-metering16:03
*** giroro_ has joined #openstack-metering16:06
*** Ruetobas has quit IRC16:07
*** ArcTanSusan has quit IRC16:20
*** nati_ueno has joined #openstack-metering16:30
*** jsergent has quit IRC17:02
*** julim has quit IRC17:03
*** SergeyLukjanov has joined #openstack-metering17:08
*** jsergent has joined #openstack-metering17:17
*** jsergent has quit IRC17:28
*** jsergent has joined #openstack-metering17:28
*** krtaylor has quit IRC17:39
*** sayali has joined #openstack-metering17:46
*** shakayumi has quit IRC17:50
*** julim has joined #openstack-metering17:51
*** eglynn has quit IRC18:05
*** lsmola has quit IRC18:07
*** shakayumi has joined #openstack-metering18:10
*** jsergent has quit IRC18:11
*** krtaylor has joined #openstack-metering18:20
*** nati_ueno has quit IRC18:21
*** nati_ueno has joined #openstack-metering18:21
*** jsergent has joined #openstack-metering18:23
*** SergeyLukjanov has quit IRC18:34
*** eglynn has joined #openstack-metering18:51
sphoortieglynn, hello :)19:00
*** sayali_ has joined #openstack-metering19:20
*** sayali has quit IRC19:24
eglynnsphoorti: hi19:45
sphoortihi eglynn19:45
sphoortimade some progress19:46
eglynnsphoorti: cool have you figured out the notification message structure?19:46
sphoortiwith thomasem and jpich help we mailed the openstack-dev group regarding a proposed solution and asking if there is restriction in the structure19:47
sphoortii also made that particular change in my file19:47
eglynnsphoorti: great!19:48
sphoortieglynn, after changing the code i ran too19:49
sphoortiand then i created one instance19:49
sphoortibut i am stuck now. I am not understanding how i should be checking what my changes have done.19:49
sphoortihave they solved the bug or not19:49
sphoortibesides eglynn i think i have failed in creating an instance in the first place19:50
eglynnsphoorti: I don't creating an instance is the best way to cause neutron to emit a notification19:52
eglynnsphoorti: better to create a resource under neutron's control19:52
eglynnsphoorti: such as a router19:52
sphoortithen eglynn how otherwise should i check if my code is working?19:52
eglynnsphoorti: as I said above, create a resource that neutron is concerned with, such as a router19:53
eglynnsphoorti: start with 'neutron help'19:54
eglynnsphoorti: that'll tell you the commands available via the neutron CLI19:54
sphoortiokay eglynn i ll do that.19:54
sphoortii deleted the instance i created19:54
eglynnsphoorti: so what does the message payload from neutron actually look like?19:55
eglynnsphoorti: ... i.e. is the tenant_id completely missing from the notification?19:55
eglynnsphoorti: ... (or just available via a key that ceilo doesn't expect?)19:56
*** vkmc has joined #openstack-metering19:56
eglynnsphoorti: ... is your change to along the lines of s/['tenant_id']/.get('tenant_id)/ ?19:57
sphoortieglynn, yes19:57
sphoortii did .get(tenant_id,None)19:57
eglynnsphoorti: the None default is redundant there I think19:57
sphoortieglynn, thomasem suggested we could use it19:58
sphoortiso that KeyError wont be thrown19:58
sphoortiif no match found19:58
thomasemsphoorti, eglynn is correct19:58
thomasemyou can just do .get(?) and it'll return None automatically.19:58
eglynnsphoorti: but in any case, that's more like making the ceilometer code tolerant of the tenant_id not being present in the notification payload19:58
eglynnthomasem: exactly19:59
eglynnsphoorti: we still have to figure out why the tenant_id is missing19:59
thomasemThat's what the email to the Neutron team is about. I was wondering if some change had gone in recently that could effect those attributes.20:00
eglynnsphoorti: ... because ceilometer needs the tenant ID to be set in order to bill by project / group statistics by project etc.20:00
thomasem^^ :)20:00
eglynnsphoorti: do you have an example of the actual notification message payload?20:00
eglynnsphoorti: (from the collector logs?)20:01
eglynnsphoorti: the tenant ID may be missing completely, or the structure of the notification may be such that the tenant ID is just not where ceilometer expects20:01
eglynnsphoorti: so next task is to get an example of such a notification payload20:02
sphoortieglynn, i ran screen -r in /opt/stack/ceilometer/collector20:02
eglynnsphoorti: "in /opt/stack/ceilometer/collector"?20:03
eglynnsphoorti: it doesn't really matter what your current directory is when you run screen -r20:03
eglynnsphoorti: run 'screen -r' from anywhere20:04
eglynnsphoorti: *then* you need to switch to the collector screen20:04
eglynnsphoorti: earlier I gave you the screen command to list the available screens ... ctrl+"20:04
eglynnsphoorti: the collector should be one of these20:04
eglynnsphoorti: make sense?20:04
sphoortieglynn, when  i ran the screen -r command it says no screen to be resumed20:05
eglynnsphoorti: hmmm, on the host you're running devstack on?20:06
eglynnsphoorti: check if the openstack services actually running20:07
eglynnsphoorti: ps -fe | grep ceilometer20:07
*** krtaylor has quit IRC20:07
eglynnsphoorti: (for example...)20:07
eglynnsphoorti: are those services actually running on that host?20:08
sphoortieglynn, screen -r also says there is screen on 24811.stack20:08
eglynnsphoorti: oh, I thought you said there was no screen to be resumed?20:09
eglynnsphoorti: try: screen -r 24811.stack20:09
sphoortiand the grep instruction does give me ceilometer20:09
eglynnsphoorti: did you run twice in succession?20:09
eglynnsphoorti: (without doing an in between?)20:10
eglynnsphoorti: anyway what does 'screen -r 24811.stack' give you?20:10
sphoortiThere is a screen on 24811.stack20:10
sphoortiand at end there is no screen to be resumed matching 24811.stack20:11
eglynnsphoorti: are you running all of this on a fresh VM, or on a baremetal host?20:12
eglynnsphoorti: is it possible that a previous running of devstack is interfering here?20:13
sphoortieglynn, should i restart my vm and see?20:13
eglynnsphoorti: first try running ...20:13 ; stack.sh20:13
sphoortiokay eglynn20:13
sphoortiwill do that20:14
eglynnsphoorti: if that doesn't work, my advice would be to simply spin up a completely fresh VM20:14
eglynnsphoorti: (as opposed to rebooting your existing one)20:14
sphoortieglynn, :)20:26
sphoortii think i wont have to install again20:27
sphoortii did stack.sh20:27
sphoortiand then screen -r20:27
sphoortiand now i dont get the same result like before20:27
sphoortiinstead i now see a whole list of staements and at the end ceilometer.alarm.service20:28
eglynnsphoorti: those statements look healthy?20:29
eglynnsphoorti: i.e. not error messages, just informational?20:30
sphoortiHTTP/1.0 200 ok20:30
eglynnsphoorti: sweet20:30
sphoortihush :)20:30
eglynnsphoorti: so if you hit ... ctrl&"20:30
eglynnsphoorti: do you see a list of numbered screens?20:31
eglynnsphoorti: if so, navigate to the ceilometer collector screen20:31
eglynnsphoorti: that's the one you're interested in from a logging perspective20:31
sphoortieglynn, sorry i am asking a really lame question20:32
sphoortiits ctrl& that i type right?20:32
eglynnsphoorti: no question is too lame :)20:32
eglynnsphoorti: sorry my fault ... hold down the Ctrl key then type "20:33
sphoortii tried that fisrt like ctrl key and shift-720:33
*** sayali_ has quit IRC20:33
eglynnsphoorti: (i.e. Ctrl and double-quote together)20:33
eglynnsphoorti: by & I meant 'as well as'20:34
sphoortioops.. sorry :(20:34
eglynnsphoorti: np!20:34
sphoortiokay i ll do that right away20:34
eglynnsphoorti: here's another of navigating20:34
eglynnsphoorti: hold down Ctrl then type p20:35
eglynnsphoorti: that moves to the previous screen20:35
eglynnsphoorti: do that a couple of times of you'll get to the collector screen20:35
sphoortibtw eglynn at the bottom of my terminal i see 25 $ ceilometer-api 26$ and so on20:36
eglynnsphoorti: k, repeat until you get to collector20:36
eglynnsphoorti: there yet?20:36
sphoortii need to press enter every time i do ctrl p right?20:38
eglynnno need for enter, ctrl p should get to the previous screen each time20:38
sphoortii dont think that is workig20:38
eglynnsphoorti: sorry, my finger memory and my typing are not in sync20:39
eglynnsphoorti: k, rewind20:39
sphoortii tried pressing ctrl key and p20:40
eglynnsphoorti: yep, my fault!20:40
sphoortibut the screen remains the same20:40
eglynnsphoorti: I meant 'ctrl and a together then hit p'20:40
* eglynn is struggling with explaining commands that his fingers type automatically ;)20:41
sphoortisorry eglynn :(20:41
eglynnsphoorti: no, my fault20:41
eglynnsphoorti: so ... hold down the ctrl key and the a character, then hit p20:42
sphoortieglynn, long press ctrl key and p simultaneously gives me ^p20:42
sphoortictrl key press a then hit p20:42
eglynnsphoorti: just do ctrl *with* a, then release, then hit p20:42
*** shakayumi has quit IRC20:43
eglynnsphoorti: similarly, to get the window list ... ctrl *with* a, then release, then hit "20:43
eglynnsphoorti: working?20:44
sphoortieglynn, perfectly working20:44
sphoortii worked out with "20:44
sphoortiand now i am on collecter screen20:45
sphoortiscreen 24 :)20:45
eglynnk, now you're on the correct screen, so detach by ... ctrl *with* a, then release, then hit d20:45
eglynnso back to your original shell20:45
eglynnthen run a neutron command20:45
*** shakayumi has joined #openstack-metering20:45
sphoortiyes eglynn back to original shell.20:46
*** thomasem has quit IRC20:46
eglynnsphoorti: neutron router-create router120:47
eglynnsphoorti: or, neutron net-create net120:47
eglynnsphoorti: ... something of that ilk20:48
eglynnsphoorti: then back to the collector screen20:48
eglynnsphoorti: via ... screen -r20:48
sphoortieglynn, i also struggle with this thing... whenever i am going to use these commands like neutron it asks me for os-username20:48
sphoortiunder such circumstance --os-username = admin is the correct syntax?20:49
eglynnsphoorti: you need to set the OS_* env variables with the username etc that you provided when running devstack20:49
sphoortihow does one do that eglynn ?20:49
eglynnsphoorti: ...20:50
eglynnexport OS_USERNAME=username20:50
eglynn export OS_PASSWORD=password20:50
eglynn export OS_TENANT_NAME=admin20:50
eglynn export OS_AUTH_URL=http://localhost:5000/v2.020:50
eglynnsphoorti: replace username with the actual username you specified, similarly for password20:51
sphoortion commanline eglynn ?20:51
eglynnsphoorti: yep20:51
sphoortiokay doing that20:51
*** chuck_ has joined #openstack-metering20:51
eglynnsphoorti: look in your localrc if you forget the username, password etc.20:51
sphoortieglynn, username can be admin as well as demo right?20:53
eglynnsphoorti: sure, export OS_USERNAME=admin20:53
*** zul has quit IRC20:53
eglynnexport OS_USERNAME=sphoorti's_favourite_password20:54
eglynnexport OS_TENANT_NAME=admin20:54
eglynn export OS_AUTH_URL=http://localhost:5000/v2.020:54
eglynnneutron router-create router120:54
eglynnscreen -r20:54
sphoortirouter has been created and i see a tabular display20:55
eglynnsphoorti: screen -r20:55
eglynnsphoorti: (to go back to the collector screen)20:56
eglynnsphoorti: then go into copy mode so you can srach back20:56
eglynnsphoorti: ctrl *with* a, then release, then hit [20:56
eglynnsphoorti: then you can page up & down20:56
eglynnsphoorti: and text search backwards with '?'20:57
*** nikhil___ is now known as |_nikhil_20:57
eglynnsphoorti: s/srach back/search backwards/20:57
sphoortieglynn, copy mode activated20:58
eglynnsphoorti: \o/20:58
sphoortinow what do i have to search?20:58
eglynnsphoorti: so have a look back at notifications.py20:59
eglynnsphoorti: look for a method called event_types20:59
eglynnsphoorti: that binds the notification handler to particular notification events21:00
sphoortieglynn, isnt there a simpler method than moving up the screen?21:00
sphoortiyes event_types is the first error line in my traceback21:00
eglynnsphoorti: page up21:00
eglynnsphoorti: so search for create.end21:01
sphoortinothing as easy as the ctrl a and "21:01
eglynnsphoorti: it's getting late here (9pm), I need to drop off and put my kids to bed21:01
eglynnsphoorti: do you have enough to go on for now?21:01
sphoortieglynn, could you just explain me the flow? i ll try working around it21:02
*** chuck_ is now known as zul21:02
eglynnsphoorti: which flow?21:02
*** nati_ueno has quit IRC21:02
*** julim has quit IRC21:02
sphoortiwhen i find the file in my screen -r what do i do next?21:03
*** nati_ueno has joined #openstack-metering21:03
eglynnsphoorti: bbiab21:03
sphoortiokay eglynn21:03
*** thomasem has joined #openstack-metering21:11
*** thomasem has quit IRC21:11
*** thomasem has joined #openstack-metering21:12
*** SergeyLukjanov has joined #openstack-metering21:12
eglynnsphoorti: search backward in the collector logging output for 'router1'21:17
eglynnsphoorti: if not already in copy mode: ctrl *with* a, then release, then hit [21:17
eglynnsphoorti: then type ?router121:17
eglynnsphoorti: got it?21:18
sphoortieglynn, is not in collector folder it is seen networks21:18
sphoortias far as i remember21:19
eglynnsphoorti: you're confusing screens and directories21:20
sphoortibut eglynn when i am in copy mode and i do ? or ?router121:20
eglynnsphoorti: but either way, can you search thru' the collector log in the screen session?21:20
sphoortino pattern found21:20
eglynnsphoorti: did you create a router called router1?21:21
sphoortiyes eglynn21:21
sphoortiand i saw the table too21:21
sphoortiand the router1 status was active in the table21:21
eglynnsphoorti: create another one, maybe the screen buffer has recycled21:21
eglynnsphoorti: detach from the screen session first21:21
sphoortiokay eglynn21:22
eglynnsphoorti: then create the 2nd router21:22
eglynnsphoorti: then resume the screen21:22
sphoortieglynn, by mistake i wrote in the screen only screen -d.. how do you delete such lines?21:23
eglynnsphoorti: no need to delete21:24
eglynnsphoorti: ctrl *with* a, then release, then hit d21:24
sphoortieglynn, two routers are there now21:25
sphoortii got a tabular display for this one too21:25
eglynnsphoorti: screen -r again21:25
sphoortiand go to collector screen right?21:26
eglynnsphoorti: look for a notification related to the second router in the collector log21:26
eglynnsphoorti: yes21:26
sphoortieglynn, so for that i have to do ?router2 right?21:27
eglynnsphoorti: copy mode first21:27
eglynnsphoorti: ctrl *with* a, then release, then hit [21:27
eglynnsphoorti: then type ?router221:27
sphoortiyes eglynn did that :) sorry forgot to write that21:28
sphoortieglynn, router2 found21:28
eglynnsphoorti: what's the notification payload?21:29
*** SergeyLukjanov is now known as _SergeyLukjanov21:29
*** _SergeyLukjanov has quit IRC21:29
eglynnsphoorti: can you copy & paste?21:30
sphoortieglynn, i ll try doing that21:30
sphoortii am on virtualbox and it wont get copied on my nativemachine21:31
sphoortiany expert solution?21:33
sphoortieglynn, ?21:33
eglynnsphoorti: hmmm, can you include the previous line also?21:38
eglynnsphoorti: note ... u'tenant_id': u'987b29aa5e014b94a0cea884b69b7eee'21:39
sphoortiokay eglynn i ll include the previous line too21:39
eglynnsphoorti: ok, that's the messge with the sample as opposed to the raw notification21:43
eglynnsphoorti: seach back again for router221:43
sphoortiagain ?21:44
eglynnyes again21:44
eglynnsphoorti: what TZ are you in?21:44
eglynnsphoorti: TZ=timezome21:44
sphoortiGMT +521:44
eglynnsphoorti: k, I'm GMT+021:45
eglynnso getting very late here, I'm gonna have to drop off very shortly21:46
sphoortiits 3:15 am here  :p21:46
*** krtaylor has joined #openstack-metering21:46
sphoortieglynn, and i searched again fo router221:47
sphoortiwhat different thing was i supposed to find? eglynn ?21:49
eglynnsphoorti: the actual incoming AMQP notification is what you're looking for ... what did you find?21:50
*** SergeyLukjanov has joined #openstack-metering21:50
sphoortiits the same as previous one21:51
eglynnand, did you search again perchance?21:51
sphoortiyes eglynn21:52
eglynn.... and what did you find?21:52
sphoortieglynn, previously you told me to note my tenant_id21:53
sphoortiits the same21:53
eglynnsphoorti: ok, so what you've proven there is?21:56
eglynnsphoorti: come on, think!21:56
eglynnsphoorti: the router.create.end notification includes what?21:56
sphoortimy tenant_id remains same21:56
sphoortimy tenant_id ?21:57
eglynnsphoorti: it includes a tenant ID, that's the important thing21:57
eglynnsphoorti: so where is the tenant ID contained?21:57
eglynnsphoorti: is it where ceilometer expects?21:58
eglynnsphoorti: where does ceilometer expect it to be?21:58
eglynnsphoorti: remember the file you looked at earlier21:58
sphoorti eglynn , yes21:58
eglynnsphoorti: ... project_id=message['payload']['tenant_id']21:58
eglynnsphoorti: so is it in the payload?21:59
eglynnsphoorti: search for payload21:59
eglynnsphoorti: what do you see21:59
sphoortipayload gives me pattern not found21:59
eglynnsphoorti: if you see something like u'payload': { ... u'tenant_id': ... }21:59
eglynnsphoorti: you've probably run out of buffer again22:00
*** nati_ueno has quit IRC22:00
eglynnsphoorti: ok, here's what you need to do22:00
sphoortii ll redo it22:00
sphoortieglynn, i could ctrl f on the pastebin page too22:00
eglynnsphoorti: ... for each of the neutron notification types, ensure that the tenant ID is present as ceilo expects it22:00
eglynnsphoorti: ... do this either by:22:01
eglynnsphoorti: (a) causing those notifications to be emitted by nuetron and examining the collector logs22:01
eglynnsphoorti: or22:01
eglynnsphoorti: (b) by code inspection on the neutron side22:01
sphoortieglynn, i ran a ctrl f but it didnt give me payload found22:02
eglynnsphoorti: you need to find the case where the tenant ID is missing, or not in the expected place in the payload22:02
eglynnsphoorti: ctrl f?22:02
*** sdake has quit IRC22:04
sphoortieglynn, i created a router22:04
sphoortibut ?payload - not found22:04
sphoortieglynn, ?22:07
eglynnsphoorti: one sec22:07
thomasemhey eglynn, is there a way for me to restore this change? John Herndon is on his way to Hong Kong, but I need this change restored so I can submit a fixed branch:
thomasemI wanted to keep the history and everything with that branch; it looks like only the owner can restore. It will be alright, I was just curious. Otherwise, I'll send John an email and see if he gets a chance to click the button.22:09
eglynnthomasem: I think only the owner or a gerrit admin22:13
eglynnsphoorti: OK, back22:13
eglynnsphoorti: so, the logs don't include logging of the actual payload22:13
eglynnsphoorti: so, what do you suggest we do?22:13
eglynnsphoorti: if the logging we need isn't there?22:13
thomasemeglynn, Gotcha. Thanks anyway. I need to take off, got a concert tonight. Anyway - have an awesome weekend everyone, and good luck sphoorti!22:13
sphoortithank you thomasem :)22:14
eglynnsphoorti: we add the logging, right?22:14
sphoortiyes eglynn22:14
eglynnsphoorti: so which source file needs to be changed?22:15
eglynnsphoorti: exactly! ceilometer/network/notifications.py22:16
eglynnsphoorti: so 3 lines need to be added22:16
eglynnsphoorti: sorry, actually just one line22:16
eglynnsphoorti: the imports are already there22:17
sphoortieglynn, which line then?22:17
eglynnsphoorti: look for a line like:'network notification %r', message)22:17
eglynnsphoorti: in  process_notification()22:18
eglynnsphoorti: why isn't that line do what we expect?22:18
*** zul has quit IRC22:19
*** thomasem has quit IRC22:19
sphoortieglynn, found the line22:19
sphoortieglynn, is it because i wrote .get(tenant_id,None)>?22:21
eglynnsphoorti: the question is, why isn't that line appearing the collector log?22:21
eglynnsphoorti: (genuine question)22:21
eglynnsphoorti: let's check if it is22:22
sphoortii am wondering that too :):P eglynn :P22:22
eglynnsphoorti: k, create yet another router22:22
eglynnsphoorti: then back to the collector log22:22
eglynnsphoorti: then copy mode22:22
eglynnsphoorti: then ?network notification22:23
sphoortithe same process again22:23
sphoortieglynn, i created a router322:25
sphoortibut when i am trying to find it pattern not found22:25
sphoortiand so for network notification22:26
sphoortithough i find router2 , the old one22:26
sphoortieglynn, i found router3 this time but no network notification22:27
*** thomasem has joined #openstack-metering22:29
sphoortino network notification found eglynn :(22:30
eglynnsphoorti: I really have to drop off now ... I'll spin up a fresh neutron-based devstack myself tmrw and figure out the logging the aspect ... I'll send you the details22:31
eglynnsphoorti: email?22:31
sphoortieglynn, sure22:31
sphoortimy email id is sphoorti.joglekar@gmail.com22:31
sphoortieven i ll grab some sleep.. its 4 am here22:32
sphoortithank you eglynn for all the help :)22:32
sphoortieglynn, would you be on irc tomorrow?22:33
*** thomasem has quit IRC22:34
eglynnsphoorti: I'll be on & off IRC tmrw, mostly off TBH22:35
eglynnsphoorti: then travelling to HK Sunday22:36
sphoortiokay eglynn22:36
sphoortihow should i get going with this bug now eglynn ?22:37
sphoortithomasem would be around?22:37
sphoortion the weekend?22:37
*** SergeyLukjanov has quit IRC23:20
*** SergeyLukjanov has joined #openstack-metering23:35
*** SergeyLukjanov is now known as _SergeyLukjanov23:37
*** _SergeyLukjanov has quit IRC23:37
*** SergeyLukjanov has joined #openstack-metering23:41

Generated by 2.14.0 by Marius Gedminas - find it at!