| anteaya | so has gerrit events per hour | 00:00 |
|---|---|---|
| *** bknudson has quit IRC | 00:00 | |
| *** hdd has quit IRC | 00:00 | |
| *** maurosr has quit IRC | 00:00 | |
| anteaya | and both of them just dropped | 00:00 |
| *** hdd_ has joined #openstack-infra | 00:00 | |
| *** dstanek has quit IRC | 00:00 | |
| clarkb | jog0: btw stopping everything has allowed logstash to catch up | 00:00 |
| *** khyati_ has quit IRC | 00:00 | |
| jeblair | clarkb: heh | 00:01 |
| *** maurosr has joined #openstack-infra | 00:01 | |
| *** bknudson has joined #openstack-infra | 00:01 | |
| clarkb | so now we should have a decent baseline of its ability to keep up | 00:01 |
| anteaya | gate jobs are all running now, check jobs should start | 00:01 |
| clarkb | jeblair: :) | 00:01 |
| anteaya | ha ha ha | 00:01 |
| jog0 | clarkb: kk | 00:01 |
| *** dpyzhov has quit IRC | 00:03 | |
| *** ociuhandu has quit IRC | 00:05 | |
| *** markmcclain has quit IRC | 00:05 | |
| clarkb | mordred: are you going to address fungi's comment? | 00:06 |
| * fungi is back. wow scrollback--i wasn't gone *that* long ;) | 00:08 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Fix more str!=int bugs https://review.openstack.org/75244 | 00:09 |
| *** markmcclain has joined #openstack-infra | 00:09 | |
| jeblair | fungi, clarkb: i believe we have too many nodes in building state due to that bug ^; will apply and restart | 00:09 |
| clarkb | looking | 00:09 |
| mordred | clarkb: fungi gave me a comment? | 00:09 |
| clarkb | mordred: yes | 00:09 |
| clarkb | on the pypy pip thing | 00:09 |
| mordred | yes. I have addressed it locally | 00:10 |
| mordred | but I have gone back to do more testing | 00:10 |
| clarkb | ah ok | 00:10 |
| mordred | and I cannot reproduce the pypy error on an unmodified py3k box | 00:10 |
| *** hogepodge has quit IRC | 00:10 | |
| mordred | so, while I like running the script I wrote, I'm not so sure that it, you know, fixes anything | 00:10 |
| mordred | especially since, upon further reflection, the system setuptools should not matter | 00:11 |
| jeblair | lifeless: it's possible the image build failure is related to https://review.openstack.org/75244 | 00:11 |
| clarkb | mordred: gotcha | 00:11 |
| *** dstanek has joined #openstack-infra | 00:12 | |
| lifeless | jeblair: one of the templates hit snapshot point | 00:13 |
| lifeless | jeblair: I'm about to check logs to see why there is no such image | 00:13 |
| lifeless | but, food in face hole first | 00:13 |
| jeblair | lifeless: oh well, maybe not then. i just restarted nodepool and deleted the building images; so those should be restarting now | 00:13 |
| clarkb | jeblair: for changes like https://review.openstack.org/#/c/75206/1 if I reapprove we should queue into check then retest and if passing there go into gate right? | 00:14 |
| clarkb | jeblair: I thought we fixed that interaction but maybe that was a different one o nthe approval | 00:14 |
| lifeless | jeblair: I'm not presuming we have a bug in the cloud, just noting that I saw a template hit 'snapshot pending' | 00:14 |
| jeblair | clarkb: not with a -1; that needs recheck. we fixed it for -2; i don't think we did that for -1 | 00:14 |
| *** hogepodge has joined #openstack-infra | 00:15 | |
| clarkb | ah | 00:15 |
| clarkb | I will shepherd those through | 00:15 |
| *** hogepodge has quit IRC | 00:15 | |
| *** ociuhandu has joined #openstack-infra | 00:16 | |
| clarkb | jeblair: I can propse the fix too and you can tell me if I have done it correctly | 00:16 |
| jeblair | clarkb: ok. | 00:16 |
| *** jroovers has quit IRC | 00:17 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add tempest coverage job to post https://review.openstack.org/75250 | 00:18 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Have zuul enqueue changes when approved https://review.openstack.org/75251 | 00:18 |
| clarkb | jeblair: ^ I think that should do it | 00:18 |
| *** sabari has quit IRC | 00:18 | |
| *** alop has joined #openstack-infra | 00:19 | |
| anteaya | we have one or two tests running in check now | 00:20 |
| *** e0ne has joined #openstack-infra | 00:21 | |
| anteaya | that's starting to look better | 00:22 |
| *** dhellmann has joined #openstack-infra | 00:22 | |
| *** CaptTofu has joined #openstack-infra | 00:23 | |
| *** ociuhandu has quit IRC | 00:24 | |
| *** changbl has quit IRC | 00:24 | |
| fungi | clarkb: lgtm. it's worth noting that at this point the ban on reverify no bug is moot because we've effectively already made recheck into reverify (by directing zuul to gate on aprv&&vrfy=+1) | 00:25 |
| clarkb | fungi: yeah | 00:25 |
| *** StevenK_ is now known as StevenK | 00:25 | |
| lifeless | jeblair: I just manually snapshotted a running server | 00:25 |
| sdague | oh, complete non-sequitor, except it seems like good news might be welcome | 00:25 |
| lifeless | jeblair: worked fine, | 00:25 |
| lifeless | jeblair: so its at the nodepool end | 00:25 |
| *** StevenK is now known as Guest99695 | 00:26 | |
| sdague | during the tempest meeting today we discovered that neutron full parallel jobs pass some times | 00:26 |
| *** e0ne has quit IRC | 00:26 | |
| sdague | which surprised us all | 00:26 |
| sdague | and is kind of awesome | 00:26 |
| fungi | sdague: does that mean that openstack works sometimes with neutron too? | 00:26 |
| sdague | yes, yes it does :) | 00:26 |
| *** Guest99695 is now known as StevenK | 00:26 | |
| lifeless | fungi: hah! ci-overcloud is neutron | 00:26 |
| sdague | so we're going to get that off of experimental, into non-voting check, make sure it's stable | 00:26 |
| fungi | lifeless: you are proving the point! | 00:27 |
| sdague | lifeless: .... that's not a ringing endorsement :) | 00:27 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/zuul: Add subject into json and change https://review.openstack.org/75252 | 00:27 |
| *** dkliban has joined #openstack-infra | 00:27 | |
| harlowja | ok, jeblair i think that will do it (not a zuul expert, ha) | 00:28 |
| anteaya | sdague: yay? | 00:28 |
| sdague | anteaya: definite yay | 00:28 |
| anteaya | yay | 00:28 |
| sdague | that's actually further than I thought we were | 00:28 |
| sdague | much further | 00:28 |
| anteaya | how do we get more consistentcy? | 00:28 |
| anteaya | what is the next step | 00:28 |
| *** hogepodge has joined #openstack-infra | 00:28 | |
| anteaya | and yay again | 00:28 |
| clarkb | fungi: https://review.openstack.org/#/c/72751/ am I on base there? | 00:28 |
| sdague | move it to non-voting check, which mtreinish was going to propose | 00:28 |
| clarkb | we were explicitly pinged for review on that one, but it doesn't actually fix anything imo | 00:29 |
| anteaya | sdague: can you include me on the patch? | 00:29 |
| alop | well, whatever it was, good work guys | 00:30 |
| *** ociuhandu has joined #openstack-infra | 00:30 | |
| *** alop has left #openstack-infra | 00:30 | |
| sdague | anteaya: yeh, as soon as he posts it | 00:31 |
| *** chris_johnson is now known as wchrisj|away | 00:31 | |
| *** matsuhashi has joined #openstack-infra | 00:31 | |
| anteaya | sdague: thanks | 00:32 |
| *** cadenzajon has quit IRC | 00:32 | |
| anteaya | sdague: this tempest test really went sideways in the gate: https://review.openstack.org/#/c/74954/ | 00:33 |
| *** oubiwann has joined #openstack-infra | 00:34 | |
| *** hashar has quit IRC | 00:35 | |
| *** dkliban has quit IRC | 00:36 | |
| clarkb | jeblair: where are we on nodepool? it look stuck building again | 00:36 |
| jeblair | clarkb: i'm not certain that's the case | 00:37 |
| *** lcheng has joined #openstack-infra | 00:37 | |
| clarkb | AllocationSubRequest for 631.630434783 (out of 745.0) of devstack-precise from rax-dfw is pretty awesome | 00:37 |
| clarkb | jeblair: hrm maybe not, I do see a bunch of adding node to jenkins messages | 00:38 |
| lifeless | its a shame gate-noop has to queue | 00:38 |
| mordred | it's also fun | 00:38 |
| lifeless | jeblair: so, can you get me another image build log? since its not the cloud, it must be nodepool or the prepare scripts | 00:39 |
| jeblair | i think zuul should handle that internally, but i don't have time to write that patch right now | 00:39 |
| jeblair | lifeless: an image is building now. btw, that image build log is not static -- that's the real thing, so you can just fetch it again later | 00:40 |
| lifeless | jeblair: ok, cool | 00:40 |
| lifeless | jeblair: yes it seems to be looping on building images | 00:40 |
| jeblair | lifeless: well at least once was due to a restart | 00:40 |
| mordred | lifeless: you don't have a proxy between you and nodepool do you? | 00:40 |
| *** hogepodge has quit IRC | 00:41 | |
| lifeless | mordred: no btw that fix should go into requests/novaclient, not nodepool IMNSHO | 00:41 |
| clarkb | lifeless: in the proxy imo | 00:41 |
| mordred | lifeless: I agree - but I need that fix sooner rather than later | 00:41 |
| lifeless | clarkb: proxies are allowed to disconnect idle connections | 00:41 |
| lifeless | clarkb: its expected, and telling squid it can't do that - hahahah. NO. | 00:42 |
| clarkb | is it idle though? maybe I misunderstood | 00:42 |
| mordred | yes | 00:42 |
| jeblair | clarkb: i think the provider workers (or the providers themselves) are a bit backed up. | 00:42 |
| clarkb | I thought it was an active connection for novaclient to do stuff | 00:42 |
| mordred | nope | 00:42 |
| clarkb | jeblair: gotcha, so we are waiting for that to get through | 00:42 |
| mordred | it's a cached connection in case novaclient wants to do more stuff | 00:42 |
| clarkb | mordred: ah | 00:42 |
| clarkb | mordred: I thought it was being killed while doing a snapshot | 00:42 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Parameterize the status page urls https://review.openstack.org/74557 | 00:42 |
| *** lttrl has quit IRC | 00:42 | |
| mordred | so I think it's actually a novaclient bug | 00:43 |
| mordred | clarkb: it dies trying to do a snapshot | 00:43 |
| clarkb | mordred: but you are saying it is killed before novaclient goes to do that because connection was dropped | 00:43 |
| mordred | clarkb: because it tries to do it on a connection that the proxy has idle'd off | 00:43 |
| clarkb | because it was idle before hand | 00:43 |
| jeblair | mordred: you may be able to get it into novaclient faster than it will make it through the infra review queue | 00:43 |
| mordred | jeblair: this is an excellent point | 00:43 |
| mordred | jeblair: but I understand the nodepool codebase better :) | 00:43 |
| *** tjones has quit IRC | 00:44 | |
| *** tjones has joined #openstack-infra | 00:44 | |
| jeblair | clarkb: yeah, i think the queue is just full of create server tasks | 00:45 |
| jeblair | (this is where the num_instances thing would really be nice) | 00:45 |
| *** hogepodge has joined #openstack-infra | 00:46 | |
| *** dkehn__ has joined #openstack-infra | 00:46 | |
| clarkb | num_instances? | 00:46 |
| *** salv-orlando_ has joined #openstack-infra | 00:47 | |
| clarkb | batched boot requests? | 00:47 |
| anteaya | sdague: 74954 is failing again, I think it should come out, what do you think? | 00:47 |
| *** oubiwann has quit IRC | 00:47 | |
| *** matsuhas_ has joined #openstack-infra | 00:48 | |
| *** nati_uen_ has joined #openstack-infra | 00:48 | |
| *** tteggel_ has joined #openstack-infra | 00:48 | |
| *** nijaba_ has joined #openstack-infra | 00:48 | |
| *** tjones has quit IRC | 00:49 | |
| sdague | anteaya: yeh, agreed | 00:49 |
| sdague | it looks like a compile error of some sort | 00:49 |
| *** dkehn___ has joined #openstack-infra | 00:49 | |
| *** guitarza1 has joined #openstack-infra | 00:49 | |
| anteaya | sdague: do you want to do it? | 00:49 |
| sdague | not really :) | 00:49 |
| sdague | go for it | 00:49 |
| *** AaronGreen has joined #openstack-infra | 00:49 | |
| anteaya | k | 00:50 |
| *** openstack_ has joined #openstack-infra | 00:50 | |
| *** notmyname_ has joined #openstack-infra | 00:50 | |
| *** hub_cap_ has joined #openstack-infra | 00:50 | |
| *** Daviey_ has joined #openstack-infra | 00:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: fix conditional with mkdir -p https://review.openstack.org/75203 | 00:52 |
| *** pmathews1 has quit IRC | 00:52 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Use the task manager to get extensions and flavors https://review.openstack.org/75206 | 00:53 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Node deletion related fixes https://review.openstack.org/75217 | 00:53 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Coerce all ids from novaclient to str https://review.openstack.org/75218 | 00:53 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Remove unhelpful log message https://review.openstack.org/75223 | 00:53 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Fix more str!=int bugs https://review.openstack.org/75244 | 00:53 |
| *** nati_ue__ has joined #openstack-infra | 00:53 | |
| clarkb | jeblair: ^ I think that means you can turn puppet back on | 00:53 |
| *** jamespage has joined #openstack-infra | 00:53 | |
| *** jamespage has joined #openstack-infra | 00:53 | |
| jeblair | clarkb: it's on (it only installs when the repo changes anyway) | 00:54 |
| clarkb | oh right | 00:54 |
| *** ruhe- has joined #openstack-infra | 00:54 | |
| *** nati_uen_ has quit IRC | 00:54 | |
| *** mayu has joined #openstack-infra | 00:54 | |
| *** SergeyLukjanov2 has joined #openstack-infra | 00:55 | |
| jeblair | clarkb: also, check the graph; seems to be catching up | 00:55 |
| *** matsuhashi has quit IRC | 00:55 | |
| *** nati_ueno has quit IRC | 00:55 | |
| *** nijaba has quit IRC | 00:55 | |
| *** jaypipes has quit IRC | 00:55 | |
| *** jamespag` has quit IRC | 00:55 | |
| *** tteggel has quit IRC | 00:55 | |
| *** jgrimm has quit IRC | 00:55 | |
| *** salv-orlando has quit IRC | 00:55 | |
| *** dkehn_ has quit IRC | 00:55 | |
| *** guitarzan has quit IRC | 00:55 | |
| *** dkehn has quit IRC | 00:55 | |
| *** Xurong has quit IRC | 00:55 | |
| *** lifeless has quit IRC | 00:55 | |
| *** SergeyLukjanov has quit IRC | 00:55 | |
| *** ruhe has quit IRC | 00:55 | |
| *** katyafervent_awa has quit IRC | 00:55 | |
| *** notmyname has quit IRC | 00:55 | |
| *** AaronGr has quit IRC | 00:55 | |
| *** mattoliverau has quit IRC | 00:55 | |
| *** ewindisch has quit IRC | 00:55 | |
| *** dtroyer has quit IRC | 00:55 | |
| *** Daviey has quit IRC | 00:55 | |
| *** hub_cap has quit IRC | 00:55 | |
| *** SergeyLukjanov2 is now known as SergeyLukjanov | 00:55 | |
| *** salv-orlando_ is now known as salv-orlando | 00:55 | |
| *** Daviey_ is now known as Daviey | 00:55 | |
| *** notmyname_ is now known as notmyname | 00:55 | |
| *** ruhe- is now known as ruhe | 00:55 | |
| *** tteggel_ is now known as tteggel | 00:56 | |
| *** katyafervent_awa has joined #openstack-infra | 00:56 | |
| clarkb | jeblair: woot looks happier, much more blue/purple | 00:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Have zuul enqueue changes when approved https://review.openstack.org/75251 | 00:57 |
| *** AaronGreen is now known as AaronGr | 00:57 | |
| *** dtroyer has joined #openstack-infra | 00:58 | |
| *** lifeless has joined #openstack-infra | 00:58 | |
| *** mattoliverau has joined #openstack-infra | 00:58 | |
| mayu | #anteaya hi | 00:58 |
| mayu | #jaypipes hi | 00:59 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add check-tempest-dsvm-neutron-full to runs https://review.openstack.org/75268 | 00:59 |
| sdague | so I realize infra is super firedrilled today | 00:59 |
| *** markwash has quit IRC | 01:00 | |
| sdague | however, https://review.openstack.org/75268 is kind of a huge step forward in neutron parity, so would be great to move into the system quickly | 01:00 |
| *** ewindisch has joined #openstack-infra | 01:00 | |
| *** yamahata has quit IRC | 01:00 | |
| *** dcramer_ has joined #openstack-infra | 01:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: gerritbot: create an #openstack-merges channel https://review.openstack.org/71319 | 01:00 |
| sdague | with the goal to drop the other neutron job by i3 and use this one instead, and finally have neutron on the same footing as all other projects | 01:01 |
| *** atiwari has quit IRC | 01:01 | |
| mayu | #jaypipes I have trouble on constructing external ci system following your blog. | 01:01 |
| *** jgrimm has joined #openstack-infra | 01:01 | |
| jeblair | sdague: lgtm; certainly merits a priority review i think | 01:01 |
| anteaya | mayu please move to the openstack-neutron channel | 01:01 |
| *** dtroyer has quit IRC | 01:02 | |
| *** lifeless has quit IRC | 01:02 | |
| *** mattoliverau has quit IRC | 01:02 | |
| clarkb | jeblair: sdague: is that job in a bunch of experimental queues? | 01:02 |
| jeblair | russellb: 01:03 < openstackgerrit> A change was merged to openstack-infra/config: gerritbot: create an #openstack-merges channel https://review.openstack.org/71319 | 01:02 |
| mayu | ok | 01:02 |
| *** dtroyer has joined #openstack-infra | 01:02 | |
| *** lifeless has joined #openstack-infra | 01:02 | |
| *** mattoliverau has joined #openstack-infra | 01:02 | |
| sdague | clarkb: the gate one is, I didn't bother to purge it yet | 01:02 |
| sdague | I can if you like in that review | 01:02 |
| clarkb | apparently not, I was going to suggest removing it from experimental queues if so | 01:02 |
| clarkb | sdague: no this is fine | 01:02 |
| sdague | well, it is | 01:02 |
| sdague | but gate- | 01:02 |
| sdague | instead of check- | 01:02 |
| clarkb | sdague: hrm, maybe we should for completeness | 01:03 |
| sdague | for... reasons? | 01:03 |
| sdague | ok, let me clean it up | 01:03 |
| clarkb | check- use different nodes | 01:03 |
| *** jaypipes has joined #openstack-infra | 01:03 | |
| clarkb | gate- is restricted to performant nodes | 01:03 |
| *** zehicle_at_dell has quit IRC | 01:04 | |
| fungi | sdague: 75268 says it's non-voting in the commit message, but i don't see it actually marked as non-voting | 01:05 |
| fungi | are you sure? | 01:05 |
| fungi | oh, wait | 01:05 |
| clarkb | mtreinish: still around? https://review.openstack.org/#/c/72385/4/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml do we need a new job for that? | 01:05 |
| sdague | fungi: it's been non voting | 01:06 |
| * fungi didn't match on the right patterns | 01:06 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add check-tempest-dsvm-neutron-full to runs https://review.openstack.org/75268 | 01:06 |
| sdague | ok, here is the version with the 3 experimental jobs pulled | 01:06 |
| fungi | sdague: yeah, i see it | 01:06 |
| *** dcramer_ has quit IRC | 01:07 | |
| fungi | and i agree with clarkb on getting it out of the experimental pipelines | 01:07 |
| *** andre__ has quit IRC | 01:07 | |
| fungi | and you did. i'm so very, very slow | 01:07 |
| jog0 | sdague fungi: quick e-r review https://review.openstack.org/#/c/75220/ | 01:07 |
| sdague | lgtm | 01:08 |
| *** wchrisj|away is now known as chris_johnson | 01:09 | |
| clarkb | sdague: https://review.openstack.org/#/c/72385/4 do you grok that change? the commit message implies it should be running neutron? but neutron isn't enabled | 01:09 |
| clarkb | oh wait, I think I get it | 01:09 |
| clarkb | all of our tempest runs are isolated now | 01:09 |
| clarkb | so add a non isolated job regardless of neutron/nova network | 01:09 |
| fungi | mordred: if you haven't tried holding a building py3k-precise node and running on that, i'll test it now | 01:10 |
| sdague | clarkb: yeh, so that | 01:10 |
| mordred | fungi: I ahve not tried that | 01:10 |
| mordred | fungi: I've only created new ones myself | 01:10 |
| sdague | basically, we no longer run any upstream versions non isolated any more | 01:10 |
| sdague | but people still try to run tempest that way | 01:11 |
| sdague | so we want a bitrot job | 01:11 |
| clarkb | sdague: got it thanks | 01:11 |
| clarkb | sdague: any chance you want to review https://review.openstack.org/#/c/72365/1 in that case? | 01:12 |
| clarkb | mtreinish's first change I linked to depends on ^ | 01:12 |
| *** dkliban has joined #openstack-infra | 01:13 | |
| clarkb | I am beginning to think we need to construct a single tempest command to run based on the crazy options d-g allows | 01:14 |
| clarkb | rather than using the simple switch | 01:14 |
| sdague | so I never remember if we want to do the d-g bit first, or the layout bit to test the d-g bit | 01:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1282795 https://review.openstack.org/75220 | 01:14 |
| clarkb | sdague: in this case because the job is periodic and not check/gate we want the d-g bit first | 01:14 |
| sdague | ok | 01:14 |
| clarkb | since we can't self test it | 01:14 |
| *** chris_johnson has quit IRC | 01:14 | |
| clarkb | that change is +2 from me but you are more on board with the intent so your review would be good too | 01:15 |
| sdague | yep, looks good, I just +Aed | 01:15 |
| *** weshay has joined #openstack-infra | 01:15 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/zuul: Re-add Zuul ref replication https://review.openstack.org/75272 | 01:15 |
| mordred | clarkb, jeblair: I need to test that - but I was hoping you could give me a once-over on if that's in the right direction | 01:16 |
| clarkb | sdague: and https://review.openstack.org/#/c/74131/ conflicts with your change? | 01:16 |
| *** openstackgerrit has quit IRC | 01:18 | |
| *** openstackgerrit has joined #openstack-infra | 01:19 | |
| kevinbenton | Hi, I have a question about the gate. Are the lower patches already rebased against the higher patches, or are they rebased against master at the time of starting the tests? If it's the latter, how are two patches which pass idividually but fail when combined prevented from getting into master? If it's the former, do all of the tests get reset and rebased when a patch higher up the queue fails verification? | 01:20 |
| clarkb | kevinbenton: each change is tested atop the others. If a failure happens everything behind the failure is stopped, the parents are updated and tests restarted | 01:20 |
| fungi | kevinbenton: the latter, and ytes | 01:20 |
| fungi | yes | 01:20 |
| sdague | clarkb: it does, but I actually think my change trumps the one mtreinish was doing, because he didn't realize how far we'd gotten | 01:21 |
| fungi | er, the former | 01:21 |
| clarkb | sdague: ok | 01:21 |
| fungi | kevinbenton: what clarkb said | 01:21 |
| kevinbenton | clarkb, fungi: thatnks | 01:21 |
| kevinbenton | thanks* | 01:21 |
| *** e0ne has joined #openstack-infra | 01:21 | |
| kevinbenton | now i can sleep better :-) | 01:21 |
| *** khyati has joined #openstack-infra | 01:21 | |
| clarkb | kevinbenton: http://docs.openstack.org/infra/publications/zuul/#%2818%29 | 01:22 |
| *** rfolco has joined #openstack-infra | 01:22 | |
| clarkb | if you click on that slide you get a nice animation | 01:22 |
| *** ociuhandu has quit IRC | 01:23 | |
| clarkb | mordred: I think it is on track. I do think it makes sense to put replication urls under the merger config section rather than a separate section | 01:24 |
| lifeless | jeblair: did those slaves attach properly ? | 01:24 |
| lifeless | I see a couple failed, I believe nova dos'd neutron. | 01:24 |
| mordred | clarkb: ++ | 01:24 |
| mordred | good call | 01:24 |
| kevinbenton | clarkb: cool. thanks | 01:25 |
| clarkb | kevinbenton: and you can watch it happen live on the zuul status page | 01:25 |
| clarkb | http://status.openstack.org/zuul/ | 01:26 |
| clarkb | if you end up fetching the zuul refs for those jobs you can see how the git graphs change | 01:26 |
| kevinbenton | yeah, i watch it there quite a bit | 01:26 |
| *** e0ne has quit IRC | 01:26 | |
| kevinbenton | how do i fetch the zuul refs? | 01:26 |
| clarkb | kevinbenton: if you look in the test logs you will see gerrit git prep fetching them and can copy pasta the command. But the basic method is go to a job parameter page eg https://jenkins02.openstack.org/job/gate-ceilometer-pep8/3385/parameters/? | 01:27 |
| clarkb | then git fetch $ZUUL_URL/$ZUUL_PROJECT $ZUUL_REF | 01:28 |
| clarkb | in the repo for that project | 01:28 |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: remove extra item from listGroups and listProjects methods https://review.openstack.org/70301 | 01:28 |
| clarkb | fetch the same ZUUL_REF in all projects and you get the state for one test iteration | 01:28 |
| fungi | poop. nodepool's hold status on a ready node does not prevent it from eventually being deleted | 01:28 |
| fungi | guess i need a used node | 01:28 |
| clarkb | fungi: it should but only for so long | 01:29 |
| clarkb | iirc the 8 hour timeout will eat it | 01:29 |
| clarkb | or some timeout | 01:29 |
| fungi | well, in this case so long was about 15 minutes | 01:29 |
| lifeless | fungi: / clarkb: so - has nodepool attached the slaves from the tripleo cloud correctly ? | 01:29 |
| clarkb | lifeless: no idea, let me look | 01:29 |
| lifeless | I see a bunch failed to spawn | 01:29 |
| kevinbenton | clarkb: cool. i definitely see how a transient unit test failure can really make the gate painfully innefficient | 01:30 |
| fungi | lifeless: yes, tripleo-precise-tripleo-test-cloud-1485787 for example is listed as in a ready state | 01:30 |
| lifeless | you can delete those - neutron got dos'ed by nova. I'm filing a bug on that | 01:30 |
| lifeless | jog0: ^ | 01:30 |
| clarkb | lifeless: https://jenkins05.openstack.org/computer/tripleo-precise-tripleo-test-cloud-1485434/ I think so | 01:30 |
| lifeless | ok, so can we merge the job enabling patches ? | 01:30 |
| clarkb | lifeless: yes I wanted jeblair to look at them though | 01:30 |
| clarkb | did he not do that? | 01:31 |
| clarkb | more for formality than anything else | 01:31 |
| clarkb | oh I see his comment is he was waiting | 01:31 |
| *** unicell1 is now known as unicell | 01:31 | |
| *** unicell has joined #openstack-infra | 01:31 | |
| clarkb | lifeless: approved | 01:31 |
| lifeless | jog0: https://bugs.launchpad.net/nova/+bug/1282842 | 01:31 |
| clarkb | lifeless: and I got both changes now | 01:32 |
| *** VijayT has joined #openstack-infra | 01:33 | |
| *** dkliban has quit IRC | 01:33 | |
| *** fifieldt has joined #openstack-infra | 01:33 | |
| *** jhesketh__ has quit IRC | 01:33 | |
| *** jhesketh_ has quit IRC | 01:34 | |
| *** nosnos has joined #openstack-infra | 01:34 | |
| fungi | mordred: i held a py3k-precise nodepool node and was able to reproduce (the problem) as the jenkins user in a fresh clone of python-swiftclient running '/usr/local/jenkins/slave_scripts/run-unittests.sh py openstack python-swiftclient' | 01:34 |
| fungi | mordred: i ran the same as the jenkins user on the precisepy3k-1.slave.o.o static system and it succeeds | 01:35 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Temporarily stop running tripleo seed/undercloud" https://review.openstack.org/75173 | 01:36 |
| fungi | mordred: the held node is 23.253.35.88 but i'm waiting to catch it transitioning into a used state so i can re-hold it | 01:36 |
| clarkb | fungi: actually cleanuponenode short curcuits if state is HOLD | 01:36 |
| jog0 | lifeless: odd | 01:36 |
| fungi | clarkb: i think the problem is the node going from hold->used | 01:36 |
| clarkb | fungi: oh, hrm | 01:37 |
| jog0 | we do have a neutron version of large ops | 01:37 |
| jog0 | but it uses fake virt | 01:37 |
| fungi | clarkb: i'm having a hard time finding an in-use slave, so holding one in a ready state | 01:37 |
| mordred | fungi: wtf | 01:37 |
| jog0 | which may not hit neutron as much | 01:37 |
| clarkb | fungi: oh so yeah I think you are write | 01:37 |
| clarkb | fungi: in the handle complete method it short circuits if state is HOLD | 01:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add experimental-tripleo checks for tripleo deps. https://review.openstack.org/73886 | 01:38 |
| clarkb | fungi: but the handleStart just sets state to USD | 01:38 |
| clarkb | fungi: I can write a quick fix for that I think it is simple | 01:38 |
| fungi | mordred: anyway, trivially reproducible there. not sure how the nodes nodepool.o.o is building differ from those you're building to test | 01:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Add non-isolated serial periodic tempest job https://review.openstack.org/72385 | 01:38 |
| mordred | fungi: this makes me unhappy | 01:38 |
| *** wenlock has quit IRC | 01:39 | |
| mordred | fungi: I'm going to run the fix script on the host | 01:39 |
| fungi | mordred: well, i haven't tried running your script on the system as root yet to see whether that solves it | 01:39 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: Improved timestamp parsing https://review.openstack.org/73741 | 01:39 |
| mordred | fungi: ok. I ran it. | 01:40 |
| fungi | mordred: but worth trying. let me offline it in its jenkins first before it tries to run something there | 01:40 |
| *** VijayT has quit IRC | 01:40 | |
| fungi | or that | 01:40 |
| mordred | fungi: :) | 01:40 |
| *** yaguang has joined #openstack-infra | 01:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Set periodic tempest jobs to run with master https://review.openstack.org/71940 | 01:40 |
| *** VijayT has joined #openstack-infra | 01:40 | |
| mordred | fungi: running test again | 01:41 |
| *** mayu has quit IRC | 01:41 | |
| fungi | i managed to offline it in jenkins03 before it got assigned any jobs | 01:42 |
| anteaya | sdague: please rebase https://review.openstack.org/#/c/75268/ | 01:42 |
| fungi | so it should be around for ~8 hours to test with | 01:42 |
| * fungi is about to the point of needing a scotch and a relaxing evening by the fire | 01:43 | |
| *** jnoller has joined #openstack-infra | 01:43 | |
| *** VijayT has quit IRC | 01:43 | |
| * anteaya pours fungi a scotch and adds another log to the fire | 01:43 | |
| *** kfox1111 has quit IRC | 01:43 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add check-tempest-dsvm-neutron-full to runs https://review.openstack.org/75268 | 01:44 |
| sdague | anteaya: yep, was just working on it | 01:44 |
| sdague | clarkb / fungi: when you get a chance, I merge conflicted against one of the tripleo patches ^^^^ | 01:45 |
| fungi | anteaya: not sure if you grew up with the muppet show, but http://youtu.be/KJNSWkgPr9g | 01:45 |
| *** khyati has quit IRC | 01:45 | |
| anteaya | sdague: ah sorry | 01:45 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Preserve HOLD state when job starts. https://review.openstack.org/75278 | 01:45 |
| clarkb | fungi: ^ that should do it | 01:45 |
| *** jhesketh_ has joined #openstack-infra | 01:46 | |
| *** jnoller has quit IRC | 01:46 | |
| fungi | clarkb: awesome--looking | 01:46 |
| *** rfolco has quit IRC | 01:46 | |
| *** jnoller has joined #openstack-infra | 01:46 | |
| clarkb | sdague: I ninja approved. the diff looked good to me | 01:47 |
| sdague | clarkb: thanks | 01:47 |
| fungi | clarkb: as did i | 01:47 |
| clarkb | fungi: bam | 01:47 |
| fungi | double-tap | 01:47 |
| *** alexpilotti has quit IRC | 01:47 | |
| clarkb | ok time to walk home | 01:47 |
| clarkb | back later | 01:47 |
| *** jhesketh__ has joined #openstack-infra | 01:47 | |
| anteaya | fungi: grew up with the muppets | 01:48 |
| sdague | fungi: on the real of the bizarre, before henson got the muppet show, he did a lot of commercial work. | 01:48 |
| anteaya | and I am a great shot, would have fired both times | 01:48 |
| sdague | including a bunch for IBM - https://www.youtube.com/watch?v=_IZw2CoYztk | 01:48 |
| *** jerryz has joined #openstack-infra | 01:48 | |
| clarkb | lol at my topic 'preserver-hold-state' | 01:48 |
| clarkb | typing is hard | 01:48 |
| sdague | totally sureal stuff | 01:48 |
| fungi | sdague: yeah, i have a lot of those on the extras for the early season boxed sets. they're great | 01:49 |
| fungi | the purina spots with rolf too | 01:49 |
| sdague | nice | 01:49 |
| fungi | they need to work out the licensing issues to get seasons >4 onto dvd | 01:50 |
| sdague | ah, that's why | 01:50 |
| fungi | er, >=4 | 01:50 |
| fungi | right now it's just 1-3 out on dvd | 01:50 |
| * fungi is done reliving his childhood for now | 01:51 | |
| jerryz | hi guys, i found my change is -1 because of the status of all jenkins jobs are lost | 01:51 |
| fungi | jerryz: recheck no bug | 01:51 |
| jerryz | will do | 01:52 |
| fungi | jerryz: stuff which was pending in the queue got dumped due to a zuul bug/incident | 01:52 |
| fungi | gearman response timeout on localhost, if memory serves | 01:52 |
| *** dcramer_ has joined #openstack-infra | 01:52 | |
| jerryz | fungi: the result of the jobs didn't get back to zuul? | 01:53 |
| fungi | jerryz: zuul lost the status of the jobs in this case, i believe | 01:53 |
| *** Sukhdev has quit IRC | 01:54 | |
| fungi | jerryz: see discussion in the channel history around 23:34 utc | 01:54 |
| jerryz | fungi: ok. thanks | 01:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Increase the timeout for check-tripleo-overcloud https://review.openstack.org/73986 | 01:54 |
| * fungi transitions to a lower-bandwidth system for the evening | 01:55 | |
| clarkb | so I started fiddling with dpms before leaving to try and get my laptop to switch back to the internal display... ugh | 01:55 |
| * clarkb does it the old fashion way | 01:55 | |
| lifeless | what, xrandr? | 01:55 |
| StevenK | xrandr | 01:55 |
| clarkb | lifeless: yes | 01:56 |
| clarkb | however xrandr doesn't work either | 01:56 |
| lifeless | headdesk | 01:56 |
| StevenK | Back in my day, xrandr hadn't been written yet! | 01:56 |
| lifeless | clarkb: you have a brick right? | 01:56 |
| clarkb | the internal display turns itself off off and requires physical toggling to come back | 01:56 |
| *** SumitNaiksatam has quit IRC | 01:56 | |
| clarkb | lifeless: no I have a folio now | 01:56 |
| clarkb | but brick did this too | 01:56 |
| clarkb | if I xrandr output internal --off and leave it all day --auto'ing it won't turn it back on | 01:57 |
| clarkb | I think there is something in acpi that needs kicking | 01:57 |
| clarkb | my cheap trick now is to close open the shell | 01:57 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Fix log Footer README's for 'check-tempest-dsvm' https://review.openstack.org/74238 | 01:58 |
| clarkb | I could run both monitors together now as this gpu doesnt suck | 01:59 |
| clarkb | but habits | 02:00 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/zuul: Re-add Zuul ref replication https://review.openstack.org/75272 | 02:01 |
| jhesketh__ | jeblair, clarkb, fungi: Further discussion on the openstack_ci db in config: https://review.openstack.org/#/c/73461/ | 02:02 |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex https://review.openstack.org/68042 | 02:03 |
| openstackgerrit | Zang MingJie proposed a change to openstack-infra/zuul: Use ssh to fetch packs instead of HTTP https://review.openstack.org/67858 | 02:04 |
| *** pcm_ has joined #openstack-infra | 02:06 | |
| clarkb | jhesketh__ about to walk home but tl;dr is old design was wrong and bad. would like to not manage any DBs however you make a good point about nova | 02:06 |
| openstackgerrit | A change was merged to openstack-infra/config: add check-tempest-dsvm-neutron-full to runs https://review.openstack.org/75268 | 02:06 |
| *** yamahata has joined #openstack-infra | 02:06 | |
| clarkb | so I can go either way but we should have testsuites just use the perms given them long run | 02:07 |
| jhesketh__ | yeah I agree, I just don't know what tests are expecting the database to be there | 02:07 |
| jhesketh__ | I guess I can do some diving | 02:07 |
| *** dkehn___ is now known as dkehn | 02:07 | |
| jhesketh__ | but don't let me hold you up :-) | 02:07 |
| lifeless | are infra git servers having trouble right now ? | 02:09 |
| lifeless | error: Unable to find 6f8f611cbeb83ea38563ca09abcde925e05e32fd under https://git.openstack.org/cgit/openstack/swift | 02:09 |
| lifeless | Cannot obtain needed tree 6f8f611cbeb83ea38563ca09abcde925e05e32fd | 02:09 |
| lifeless | while processing commit a94be9443d639dc0ac8498d8a35fed2832379ee7. | 02:09 |
| lifeless | error: Fetch failed. | 02:09 |
| lifeless | also error: Failed to connect to 2001:4800:7813:516:3bc3:d7f6:ff04:aacb: Network is unreachable (curl_result = 7, http_code = 0, sha1 = 71bf77edc31a9d9fa007c3d5898a7ae5bbf1b228) | 02:09 |
| lifeless | which may be from the same git call | 02:09 |
| lifeless | though its a little weird, as this machine doesn't ahvea public ipv6 addr on it | 02:10 |
| *** dkehn__ is now known as dkehn_ | 02:10 | |
| StevenK | I think the always trying ipv6 is a git oddity, but I've not investigated throughly | 02:12 |
| *** guitarza1 is now known as guitarzan | 02:17 | |
| *** yamahata has quit IRC | 02:19 | |
| *** yamahata has joined #openstack-infra | 02:19 | |
| *** ryanpetrello has quit IRC | 02:21 | |
| *** dkliban has joined #openstack-infra | 02:21 | |
| *** e0ne has joined #openstack-infra | 02:21 | |
| fungi | lifeless: don't use the cgit urls for git remotes | 02:21 |
| anteaya | 1 in the gate | 02:22 |
| anteaya | wow | 02:22 |
| *** zehicle_at_dell has joined #openstack-infra | 02:22 | |
| lifeless | fungi: hmmm, thats interesting, why are we | 02:22 |
| fungi | lifeless: cgit has some issues as a git server apparently (there is an openstack-ci bug open) | 02:23 |
| lifeless | stabbbity stab stab | 02:23 |
| lifeless | yes | 02:23 |
| fungi | drop the /cgit and should be just fine | 02:23 |
| fungi | apache hasn't been exhibiting this issue | 02:24 |
| *** sarob has joined #openstack-infra | 02:24 | |
| lifeless | there we go. https://review.openstack.org/75283 if you want to weigh in | 02:25 |
| *** e0ne has quit IRC | 02:26 | |
| fungi | lifeless: the network vnreachable for an ipv6 addr suggests those systems have a global v6 assignment and a broken gateway | 02:27 |
| fungi | oh, you said that | 02:27 |
| *** relaxdiego has quit IRC | 02:28 | |
| *** weshay has quit IRC | 02:28 | |
| *** Hunner has quit IRC | 02:33 | |
| *** matsuhas_ has quit IRC | 02:34 | |
| *** sarob has quit IRC | 02:36 | |
| lifeless | fungi: they don't have a ipv6 assignment :( | 02:36 |
| lifeless | just fe80 addresses | 02:36 |
| *** sarob has joined #openstack-infra | 02:36 | |
| *** matsuhas_ has joined #openstack-infra | 02:36 | |
| *** Hunner has joined #openstack-infra | 02:37 | |
| fungi | yeah, StevenK may be onto something | 02:37 |
| *** marun has quit IRC | 02:37 | |
| *** sarob_ has joined #openstack-infra | 02:39 | |
| *** sarob has quit IRC | 02:40 | |
| *** SumitNaiksatam has joined #openstack-infra | 02:41 | |
| *** Guest91627 is now known as persia | 02:42 | |
| *** gokrokve has joined #openstack-infra | 02:43 | |
| *** khyati has joined #openstack-infra | 02:45 | |
| *** markmcclain has quit IRC | 02:46 | |
| *** openstack_ is now known as Xurong | 02:47 | |
| anteaya | gate is empty | 02:47 |
| *** jishaom has joined #openstack-infra | 02:47 | |
| *** bhuvan has joined #openstack-infra | 02:47 | |
| *** starmer has joined #openstack-infra | 02:51 | |
| *** starmer has quit IRC | 02:52 | |
| *** jhesketh__ has quit IRC | 02:53 | |
| jishaom | @fungi hi fungi, would you please open our service account 'IBM DB2 Test', so we can post our result to community, thx. | 02:54 |
| *** jhesketh__ has joined #openstack-infra | 02:54 | |
| clarkb | jishaom: are you looking for voting ability? | 02:54 |
| clarkb | jishaom: I think that request needs to come from the community when it is happy with the existing results | 02:54 |
| *** jhesketh__ has quit IRC | 02:58 | |
| *** Daisy has joined #openstack-infra | 02:58 | |
| *** jhesketh__ has joined #openstack-infra | 02:58 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** thomasem has quit IRC | 02:59 | |
| *** Steap_ has quit IRC | 03:01 | |
| *** matsuhas_ has quit IRC | 03:03 | |
| *** matsuhas_ has joined #openstack-infra | 03:06 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Remove neutron isolated jobs https://review.openstack.org/71947 | 03:12 |
| *** relaxdiego has joined #openstack-infra | 03:13 | |
| pcm_ | Hi all. Had a question about requirements.txt | 03:14 |
| lifeless | clarkb: woo - we has tests running | 03:15 |
| pcm_ | I have a patch out for review, where I changed requests to 2.1.0 (from 1.1) in requirements.txt. Is that the proper approach, or should I do a separate patch for that change? | 03:15 |
| pcm_ | (this is for Neutron I-3 patch) | 03:15 |
| clarkb | pcm_: you have to make it matche openstack/requirements global-requirements.txt | 03:16 |
| *** jcooley_ has joined #openstack-infra | 03:17 | |
| pcm_ | the package is in requirements.txt, but the version there is 1.1, whereas I want to use 2.1.0 | 03:17 |
| *** mgagne has quit IRC | 03:17 | |
| clarkb | you have a change that updates http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n95 ? | 03:18 |
| clarkb | you needt o update ^ before you can update the individual projects | 03:18 |
| pcm_ | No. Sorry, I just updated requirements.txt in Neutron. | 03:19 |
| clarkb | right so you need to update global-requirements first | 03:19 |
| pcm_ | OK so in a separate patch, in that repo, update the file and push for review? | 03:20 |
| pcm_ | I also want to add a new package to test-requirements.txt (in Neutron). What is the procedure to update that? | 03:21 |
| clarkb | yes, please give some detail in the commit message (I think we need to do a better job of tracking changes so that packagers and the like don't go crazy) | 03:21 |
| *** e0ne has joined #openstack-infra | 03:21 | |
| clarkb | pcm_: same thing, update global-requirements, note the reason for it then update in neutron | 03:21 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix install-buck macro https://review.openstack.org/75293 | 03:23 |
| pcm_ | Gotcha. Thanks. Will work on doing this ASAP. | 03:23 |
| jishaom | @clarkb Hi, clarkb. I am running my jobs, then I will paste my result info for your checking, thx. | 03:23 |
| *** dolphm_503 is now known as dolphm | 03:25 | |
| pcm_ | clarkb: Since I've only worked in my little world of Neutron, where is the repo that I pull from/to? | 03:25 |
| clarkb | jishaom: do you have a third party account already? | 03:25 |
| * pcm_ usually I go to /opt/stack/neutron - wondering where this is | 03:25 | |
| clarkb | jishaom: if so you can post results jut without leaving a +1 or -1 | 03:25 |
| clarkb | pcm_: git clone https://git.openstack.org/openstack/requirements | 03:26 |
| pcm_ | clarkb: thanks! | 03:26 |
| *** e0ne has quit IRC | 03:26 | |
| *** pcrews_ has quit IRC | 03:28 | |
| *** mayu has joined #openstack-infra | 03:28 | |
| mayu | jaypipes: hi | 03:29 |
| *** matsuhas_ has quit IRC | 03:29 | |
| mayu | jaypipes: I follow your blog to contruct local ci, there is a trouble | 03:29 |
| mayu | jaypipes: can you help me ? | 03:30 |
| zaro | clarkb: anyway you can fast track this? https://review.openstack.org/#/c/75293 | 03:31 |
| clarkb | mayu: not sure if jaypipes is around but I may be able to help | 03:31 |
| *** david-lyle has joined #openstack-infra | 03:31 | |
| clarkb | zaro: looking | 03:31 |
| clarkb | zaro: tell me if this is a terrible idea, but we should get ramen tomorrow (nevermind catered lunch) | 03:32 |
| clarkb | zaro: fast tracked | 03:33 |
| zaro | clarkb: i'm down for that. samurai is good. | 03:33 |
| clarkb | zaro: awesome, I am craving ramen | 03:33 |
| clarkb | mayu: whats up? | 03:33 |
| *** jcooley_ has quit IRC | 03:34 | |
| *** dolphm is now known as dolphm_503 | 03:34 | |
| *** jcooley_ has joined #openstack-infra | 03:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: fix install-buck macro https://review.openstack.org/75293 | 03:36 |
| clarkb | zaro: ^ | 03:36 |
| mayu | clarkb: hi | 03:36 |
| mayu | clarkb: can you help me? | 03:36 |
| clarkb | mayu: maybe, what is the problem | 03:37 |
| mayu | http://paste.openstack.org/show/67849/ | 03:37 |
| mayu | I get an error when I using jay's script to deploy jenkins | 03:38 |
| mayu | It is the screen content | 03:38 |
| clarkb | mayu: https://review.openstack.org/#/c/74443/ you need that fix to the openstack-infra/config repo | 03:39 |
| *** jcooley_ has quit IRC | 03:39 | |
| openstackgerrit | Paul Michali proposed a change to openstack/requirements: Update requests to 2.10 and add httmock to tests https://review.openstack.org/75296 | 03:39 |
| mayu | clarkb: Is it the reason ? | 03:40 |
| clarkb | mayu: yes jaypipes commit message explains it | 03:40 |
| *** gyee has quit IRC | 03:40 | |
| clarkb | and I have reviewed his change now. | 03:40 |
| mayu | clarkb: thanks | 03:42 |
| *** dcramer_ has quit IRC | 03:43 | |
| *** ArxCruz has quit IRC | 03:43 | |
| clarkb | mayu: I am guessing that jaypipes os-repotestthing clones openstack-infra/config, if you can find where it cloned that you can apply his patch by fetching it | 03:44 |
| clarkb | the command to fetch it is on the chage | 03:44 |
| *** masayukig has quit IRC | 03:44 | |
| *** harlowja is now known as harlowja_away | 03:45 | |
| mayu | clarkb: ok | 03:45 |
| *** dstanek has quit IRC | 03:50 | |
| *** changbl has joined #openstack-infra | 03:53 | |
| *** miqui has quit IRC | 03:54 | |
| *** dkranz has joined #openstack-infra | 03:54 | |
| *** dstanek has joined #openstack-infra | 03:54 | |
| *** dcramer_ has joined #openstack-infra | 03:56 | |
| *** dpyzhov has joined #openstack-infra | 04:01 | |
| mayu | clarkb: I modify the init.pp manually, and exec install_master.sh again, now , I am here waiting for the result | 04:02 |
| *** CaptTofu has quit IRC | 04:02 | |
| *** dolphm_503 is now known as dolphm | 04:03 | |
| *** Daisy has quit IRC | 04:04 | |
| *** pmathews has joined #openstack-infra | 04:08 | |
| *** lcheng has quit IRC | 04:10 | |
| *** bhuvan has quit IRC | 04:13 | |
| *** dolphm is now known as dolphm_503 | 04:13 | |
| *** jishaom has quit IRC | 04:14 | |
| *** lcheng has joined #openstack-infra | 04:15 | |
| *** relaxdiego has quit IRC | 04:19 | |
| *** e0ne has joined #openstack-infra | 04:21 | |
| *** mgagne has joined #openstack-infra | 04:22 | |
| *** mrodden has quit IRC | 04:24 | |
| *** locke1051 has quit IRC | 04:24 | |
| *** talluri has joined #openstack-infra | 04:24 | |
| *** jnoller has quit IRC | 04:24 | |
| *** talluri has quit IRC | 04:25 | |
| *** matsuhashi has joined #openstack-infra | 04:25 | |
| *** dstanek has quit IRC | 04:25 | |
| *** talluri has joined #openstack-infra | 04:25 | |
| *** nati_ue__ has quit IRC | 04:25 | |
| *** dolphm_503 is now known as dolphm | 04:25 | |
| *** e0ne has quit IRC | 04:26 | |
| *** pcm_ has quit IRC | 04:31 | |
| *** dpyzhov has quit IRC | 04:34 | |
| *** dolphm is now known as dolphm_503 | 04:35 | |
| *** jcooley_ has joined #openstack-infra | 04:35 | |
| *** jcooley_ has quit IRC | 04:40 | |
| *** dpyzhov has joined #openstack-infra | 04:41 | |
| *** mgagne has quit IRC | 04:47 | |
| *** bhuvan has joined #openstack-infra | 04:52 | |
| *** dstanek has joined #openstack-infra | 04:54 | |
| *** Daisy has joined #openstack-infra | 04:55 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: fix indentation in install-buck macro https://review.openstack.org/75310 | 04:55 |
| openstackgerrit | daisy-ycguo proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex https://review.openstack.org/68042 | 04:56 |
| *** dpyzhov has quit IRC | 04:56 | |
| *** starmer has joined #openstack-infra | 04:59 | |
| *** oda-g_ has joined #openstack-infra | 05:02 | |
| *** jcooley_ has joined #openstack-infra | 05:03 | |
| *** zhiyan_ is now known as zhiyan | 05:04 | |
| *** fcarpenter has joined #openstack-infra | 05:05 | |
| *** mgagne has joined #openstack-infra | 05:06 | |
| *** dstanek has quit IRC | 05:07 | |
| *** jcooley_ has quit IRC | 05:11 | |
| *** michchap has quit IRC | 05:14 | |
| *** oda-g_ has quit IRC | 05:15 | |
| *** e0ne has joined #openstack-infra | 05:21 | |
| *** relaxdiego has joined #openstack-infra | 05:21 | |
| *** matsuhashi has quit IRC | 05:23 | |
| *** jroovers has joined #openstack-infra | 05:24 | |
| *** markwash has joined #openstack-infra | 05:24 | |
| *** e0ne has quit IRC | 05:26 | |
| *** dolphm_503 is now known as dolphm | 05:26 | |
| *** bhuvan has quit IRC | 05:30 | |
| *** Daisy has quit IRC | 05:33 | |
| *** matsuhashi has joined #openstack-infra | 05:33 | |
| *** Daisy has joined #openstack-infra | 05:33 | |
| *** dstanek has joined #openstack-infra | 05:34 | |
| *** lcheng has quit IRC | 05:34 | |
| *** dolphm is now known as dolphm_503 | 05:36 | |
| *** nicedice has quit IRC | 05:38 | |
| openstackgerrit | Jay Pipes proposed a change to openstack-infra/config: Adds ! defined() guards around a2mod declarations https://review.openstack.org/74443 | 05:39 |
| *** fcarpenter has quit IRC | 05:40 | |
| *** relaxdiego has quit IRC | 05:40 | |
| *** locke105 has joined #openstack-infra | 05:40 | |
| *** DinaBelova_ is now known as DinaBelova | 05:43 | |
| *** sarob_ has quit IRC | 05:43 | |
| *** sarob has joined #openstack-infra | 05:43 | |
| *** lcheng has joined #openstack-infra | 05:46 | |
| *** jcooley_ has joined #openstack-infra | 05:47 | |
| *** dstanek has quit IRC | 05:47 | |
| *** mgagne has quit IRC | 05:47 | |
| *** ociuhandu has joined #openstack-infra | 05:48 | |
| *** sarob has quit IRC | 05:48 | |
| *** dstanek has joined #openstack-infra | 05:49 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: rename maven-properties.sh script to version-properties.sh https://review.openstack.org/75318 | 05:50 |
| *** mrda is now known as mrda_weekending | 05:52 | |
| *** dstanek has quit IRC | 05:54 | |
| *** ociuhandu has quit IRC | 05:54 | |
| *** lcheng has quit IRC | 05:56 | |
| *** comstud has joined #openstack-infra | 05:57 | |
| comstud | what do i use to re-check things that are obviously transient? | 05:57 |
| comstud | like a hangup in the middle of a git clone | 05:58 |
| *** lcheng has joined #openstack-infra | 05:59 | |
| *** CaptTofu has joined #openstack-infra | 06:03 | |
| *** gokrokve has quit IRC | 06:05 | |
| *** lcheng has quit IRC | 06:07 | |
| *** CaptTofu has quit IRC | 06:08 | |
| *** markwash has quit IRC | 06:09 | |
| *** lcheng has joined #openstack-infra | 06:10 | |
| SergeyLukjanov | morning | 06:10 |
| *** gokrokve has joined #openstack-infra | 06:12 | |
| *** coolsvap has joined #openstack-infra | 06:13 | |
| *** locke105 has quit IRC | 06:16 | |
| *** yolanda has joined #openstack-infra | 06:17 | |
| *** lcheng has quit IRC | 06:18 | |
| *** thomasbiege has joined #openstack-infra | 06:18 | |
| *** thomasbiege has quit IRC | 06:18 | |
| *** protux has joined #openstack-infra | 06:19 | |
| *** lcheng has joined #openstack-infra | 06:20 | |
| *** e0ne has joined #openstack-infra | 06:21 | |
| *** e0ne has quit IRC | 06:26 | |
| *** dolphm_503 is now known as dolphm | 06:27 | |
| *** lcheng has quit IRC | 06:28 | |
| *** relaxdiego has joined #openstack-infra | 06:30 | |
| *** gokrokve has quit IRC | 06:30 | |
| mayu | clarkb: hi | 06:30 |
| *** gokrokve has joined #openstack-infra | 06:31 | |
| *** lcheng has joined #openstack-infra | 06:31 | |
| mayu | #jaypipes: hi | 06:31 |
| jaypipes | mayu: hi there :) (although... I am heading to bed soon...) | 06:31 |
| *** zhiyan is now known as zhiyan_ | 06:31 | |
| *** relaxdiego has quit IRC | 06:31 | |
| mayu | there is no gearman in my jenkins | 06:32 |
| mayu | I follow your blog to construct my local ci | 06:32 |
| jaypipes | mayu: what OS are you installing on? | 06:32 |
| mayu | ubuntu | 06:32 |
| mayu | 12.04 | 06:33 |
| jaypipes | k | 06:33 |
| jaypipes | do you have the latest from both my repo and the openstack-infra/config repo? | 06:33 |
| mayu | yes | 06:33 |
| *** vkozhukalov has joined #openstack-infra | 06:34 | |
| mayu | yesterday | 06:34 |
| jaypipes | mayu: and no errors when you run install_master.sh, yes? | 06:34 |
| mayu | I git clone your repe | 06:34 |
| mayu | repo | 06:34 |
| *** skraynev_afk is now known as skraynev | 06:34 | |
| mayu | I got error | 06:34 |
| jaypipes | mayu: about the A2mod thing? | 06:34 |
| mayu | yes | 06:35 |
| mayu | https://review.openstack.org/#/c/74443/1/modules/zuul/manifests/init.pp | 06:35 |
| *** zhiyan_ is now known as zhiyan | 06:35 | |
| mayu | clarkb told me to refer that | 06:35 |
| *** gokrokve has quit IRC | 06:35 | |
| jaypipes | mayu: yes, you can comment out those lines in the zuul init.pp for right now. | 06:35 |
| mayu | comment ? | 06:36 |
| *** morganfainberg is now known as morganfainberg_Z | 06:36 | |
| *** saju_m has joined #openstack-infra | 06:36 | |
| *** dolphm is now known as dolphm_503 | 06:36 | |
| mayu | I run install_master success | 06:36 |
| mayu | I login to jenkins | 06:37 |
| jaypipes | mayu: if you look at your jenkins log file (in /var/log/jenkins), do you see a message about the Gearman plugin being enabled?' | 06:37 |
| mayu | Following your blog, I want to configure gearman, but there is no gearman on the system configure page | 06:38 |
| mayu | wait | 06:38 |
| jaypipes | mayu: if you click Manage Jenkins -> Manage Plugins, do you see the Gearman plugin in the "Enabled" tab? | 06:38 |
| mayu | I will see | 06:39 |
| *** lcheng has quit IRC | 06:39 | |
| mayu | yes | 06:40 |
| mayu | I see that on the option tab | 06:40 |
| jaypipes | mayu: OK, good :) and when you click Manage Jenkins -> Configure System, scroll down near the bottom, there should be a Gearman Plugin Config section. | 06:41 |
| *** zhiyan is now known as zhiyan_ | 06:42 | |
| mayu | there is no gearman plugin config section | 06:42 |
| *** lcheng has joined #openstack-infra | 06:43 | |
| mayu | I think, I shoud install the plugin first | 06:43 |
| jaypipes | mayu: it should be installed already... | 06:43 |
| jaypipes | mayu: the jenkins Puppet module should install it. | 06:43 |
| jaypipes | mayu: but if it is not, sure, you can do it manually. | 06:44 |
| mayu | ok | 06:44 |
| mayu | I select gearman plugin to install, It is now updating | 06:46 |
| jaypipes | k | 06:46 |
| mayu | jaypipes: your blog help me a lot | 06:46 |
| mayu | thanks for your work | 06:47 |
| jaypipes | mayu: no problem :) happy to help! | 06:47 |
| comstud | jaypipes: good job, buddy | 06:47 |
| mayu | yes, I get the gearman plugin | 06:47 |
| jaypipes | mayu: I know just how much of a pain it can be to get all these pieces working together, so happy to help others through a little bit. | 06:47 |
| jaypipes | comstud: aw, shucks, thx Chris :) | 06:48 |
| mayu | can you give me your email ? | 06:48 |
| comstud | :) | 06:48 |
| jaypipes | mayu: jaypipes@gmail.com | 06:49 |
| comstud | jaypipes: Do you happen to know how we recheck something transient like disconnection in middle of git clone? | 06:49 |
| mayu | thanks | 06:49 |
| jaypipes | comstud: not quite sure what you mean... | 06:49 |
| comstud | i'm assuming there's not a bug for that, but i admit i haven't looked | 06:49 |
| comstud | :) | 06:49 |
| comstud | gate jobs failing because of disconnect in middle of git clone | 06:50 |
| jaypipes | comstud: do you mean how to request that Zuul re-run a check pipeline when there's no bug but just a transient issue? | 06:50 |
| comstud | we took away 'recheck no bug' right? | 06:50 |
| *** lcheng has quit IRC | 06:50 | |
| comstud | right | 06:50 |
| jaypipes | comstud: "recheck no bother clarkb"? | 06:51 |
| comstud | https://bugs.launchpad.net/openstack-ci/+bug/1279963 | 06:51 |
| comstud | i guess that one will work | 06:51 |
| mayu | jaypipes: test connection failed ? | 06:51 |
| comstud | well, that's not the same thing, hm | 06:51 |
| jaypipes | comstud: hehe | 06:51 |
| comstud | well screw it, i'll file a bug :) | 06:52 |
| jaypipes | mayu: ok, that means that zuul-server isn't started correctly. | 06:52 |
| jaypipes | comstud: lol | 06:52 |
| jaypipes | mayu: what does ps aux | grep zuul look like? | 06:52 |
| mayu | wait | 06:52 |
| mayu | nothing | 06:52 |
| *** lcheng has joined #openstack-infra | 06:53 | |
| mayu | you mean, restart zuul ? | 06:53 |
| mayu | jaypipes: restart zuul ? | 06:53 |
| jaypipes | mayu: no, I mean what is the output of running "ps aux | grep zuul" | 06:53 |
| comstud | aha, https://bugs.launchpad.net/openstack-ci/+bug/1254142 | 06:54 |
| mayu | http://paste.openstack.org/show/67889/ | 06:55 |
| jaypipes | mayu: so zuul isn't running at all. do: | 06:55 |
| jaypipes | sudo service zuul start; sudo service zuul-merger start | 06:55 |
| mayu | jaypipes:ok | 06:55 |
| mayu | jaypipes: not work | 06:57 |
| jaypipes | mayu: you're going to need to be more specific :) | 06:57 |
| mayu | I start the zuul, zuul-merger | 06:58 |
| mayu | ps aux | 06:58 |
| mayu | ps aux | grep zuul, get nothing | 06:58 |
| mayu | start fail | 06:58 |
| jaypipes | mayu: please check /var/log/zuul/debug.log for error messages | 06:59 |
| mayu | there is no log | 06:59 |
| jhesketh__ | mayu: also check there is no lock file in /var/run/zuul/* | 06:59 |
| jaypipes | jhesketh_: yes, good point, thx! | 06:59 |
| mayu | nothing | 07:00 |
| jhesketh__ | mayu: run "zuul-server -d" and see if that prints anything | 07:00 |
| jaypipes | mayu: what does ls -l /var/run/zuul show you? | 07:00 |
| mayu | zuul-server: command not found | 07:00 |
| jaypipes | mayu: what does ls -l /var/run/zuul show you? | 07:00 |
| mayu | total 0 | 07:01 |
| jaypipes | hmmm :( | 07:01 |
| mayu | as https://review.openstack.org/#/c/74443/1/modules/zuul/manifests/init.pp describe, I make change on the /root/config/modules/zuul/manifests/init.pp | 07:02 |
| jaypipes | mayu: I think zuul is installed. it's just not working for some reason... | 07:04 |
| mayu | yes | 07:04 |
| mayu | jaypipes: reinstall ? | 07:05 |
| jaypipes | mayu: I'm trying to think whether you would need to do that or not... | 07:05 |
| mayu | jaypipes:ok | 07:06 |
| jaypipes | mayu: cd os-ext-testing; git pull | 07:06 |
| jaypipes | does that pull in anything? | 07:06 |
| mayu | jaypipes: Already up-to-date. | 07:07 |
| jaypipes | k | 07:07 |
| jaypipes | mayu: and there is absolutely nothing in /var/log/zuul? | 07:08 |
| mayu | nothing | 07:08 |
| *** afazekas has quit IRC | 07:09 | |
| jaypipes | hmm, very strange indeed. | 07:09 |
| jaypipes | mayu: I'm afraid I'm out of ideas. :( | 07:09 |
| mayu | do you want to check zuul directory ? | 07:10 |
| jaypipes | mayu: perhaps you can try reinstalling. it's almost like there's something borked with the Puppet installation. | 07:10 |
| jaypipes | mayu: zuul directory? | 07:10 |
| mayu | yes | 07:10 |
| mayu | bashrc ? | 07:10 |
| jaypipes | sorry, I don't understand | 07:10 |
| mayu | http://paste.openstack.org/show/67890/ | 07:11 |
| mayu | jaypipes: do you want to check these directory on the paste | 07:12 |
| *** jcooley_ has quit IRC | 07:12 | |
| *** basha has joined #openstack-infra | 07:13 | |
| mayu | I i want to reinstall, how to clean the envirment? remove files and directory on the paste ? | 07:14 |
| *** afazekas_ has quit IRC | 07:14 | |
| jaypipes | mayu: it looks like you have run the wget and bash install_master.sh from /home | 07:14 |
| mayu | jaypipes: yes | 07:14 |
| *** jcooley_ has joined #openstack-infra | 07:15 | |
| jaypipes | mayu: that *may* be why some things are not working... since /home is a special directory in Linux ... at least, it's a directory that matters for package installation. | 07:15 |
| jaypipes | mayu: I would recommend running wget and bash install_master from the home directory of a non-root user. | 07:16 |
| openstackgerrit | Lukasz Jernas proposed a change to openstack-infra/config: Add fonts-takao package on the Debian build slave https://review.openstack.org/72287 | 07:16 |
| mayu | jaypipes: change a directory ? | 07:16 |
| mayu | jaypipes: ok, I will try | 07:16 |
| jaypipes | mayu: is this a VM or a bare metal machine? | 07:16 |
| mayu | jaypipes: vm | 07:17 |
| *** jcooley_ has quit IRC | 07:18 | |
| jaypipes | mayu: OK. I would recommend just destroying this VM entirely and restarting. make sure you log into your VM as non-root, and then execute the wget and bash install_master.sh as non-root. the script will ask for your sudo password when needed. | 07:18 |
| *** bhuvan has joined #openstack-infra | 07:18 | |
| mayu | jaypipes: ok | 07:18 |
| jaypipes | mayu: for future reference, it's not a good idea to add directories to /home that are not Linux homedirs for the system's users. | 07:18 |
| mayu | jaypipes: then i have to apply a new service count | 07:19 |
| jaypipes | mayu: start fresh with a new VM and let me know how you fare, ok? It's 2:20am here and I'm going to sleep now :) | 07:19 |
| mayu | jaypipes: ok ,thanks, have a good sleep | 07:20 |
| jaypipes | thx. chat with you later. | 07:20 |
| *** jamespage_ has quit IRC | 07:21 | |
| *** e0ne has joined #openstack-infra | 07:21 | |
| *** banix has joined #openstack-infra | 07:21 | |
| *** e0ne has quit IRC | 07:26 | |
| *** lcheng has quit IRC | 07:26 | |
| *** dolphm_503 is now known as dolphm | 07:27 | |
| *** basha has quit IRC | 07:28 | |
| *** Daisy has quit IRC | 07:29 | |
| *** gokrokve has joined #openstack-infra | 07:29 | |
| *** gokrokve_ has joined #openstack-infra | 07:30 | |
| *** Daisy has joined #openstack-infra | 07:31 | |
| *** vkozhukalov has quit IRC | 07:31 | |
| *** gokrokve has quit IRC | 07:33 | |
| *** gokrokve_ has quit IRC | 07:35 | |
| *** bhuvan has quit IRC | 07:37 | |
| *** dolphm is now known as dolphm_503 | 07:37 | |
| *** zhiyan_ is now known as zhiyan | 07:39 | |
| *** jamielennox is now known as jamielennox|away | 07:44 | |
| *** jcooley_ has joined #openstack-infra | 07:49 | |
| *** dstanek has joined #openstack-infra | 07:51 | |
| *** jcooley_ has quit IRC | 07:54 | |
| *** dstanek has quit IRC | 07:55 | |
| *** bhuvan has joined #openstack-infra | 07:56 | |
| *** afazekas_ has joined #openstack-infra | 07:56 | |
| *** flaper87 has quit IRC | 07:57 | |
| *** flaper87 has joined #openstack-infra | 07:57 | |
| *** mrmartin has joined #openstack-infra | 07:58 | |
| *** DinaBelova is now known as DinaBelova_ | 07:58 | |
| *** e0ne has joined #openstack-infra | 08:01 | |
| *** basha has joined #openstack-infra | 08:02 | |
| *** CaptTofu has joined #openstack-infra | 08:04 | |
| *** che-arne has quit IRC | 08:05 | |
| *** e0ne has quit IRC | 08:05 | |
| *** Daisy has quit IRC | 08:06 | |
| *** Daisy has joined #openstack-infra | 08:07 | |
| *** CaptTofu has quit IRC | 08:09 | |
| *** bhuvan has quit IRC | 08:14 | |
| *** khyati has quit IRC | 08:15 | |
| *** luqas has joined #openstack-infra | 08:17 | |
| *** shardy_afk is now known as shardy | 08:20 | |
| *** dolphm_503 is now known as dolphm | 08:28 | |
| *** jgallard has joined #openstack-infra | 08:30 | |
| *** ankit_ has joined #openstack-infra | 08:30 | |
| ankit_ | hi, I am new to opnestack | 08:31 |
| ankit_ | when I try to push a patch it shows me author as Jenkins | 08:31 |
| ankit_ | How can I set my name as author | 08:31 |
| *** saju_m has quit IRC | 08:31 | |
| *** gokrokve has joined #openstack-infra | 08:31 | |
| ankit_ | can anyone help me with this problem? | 08:32 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 08:34 |
| *** gokrokve has quit IRC | 08:36 | |
| *** dpyzhov has joined #openstack-infra | 08:36 | |
| *** dolphm is now known as dolphm_503 | 08:38 | |
| *** jcooley_ has joined #openstack-infra | 08:41 | |
| *** thomasbiege has joined #openstack-infra | 08:42 | |
| *** pblaho has joined #openstack-infra | 08:42 | |
| *** dstufft_ has joined #openstack-infra | 08:42 | |
| *** dstufft has quit IRC | 08:42 | |
| *** rossella-s has joined #openstack-infra | 08:43 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Auth Token Middleware https://review.openstack.org/74735 | 08:44 |
| *** masayukig has joined #openstack-infra | 08:47 | |
| *** e0ne has joined #openstack-infra | 08:48 | |
| *** jcooley_ has quit IRC | 08:51 | |
| openstackgerrit | Florian Klink proposed a change to openstack-infra/git-review: Use LC_ALL="C" for run_command_status() https://review.openstack.org/71276 | 08:52 |
| *** Daisy has quit IRC | 08:52 | |
| *** jhesketh__ has quit IRC | 08:52 | |
| *** jhesketh_ has quit IRC | 08:52 | |
| *** Daisy has joined #openstack-infra | 08:54 | |
| *** sarob has joined #openstack-infra | 08:55 | |
| *** pblaho has quit IRC | 08:56 | |
| *** ociuhandu has joined #openstack-infra | 09:00 | |
| *** sarob has quit IRC | 09:00 | |
| *** saju_m has joined #openstack-infra | 09:01 | |
| *** DinaBelova_ is now known as DinaBelova | 09:01 | |
| *** banix has quit IRC | 09:02 | |
| *** saju_m has quit IRC | 09:02 | |
| *** rossella-s has quit IRC | 09:04 | |
| *** yassine has joined #openstack-infra | 09:04 | |
| *** thomasbiege has quit IRC | 09:05 | |
| *** vkozhukalov has joined #openstack-infra | 09:07 | |
| *** dpyzhov has quit IRC | 09:08 | |
| *** fbo_away is now known as fbo | 09:09 | |
| *** derekh has joined #openstack-infra | 09:10 | |
| *** ociuhandu has quit IRC | 09:11 | |
| *** Daisy has quit IRC | 09:13 | |
| *** luqas has quit IRC | 09:17 | |
| *** noorul has joined #openstack-infra | 09:17 | |
| noorul | solum devstack is failing https://review.openstack.org/#/c/69855/ | 09:17 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Auth Token Middleware https://review.openstack.org/74735 | 09:18 |
| noorul | looks like another git issue http://logs.openstack.org/55/69855/21/check/gate-solum-devstack-dsvm/b591565/logs/devstack-gate-setup-workspace-new.txt | 09:18 |
| *** mflobo has joined #openstack-infra | 09:18 | |
| noorul | Is this related to https://bugs.launchpad.net/openstack-ci/+bug/1282136 ? | 09:19 |
| *** basha has quit IRC | 09:19 | |
| *** saju_m has joined #openstack-infra | 09:21 | |
| *** dolphm_503 is now known as dolphm | 09:23 | |
| *** chandan_kumar has joined #openstack-infra | 09:25 | |
| *** gokrokve has joined #openstack-infra | 09:29 | |
| *** ankit_ has quit IRC | 09:31 | |
| *** gokrokve has quit IRC | 09:34 | |
| *** lttrl has joined #openstack-infra | 09:36 | |
| *** dpyzhov has joined #openstack-infra | 09:41 | |
| *** max_lobur_afk is now known as max_lobur | 09:43 | |
| *** sarob has joined #openstack-infra | 09:47 | |
| *** jcooley_ has joined #openstack-infra | 09:47 | |
| *** matrohon has joined #openstack-infra | 09:50 | |
| *** masayukig has quit IRC | 09:51 | |
| *** chandan_kumar has quit IRC | 09:51 | |
| *** sarob has quit IRC | 09:51 | |
| *** jcooley_ has quit IRC | 09:52 | |
| *** luqas has joined #openstack-infra | 09:53 | |
| *** che-arne has joined #openstack-infra | 09:56 | |
| *** dizquierdo has joined #openstack-infra | 09:57 | |
| amotoki | hi, i see "git remote update" error many times now. is there any error on git.openstack.org or CI infra? | 09:58 |
| amotoki | for example: http://logs.openstack.org/03/69803/3/check/check-tempest-dsvm-neutron-full/ae3415a/logs/devstack-gate-setup-workspace-new.txt | 09:59 |
| amotoki | sorry, there is the similar post just above. | 10:00 |
| noorul | It looks like that got settled | 10:00 |
| *** CaptTofu has joined #openstack-infra | 10:05 | |
| *** chandan_kumar has joined #openstack-infra | 10:05 | |
| *** CaptTofu has quit IRC | 10:09 | |
| *** pmathews has quit IRC | 10:12 | |
| *** psedlak has quit IRC | 10:22 | |
| *** derekh has quit IRC | 10:27 | |
| *** thomasbiege has joined #openstack-infra | 10:28 | |
| *** chandan_kumar has quit IRC | 10:28 | |
| *** psedlak has joined #openstack-infra | 10:29 | |
| *** gokrokve has joined #openstack-infra | 10:29 | |
| *** gokrokve_ has joined #openstack-infra | 10:31 | |
| *** derekh has joined #openstack-infra | 10:31 | |
| *** jp_at_hp has joined #openstack-infra | 10:34 | |
| *** gokrokve has quit IRC | 10:34 | |
| *** gokrokve_ has quit IRC | 10:36 | |
| *** masayukig has joined #openstack-infra | 10:36 | |
| *** rfolco has joined #openstack-infra | 10:36 | |
| *** yassine has quit IRC | 10:40 | |
| *** yassine has joined #openstack-infra | 10:40 | |
| *** masayukig has quit IRC | 10:40 | |
| *** jcooley_ has joined #openstack-infra | 10:41 | |
| *** alexpilotti has joined #openstack-infra | 10:45 | |
| *** coolsvap has quit IRC | 10:45 | |
| *** jcooley_ has quit IRC | 10:46 | |
| *** starmer has quit IRC | 10:47 | |
| *** jcooley_ has joined #openstack-infra | 10:47 | |
| *** sarob has joined #openstack-infra | 10:47 | |
| *** jcooley_ has quit IRC | 10:52 | |
| *** coolsvap has joined #openstack-infra | 10:52 | |
| *** sarob has quit IRC | 10:54 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:54 | |
| *** luqas has quit IRC | 10:56 | |
| *** apevec has joined #openstack-infra | 10:57 | |
| *** CaptTofu has joined #openstack-infra | 10:57 | |
| *** nosnos has quit IRC | 10:58 | |
| *** matsuhashi has quit IRC | 11:02 | |
| *** dizquierdo has quit IRC | 11:03 | |
| *** dolphm is now known as dolphm_503 | 11:03 | |
| *** DinaBelova is now known as DinaBelova_ | 11:03 | |
| *** dolphm_503 is now known as dolphm | 11:03 | |
| *** dpyzhov has quit IRC | 11:06 | |
| *** vkozhukalov has quit IRC | 11:08 | |
| *** dpyzhov has joined #openstack-infra | 11:10 | |
| *** jgallard has quit IRC | 11:12 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 11:16 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Remove oslo.sphinx from global requirements https://review.openstack.org/75371 | 11:19 |
| *** DinaBelova_ is now known as DinaBelova | 11:20 | |
| *** talluri has quit IRC | 11:20 | |
| *** talluri has joined #openstack-infra | 11:20 | |
| *** vkozhukalov has joined #openstack-infra | 11:21 | |
| *** talluri has quit IRC | 11:24 | |
| *** dpyzhov has quit IRC | 11:26 | |
| *** dpyzhov has joined #openstack-infra | 11:26 | |
| *** skolekonov has joined #openstack-infra | 11:26 | |
| *** dolphm is now known as dolphm_503 | 11:27 | |
| *** coolsvap has quit IRC | 11:28 | |
| *** gokrokve has joined #openstack-infra | 11:29 | |
| *** noorul has left #openstack-infra | 11:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 11:31 |
| *** masayukig has joined #openstack-infra | 11:32 | |
| *** gokrokve has quit IRC | 11:33 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 11:46 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add posix_ipc as requirement https://review.openstack.org/68945 | 11:46 |
| *** andre__ has joined #openstack-infra | 11:46 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Add cassandra-driver dependency https://review.openstack.org/62726 | 11:46 |
| *** jcooley_ has joined #openstack-infra | 11:48 | |
| *** che-arne has quit IRC | 11:50 | |
| *** sarob has joined #openstack-infra | 11:51 | |
| *** jcooley_ has quit IRC | 11:53 | |
| *** dstanek has joined #openstack-infra | 11:54 | |
| *** jcoufal has joined #openstack-infra | 11:54 | |
| *** yamahata has quit IRC | 11:54 | |
| *** dizquierdo has joined #openstack-infra | 11:54 | |
| *** yaguang has quit IRC | 11:55 | |
| *** sarob has quit IRC | 11:55 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Bump hacking version in requirements https://review.openstack.org/49486 | 11:55 |
| *** ArxCruz has joined #openstack-infra | 11:58 | |
| *** dstanek has quit IRC | 11:59 | |
| *** johnthetubaguy has joined #openstack-infra | 11:59 | |
| *** dstufft_ is now known as dstufft | 12:04 | |
| *** luqas has joined #openstack-infra | 12:07 | |
| *** luqas has quit IRC | 12:08 | |
| *** luqas has joined #openstack-infra | 12:09 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 12:09 | |
| openstackgerrit | A change was merged to openstack-infra/git-review: Bump hacking version in requirements https://review.openstack.org/49486 | 12:14 |
| *** Daisy has joined #openstack-infra | 12:16 | |
| ArxCruz | ALL: I'm running devstack in my machine, and I'm getting this error when it runs nova x509-get-root-cert: | 12:18 |
| ArxCruz | 06:22:15 2014-02-21 01:22:15 + nova x509-get-root-cert /opt/stack/new/devstack/accrc/cacert.pem | 12:18 |
| ArxCruz | 06:23:16 2014-02-21 01:23:16 ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-a84dcff7-6da6-4790-a57d-d12ac9afb337) | 12:18 |
| ArxCruz | the nova-cert is running, and the log is showing this http://paste.openstack.org/show/67971/ | 12:18 |
| ArxCruz | does anyone knows what's wrong ? | 12:18 |
| *** lcostantino has joined #openstack-infra | 12:22 | |
| *** dpyzhov has quit IRC | 12:23 | |
| *** dpyzhov has joined #openstack-infra | 12:24 | |
| *** gokrokve has joined #openstack-infra | 12:29 | |
| *** luqas has quit IRC | 12:31 | |
| *** luqas has joined #openstack-infra | 12:31 | |
| *** luqas has quit IRC | 12:33 | |
| *** gokrokve has quit IRC | 12:34 | |
| *** yamahata has joined #openstack-infra | 12:37 | |
| *** dpyzhov has quit IRC | 12:37 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 12:39 |
| *** vkozhukalov has quit IRC | 12:39 | |
| *** Daisy has quit IRC | 12:40 | |
| *** masayukig has quit IRC | 12:42 | |
| *** masayukig has joined #openstack-infra | 12:43 | |
| *** masayukig has quit IRC | 12:47 | |
| *** dolphm_503 is now known as dolphm | 12:49 | |
| *** e0ne_ has joined #openstack-infra | 12:49 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: [WIP] Auth Token Middleware https://review.openstack.org/74735 | 12:50 |
| *** weshay has joined #openstack-infra | 12:51 | |
| *** masayukig has joined #openstack-infra | 12:51 | |
| *** sarob has joined #openstack-infra | 12:51 | |
| *** e0ne has quit IRC | 12:53 | |
| *** vkozhukalov has joined #openstack-infra | 12:55 | |
| *** sarob has quit IRC | 12:58 | |
| *** sarob has joined #openstack-infra | 13:01 | |
| *** sarob has quit IRC | 13:05 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 13:05 | |
| *** dpyzhov has joined #openstack-infra | 13:07 | |
| *** zul has quit IRC | 13:09 | |
| *** zul has joined #openstack-infra | 13:09 | |
| *** mwagner_lap has joined #openstack-infra | 13:11 | |
| *** julim has joined #openstack-infra | 13:13 | |
| *** julim has quit IRC | 13:14 | |
| *** david-lyle has quit IRC | 13:15 | |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:16 | |
| *** CaptTofu has quit IRC | 13:19 | |
| *** dcramer_ has quit IRC | 13:19 | |
| *** Daisy has joined #openstack-infra | 13:22 | |
| *** esker has quit IRC | 13:25 | |
| *** esker has joined #openstack-infra | 13:26 | |
| *** jgallard has joined #openstack-infra | 13:26 | |
| *** gokrokve has joined #openstack-infra | 13:29 | |
| ihrachys | hi all. it seems that stable/havana branch for 'trove' is unmaintained and fail on gate if any patch is sent (pyflakes fail... doc build fail... some tests fail...) can we consider removal of the branch? it causes failures like following in other projects (requirements in this case): http://logs.openstack.org/14/75014/1/check/check-requirements-integration-dsvm/2d48496/console.html | 13:30 |
| SergeyLukjanov | fungi, clarkb, jeblair, mordred, ^^ | 13:30 |
| *** esker has quit IRC | 13:30 | |
| ihrachys | I've made a fix which should mostly get the branch into good shape (https://review.openstack.org/#/c/75386/), but it still fails on reddwarf checks, so I'm not sure whether I even need to proceed with the fix for branch, or we can just nuke it | 13:30 |
| *** IvanBerezovskiy has joined #openstack-infra | 13:31 | |
| ttx | ihrachys: since trove was not integrated yet, that branch is not maintained by the stable branch team, so +2 on removing it | 13:31 |
| ttx | but i can't remove it myself, need god power | 13:31 |
| SergeyLukjanov | ihrachys, ttx, agreed, have no permissions for removing branches, so, waiting for infra-root guys | 13:32 |
| *** dpyzhov has quit IRC | 13:32 | |
| *** luqas has joined #openstack-infra | 13:35 | |
| *** dstanek has joined #openstack-infra | 13:36 | |
| *** thomasem has joined #openstack-infra | 13:38 | |
| *** mwagner_lap has quit IRC | 13:38 | |
| *** smarcet has joined #openstack-infra | 13:38 | |
| *** thomasem has quit IRC | 13:38 | |
| *** thomasem has joined #openstack-infra | 13:39 | |
| *** ArxCruz has quit IRC | 13:39 | |
| ihrachys | SergeyLukjanov: who are those root guys? | 13:40 |
| SergeyLukjanov | ihrachys, fungi, clarkb, jeblair, mordred | 13:40 |
| ihrachys | ah, ok, thanks :) | 13:40 |
| *** max_lobur is now known as max_lobur_afk | 13:41 | |
| *** dpyzhov has joined #openstack-infra | 13:42 | |
| *** ArxCruz has joined #openstack-infra | 13:43 | |
| *** russellb is now known as rustlebee | 13:44 | |
| *** dpyzhov has quit IRC | 13:45 | |
| *** jcooley_ has joined #openstack-infra | 13:47 | |
| *** mriedem has joined #openstack-infra | 13:48 | |
| *** jcooley_ has quit IRC | 13:52 | |
| *** zhiyan is now known as zhiyan_ | 13:52 | |
| *** leifmadsen is now known as blitzrage | 13:54 | |
| *** che-arne has joined #openstack-infra | 13:55 | |
| *** sarob has joined #openstack-infra | 13:55 | |
| *** dprince has joined #openstack-infra | 13:55 | |
| *** amotoki has quit IRC | 13:57 | |
| mriedem | roughly how many nodes are available in the check queue for running tests? | 13:58 |
| mriedem | i'm trying to explain to some people internally that a team of one guy in my dept can't duplicate what happens upstream with a handful of VMs | 13:59 |
| *** dpyzhov has joined #openstack-infra | 13:59 | |
| *** sarob has quit IRC | 13:59 | |
| *** Daisy has quit IRC | 13:59 | |
| *** mfer has joined #openstack-infra | 14:00 | |
| SergeyLukjanov | mriedem, all VMs are handled by nodepool | 14:03 |
| SergeyLukjanov | mriedem, you can find some stats in the bottom of http://status.openstack.org/zuul/ | 14:03 |
| mriedem | wow, so it can climb to ~750 test nodes | 14:04 |
| mriedem | the jenkins gearman plugin page says 200+ slaves | 14:04 |
| *** dizquierdo has quit IRC | 14:07 | |
| fungi | ihrachys: sure. can we get hub_cap_ to okay it? | 14:08 |
| *** oubiwann has joined #openstack-infra | 14:08 | |
| SergeyLukjanov | fungi, morning! | 14:09 |
| fungi | SergeyLukjanov: thanks! and good afternoon to you | 14:09 |
| *** saju_m has quit IRC | 14:10 | |
| fungi | the gate pipeline looks short and everything in check has been there less than an hour (except for a couple at the very top). then again, it *is* friday | 14:10 |
| *** relaxdiego has joined #openstack-infra | 14:10 | |
| SergeyLukjanov | fungi, I'm afraid that we have a lot of failing jobs due to the | 14:11 |
| fungi | nodepool appears to be acting sanely judging from the node graph | 14:11 |
| SergeyLukjanov | >> fatal: Not a git repository (or any parent up to mount parent ) | 14:11 |
| apevec | so after verification fails, new +1 check is required again, even if there's recent one? https://review.openstack.org/73404 | 14:11 |
| SergeyLukjanov | like http://logs.openstack.org/61/75261/1/check/check-tempest-dsvm-savanna-neutron/26caad8/logs/devstack-gate-setup-workspace-new.txt | 14:11 |
| *** miqui has joined #openstack-infra | 14:11 | |
| *** mrmartin has quit IRC | 14:11 | |
| fungi | SergeyLukjanov: having a look to see where that's happening--thanks | 14:11 |
| apevec | it's close to impossible to merge to stable/havana neutron | 14:11 |
| SergeyLukjanov | fungi, rax-iad I think | 14:11 |
| *** julim has joined #openstack-infra | 14:12 | |
| dstanek | 'reverify no bug' shouldn't be used anymore right? | 14:12 |
| SpamapS | it can't | 14:12 |
| SpamapS | you must have a bug to reverify | 14:12 |
| SergeyLukjanov | dstanek, yup | 14:12 |
| SergeyLukjanov | dstanek, it was disabled about a month ago I think | 14:12 |
| dstanek | that's what i thought | 14:13 |
| *** oubiwann has quit IRC | 14:13 | |
| fungi | dstanek: right. we found that people were actively introducing nondeterministic failures by repeatedly reverifying until they could get their buggy code to merge, which is part of why things got so bad | 14:13 |
| SpamapS | and off I go into the wild blue yonder | 14:13 |
| dstanek | fungi: yuck | 14:14 |
| openstackgerrit | A change was merged to openstack-dev/hacking: More portable way to detect modules for H302 https://review.openstack.org/68858 | 14:14 |
| dstanek | i was just helping someone understand the process and they were asking when to use that | 14:14 |
| fungi | dstanek: it was an unintended psychological side effect... sort of a snowball effect. having nondeterministic failures caused people to introduce new ones by not paying close attention to whether they were the ones introducing them | 14:15 |
| SergeyLukjanov | fungi, could we search for the contents of devstack-gate-setup-workspace-new.txt using logstash? | 14:15 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Enhance H233 rule https://review.openstack.org/68573 | 14:15 |
| *** mbacchi has joined #openstack-infra | 14:16 | |
| *** banix has joined #openstack-infra | 14:16 | |
| fungi | SergeyLukjanov: no, it's not currently being indexed, so i'm just trying to collect some examples to narrow down whether it's just the current rax-iad devstack-precise image | 14:16 |
| SergeyLukjanov | fungi, I'm checking now savanna jobs | 14:16 |
| SergeyLukjanov | that was failing today | 14:17 |
| fungi | i think probably a next step for nodepool is to add image acceptance tests (run a tempest-full job or something) before moving the image state from building to ready | 14:17 |
| fungi | anyway, i'm grabbing a devstack-precise-rax-iad node to poke around in | 14:18 |
| SergeyLukjanov | fungi, we're using the same approach for pre-built by dib images in savanna-ci | 14:18 |
| SergeyLukjanov | fungi, oh, see two jobs failed on rax-dfw | 14:18 |
| fungi | devstack-precise or some other image? | 14:18 |
| SergeyLukjanov | fungi, devstack-precise | 14:19 |
| fungi | okay, i'll check that one out too | 14:19 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 14:19 |
| *** zul has quit IRC | 14:20 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Migration to add the openid field https://review.openstack.org/75381 | 14:21 |
| *** zul has joined #openstack-infra | 14:21 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 14:21 |
| fungi | so, a random devstack-precise-rax-iad node i grabbed is missing (at least) nova in /opt/git/openstack | 14:23 |
| SergeyLukjanov | fungi, :( | 14:23 |
| fungi | which is causing jobs to re-clone nova over the network instead, looks like | 14:24 |
| fungi | i'm going to start an image update for that one and have a look at dfw | 14:25 |
| fungi | SergeyLukjanov: if you want, you should be able to have a look at the image build log at http://nodepool.openstack.org/ | 14:25 |
| fungi | maybe spot whether we failed cloning nova when that image was built, which should tell us where we ought to be trapping errors and are failing to do so | 14:26 |
| SergeyLukjanov | fungi, looking on it | 14:26 |
| fungi | awesome--thanks! | 14:26 |
| *** zehicle_at_dell has quit IRC | 14:27 | |
| *** dpyzhov has quit IRC | 14:28 | |
| *** rossella-s has joined #openstack-infra | 14:29 | |
| sdague | so it looks like the neutron-full job fails about twice as often as the neutron regular job | 14:33 |
| sdague | but that's not bad as a starting point | 14:33 |
| fungi | found a random devstack-precise-rax-dfw is also missing /opt/git/openstack/nova | 14:33 |
| fungi | so i'll rebuild that image too | 14:33 |
| *** Daisy has joined #openstack-infra | 14:33 | |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: Clean up puppet (deploy LAMP / setup app config) https://review.openstack.org/69636 | 14:33 |
| *** masayukig has quit IRC | 14:34 | |
| fungi | devstack-precise-rax-ord has an /opt/git/openstack/nova so this doesn't appear to be entirely endemic | 14:36 |
| *** jeckersb_gone is now known as jeckersb | 14:37 | |
| *** gokrokve has quit IRC | 14:37 | |
| *** gokrokve has joined #openstack-infra | 14:39 | |
| *** jgrimm has quit IRC | 14:41 | |
| SergeyLukjanov | fungi, heh, just downloaded the log file :( | 14:42 |
| SergeyLukjanov | poor connection | 14:42 |
| openstackgerrit | sebastian marcet proposed a change to openstack-infra/config: OpenstackID Documentation https://review.openstack.org/69620 | 14:42 |
| *** Daisy has quit IRC | 14:42 | |
| *** markmc has joined #openstack-infra | 14:43 | |
| fungi | i'm going to work on a patch to health-check the git repositories which get cloned, and also make sure the chain of various prep scripts is actually expected to fail on git cloning errors while i'm at it | 14:43 |
| *** dpyzhov has joined #openstack-infra | 14:43 | |
| *** wenlock has joined #openstack-infra | 14:44 | |
| SergeyLukjanov | fungi, I see tons of: | 14:44 |
| SergeyLukjanov | image.log:34093:2014-02-21 03:16:57,192 INFO nodepool.image.build.rax-dfw.bare-centos6: error: Unable to get pack index https://git.openstack.org/openstack/nova/objects/pack/pack-bc58e2fba84295d20d8ec80408dab3fcef85541f.idx | 14:44 |
| SergeyLukjanov | SergeyLukjanov, 95 entries in log | 14:44 |
| fungi | SergeyLukjanov: did the same thing happen for iad as well? and was it just nova or other repositories too? | 14:44 |
| SergeyLukjanov | fungi, looking now | 14:45 |
| fungi | and if you can spot other images which we don't yet know about that ran into similar trouble, i'll get to work rebuilding them too | 14:45 |
| *** dpyzhov has quit IRC | 14:45 | |
| *** jcooley_ has joined #openstack-infra | 14:48 | |
| *** max_lobur_afk is now known as max_lobur | 14:49 | |
| *** CaptTofu has joined #openstack-infra | 14:50 | |
| dolphm | fungi: this is an instance of the nova clone / IAD issue you're chasing down- https://bugs.launchpad.net/openstack-ci/+bug/1283038 | 14:51 |
| SergeyLukjanov | fungi, there are failures to clone neutron and nova on rax-dfw and rax-ida | 14:51 |
| SergeyLukjanov | rax-iad | 14:51 |
| dolphm | SergeyLukjanov: ^ | 14:52 |
| SergeyLukjanov | fungi, my crazy script - grep -irn 'error: Unable to get pack index' image.log | cut -c 67-160 | sort -u | 14:52 |
| SergeyLukjanov | rax-dfw.bare-centos6: error: Unable to get pack index https://git.openstack.org/openstack/neut | 14:52 |
| SergeyLukjanov | rax-dfw.bare-centos6: error: Unable to get pack index https://git.openstack.org/openstack/nova | 14:52 |
| SergeyLukjanov | rax-iad.bare-centos6: error: Unable to get pack index https://git.openstack.org/openstack/nova | 14:52 |
| fungi | SergeyLukjanov: good, so hopefully this will be resolved shortly when the image rebuilds for those two zones complete | 14:52 |
| fungi | dolphm: yes, exactly | 14:53 |
| SergeyLukjanov | dolphm, yup, find the same in logs | 14:53 |
| fungi | dolphm: though the bug misinterprets teh logs a bit | 14:53 |
| *** eharney has joined #openstack-infra | 14:54 | |
| fungi | dolphm: the "Couldn't find remote ref" is normal. that's just devstack-gate's setup checking to see whether zuul is providing specific merge refs for various projects in the integration set | 14:54 |
| *** jcooley_ has quit IRC | 14:54 | |
| dolphm | fungi: hmm, then it shouldn't say "fatal" ;) | 14:54 |
| fungi | dolphm: the actual errors are the fact that the local cached clones of some repositories are broken | 14:54 |
| fungi | dolphm: true. i suppose we could write a parser for teh git utility's error messages or something | 14:55 |
| fungi | but that's git reporting "fatal" not the actual script running it | 14:55 |
| *** sarob has joined #openstack-infra | 14:55 | |
| dolphm | fungi: so is the real source the "error: RPC failed; result=56, HTTP code = 200" ? | 14:56 |
| fungi | dolphm: the error of note is "fatal: Not a git repository (or any parent up to mount parent )" | 14:57 |
| fungi | it's trying to run remote update in an existing (pre-cached) clone of a project, and that clone isn't there or is missing the .git subdirectory | 14:57 |
| fungi | because the image builds broke when setting that up | 14:58 |
| *** prad_ has joined #openstack-infra | 14:58 | |
| fungi | i'm working right now on making that more robust | 14:58 |
| fungi | SergeyLukjanov: so, re-reading what you pasted above, bare-centos6-rax-dfw and bare-centos6-rax-iad are being added to the list of broken images (i misread earlier). i'll rebuild those too | 14:59 |
| *** dkliban has quit IRC | 14:59 | |
| fungi | sounds like rackspace dfw and iad may have been a consistent factor here | 14:59 |
| *** dcramer_ has joined #openstack-infra | 15:00 | |
| ihrachys | fungi: is the 'error: RPC failed; result=56, HTTP code = 200' failure worth being rechecked now? or should we wait some time? | 15:00 |
| dolphm | fungi: revised the report, fwiw | 15:00 |
| fungi | dolphm: thanks! | 15:01 |
| fungi | ihrachys: that error doesn't sound familiar. is there a bug for that one? | 15:01 |
| dolphm | fungi: that's teh same issue (bug 1283038) | 15:01 |
| SergeyLukjanov | fungi, thx | 15:02 |
| *** sarob has quit IRC | 15:02 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Rename oslo.sphinx to oslosphinx https://review.openstack.org/73709 | 15:02 |
| fungi | dolphm: ihrachys: oh, i see, that's what is happening sometimes when the slave tries to reclone nova or neutron over the network while setting up the job (since the local copy was broken) | 15:02 |
| *** jnoller has joined #openstack-infra | 15:02 | |
| *** wenlock has quit IRC | 15:02 | |
| fungi | ihrachys: it's safe to recheck--that's an inconsistent error, determined by network performance/stability i suspect | 15:03 |
| *** jnoller has quit IRC | 15:03 | |
| *** dizquierdo has joined #openstack-infra | 15:03 | |
| *** jnoller has joined #openstack-infra | 15:05 | |
| *** masayukig has joined #openstack-infra | 15:05 | |
| *** afazekas_ has quit IRC | 15:07 | |
| *** luqas has quit IRC | 15:07 | |
| *** zarric has joined #openstack-infra | 15:08 | |
| *** masayukig has quit IRC | 15:09 | |
| *** jishaom has joined #openstack-infra | 15:09 | |
| *** jcooley_ has joined #openstack-infra | 15:09 | |
| *** alaski_eto is now known as alaski | 15:09 | |
| *** luqas has joined #openstack-infra | 15:10 | |
| *** coolsvap has joined #openstack-infra | 15:10 | |
| *** rossella-s has quit IRC | 15:11 | |
| *** bhuvan has joined #openstack-infra | 15:14 | |
| *** mayu has quit IRC | 15:16 | |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Add gate-murano-api-devstack job https://review.openstack.org/75078 | 15:18 |
| *** mgagne has joined #openstack-infra | 15:18 | |
| fungi | devstack-precise-rax-iad image finished rebuilding a few minutes ago | 15:20 |
| SergeyLukjanov | fungi, cool! | 15:20 |
| *** pdmars has joined #openstack-infra | 15:20 | |
| *** smarcet has quit IRC | 15:21 | |
| BobBall | if I wipe the nodepool DB does it re-discover (and potentially delete) nodes - e.g. based on a wildcard for the hostname / nodename? | 15:21 |
| *** jishaom has quit IRC | 15:21 | |
| fungi | BobBall: no, but it does have an alien list command line option in the nodepool client you should be able to use to list servers in all configured providers which nodepool doesn't have a record of in its database | 15:25 |
| fungi | you could use that to either build insert queries to repopulate the database or as something to feed into a nova delete | 15:26 |
| *** rossella-s has joined #openstack-infra | 15:26 | |
| BobBall | OK - was trying to rebuild our nodepool instance and have it re-generate the state | 15:26 |
| BobBall | but I guess I can backup the DB | 15:26 |
| *** mgagne1 has joined #openstack-infra | 15:26 | |
| *** ArxCruz has quit IRC | 15:26 | |
| BobBall | which isn't too hard ;) | 15:26 |
| *** alexpilotti has quit IRC | 15:27 | |
| fungi | i don't think rediscovery and database repopulation would necessarily be a bad feature for nodepool to grow, it just doesn't do that right now | 15:27 |
| *** tjones has joined #openstack-infra | 15:27 | |
| *** apevec has quit IRC | 15:27 | |
| *** mgagne has quit IRC | 15:27 | |
| fungi | really what nodepool is in most dire need of, i think is unit tests | 15:27 |
| fungi | before we go bolting new non-critical features onto it | 15:28 |
| *** esker has joined #openstack-infra | 15:29 | |
| BobBall | Fair point :) | 15:29 |
| fungi | also, the devstack-precise-rax-dfw image finished rebuilding just now | 15:29 |
| *** david-lyle has joined #openstack-infra | 15:29 | |
| *** ArxCruz has joined #openstack-infra | 15:29 | |
| *** bknudson has quit IRC | 15:29 | |
| *** bhuvan has quit IRC | 15:30 | |
| *** dkliban has joined #openstack-infra | 15:30 | |
| *** smarcet has joined #openstack-infra | 15:31 | |
| BobBall | how long do those images take to build fungi ? just ooi and comparison for the XS images (which I know are doing a _lot_ more - take about an hour) | 15:31 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add new oslo libs to ATC stats program https://review.openstack.org/75437 | 15:32 |
| *** thomasbiege1 has joined #openstack-infra | 15:32 | |
| fungi | BobBall: it depends on provider performance, operating system and what's being configured/added... anywhere from 45 to 90 minutes from what i've seen | 15:32 |
| *** tjones has quit IRC | 15:32 | |
| *** bknudson has joined #openstack-infra | 15:33 | |
| BobBall | oh wow | 15:33 |
| BobBall | I thought it'd be a load less than that | 15:33 |
| *** thomasbiege1 has quit IRC | 15:36 | |
| *** thomasbiege has quit IRC | 15:36 | |
| anteaya | jishom: there is no service account by the name of 'IBM DB2 Test' that I can see in our list of current 3rd part test accounts, nor do I see a request for one in our email archives | 15:36 |
| anteaya | https://review.openstack.org/#/admin/groups/270,members | 15:36 |
| *** jgrimm has joined #openstack-infra | 15:37 | |
| fungi | anteaya: that one may have been revoked for problems back before we had a separate sandbox group to track them in | 15:38 |
| fungi | anteaya: i've added them now | 15:39 |
| *** wenlock has joined #openstack-infra | 15:39 | |
| *** alexpilotti has joined #openstack-infra | 15:41 | |
| *** mkoderer has quit IRC | 15:41 | |
| *** flaper87 is now known as flaper87|afk | 15:42 | |
| *** lnxnut has joined #openstack-infra | 15:43 | |
| *** max_lobur is now known as max_lobur_afk | 15:45 | |
| *** flaper87|afk is now known as flaper87 | 15:46 | |
| *** jroovers|afk has joined #openstack-infra | 15:47 | |
| *** smarcet has quit IRC | 15:47 | |
| *** smarcet has joined #openstack-infra | 15:49 | |
| *** luis_ has joined #openstack-infra | 15:49 | |
| *** jroovers has quit IRC | 15:50 | |
| *** luisg has quit IRC | 15:50 | |
| jeblair | fungi, SergeyLukjanov: there are a lot of devstack related gate failures right now; i spot checked one and it seems that the job is trying to clone nova | 15:51 |
| jeblair | fungi, SergeyLukjanov: it should not need to do that because of the cached git repos | 15:51 |
| fungi | jeblair: yeah, see scrollback | 15:51 |
| SergeyLukjanov | jeblair, morning | 15:51 |
| jeblair | ah yep | 15:51 |
| fungi | jeblair: i believe it should be mostly resolved now | 15:51 |
| SergeyLukjanov | jeblair, yeah, we've tried to do something ;) | 15:52 |
| jeblair | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 | 15:52 |
| jeblair | the git servers have had to work very hard since roughly around when the images were built | 15:53 |
| jeblair | fungi: so you deleted images; do you know why there were built incorrectly? | 15:53 |
| *** amcrn has joined #openstack-infra | 15:53 | |
| *** jroovers|afk has quit IRC | 15:53 | |
| fungi | jeblair: from the logs it looks like the prep script encountered pack errors retrieving nova and neutron while building new images in rax dfw and iad | 15:54 |
| jeblair | fungi: and it didn't stop? | 15:54 |
| fungi | i've tried to manually reproduce that failure and the repos on our git servers seem fine | 15:54 |
| fungi | and no, it didn't stop. i'm fixing that | 15:54 |
| fungi | patch eta is about another 5 minutes | 15:54 |
| jeblair | fungi: ok cool; thx | 15:54 |
| anteaya | fungi: ah did you find an email to go with that account name? I didn't find an email for it | 15:57 |
| *** jcoufal has quit IRC | 15:57 | |
| fungi | anteaya: it shows one when you hover over it in the member list in that group | 15:57 |
| *** jcoufal has joined #openstack-infra | 15:57 | |
| * anteaya hovers | 15:58 | |
| anteaya | I see it, no idea where you got it from | 15:58 |
| anteaya | you must have a great memory | 15:59 |
| *** sarob has joined #openstack-infra | 15:59 | |
| fungi | anteaya: no, gerrit gives you selection options matching parts of the display name if you type them in, same as with adding requested reviewers to a change | 15:59 |
| * fungi has an awful memory | 15:59 | |
| relaxdiego | hi, can anybody point me to the devstack local.conf used for the gating process? | 15:59 |
| anteaya | fungi: also salv-orlando asked about how to add an email for the vmmine sweeper account, I suggested he email the infra list, but does he have access to add the email himself? | 16:00 |
| anteaya | ah | 16:00 |
| fungi | anteaya: he does not have access to modify that account. i'm days behind on e-mails and bug triage, but maybe i can catch up over the weekend | 16:00 |
| *** jcooley_ has quit IRC | 16:00 | |
| anteaya | fungi: I understand, no rush, I just want to make sure I am giving out accurate information | 16:01 |
| *** jcooley_ has joined #openstack-infra | 16:01 | |
| *** markmcclain has joined #openstack-infra | 16:01 | |
| fungi | if he either sends e-mail to the infra ml or opens a bug against openstack-ci, i'll get to it soonish | 16:01 |
| anteaya | the gate comes first | 16:01 |
| anteaya | k, I suggested the email route | 16:01 |
| *** markwash has joined #openstack-infra | 16:01 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add gear statsd support https://review.openstack.org/75441 | 16:02 |
| *** skolekonov has quit IRC | 16:03 | |
| *** sarob has quit IRC | 16:03 | |
| anteaya | relaxdiego: I'm looking for it, the closest I have found is: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/devstack-gate-secure.conf.erb | 16:04 |
| anteaya | which is just the template, not the conf | 16:04 |
| *** prad__ has joined #openstack-infra | 16:04 | |
| *** masayukig has joined #openstack-infra | 16:05 | |
| *** jcooley_ has quit IRC | 16:06 | |
| *** dcramer_ has quit IRC | 16:06 | |
| *** prad_ has quit IRC | 16:06 | |
| *** mflobo has quit IRC | 16:07 | |
| fungi | anteaya: relaxdiego: the local.conf is different for different jobs, and gets built by devstack-gate depending on environment variables passed into it. for any devstack-based job result reported on a change, if you browse the log link, you should see a copy of it in the logs subdirectory | 16:07 |
| *** coolsvap has quit IRC | 16:08 | |
| *** therve_ has joined #openstack-infra | 16:08 | |
| *** coolsvap1 has joined #openstack-infra | 16:08 | |
| *** luis_ has quit IRC | 16:08 | |
| *** coolsvap1 has quit IRC | 16:08 | |
| *** luisg has joined #openstack-infra | 16:09 | |
| anteaya | fungi: thanks | 16:10 |
| *** jcooley_ has joined #openstack-infra | 16:10 | |
| *** masayukig has quit IRC | 16:10 | |
| *** pcrews_ has joined #openstack-infra | 16:11 | |
| *** DinaBelova is now known as DinaBelova_ | 16:11 | |
| *** luqas has quit IRC | 16:13 | |
| *** coolsvap1 has joined #openstack-infra | 16:14 | |
| *** luisg has quit IRC | 16:14 | |
| *** ArxCruz has quit IRC | 16:14 | |
| *** luisg has joined #openstack-infra | 16:14 | |
| *** atiwari has joined #openstack-infra | 16:15 | |
| *** oubiwann has joined #openstack-infra | 16:16 | |
| *** CaptTofu has quit IRC | 16:16 | |
| *** jcoufal has quit IRC | 16:16 | |
| fungi | i'm seeing "Unable to get pack index https://git.openstack.org/openstack/neutron/objects/pack/pack-04432e30fc1e9057070595b516b8e136e5ea77ea.idx" and similar on the bare-centos6-rax-iad update... i wonder if there's something going on with a proxy in rackspace | 16:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Geard: Report packet timing to statsd https://review.openstack.org/75449 | 16:22 |
| fungi | but i'll give the git farm another health check too, just to be sure | 16:22 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Use client_id as part of the logger name https://review.openstack.org/73118 | 16:22 |
| *** DinaBelova_ is now known as DinaBelova | 16:22 | |
| jeblair | fungi: need help? | 16:22 |
| *** ArxCruz has joined #openstack-infra | 16:24 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Use six.moves.urllib.parse instead of urlparse https://review.openstack.org/72887 | 16:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Make nodepool git repo caching more robust https://review.openstack.org/75450 | 16:25 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Stop separately caching repos for devstack images https://review.openstack.org/73076 | 16:25 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Clone git repos for images via git protocol https://review.openstack.org/73073 | 16:25 |
| fungi | jeblair: there's those ^ (not tested yet, but the first two are just rebases of stuff which already got reviewed and/or approved) | 16:25 |
| jeblair | fungi: cool | 16:25 |
| *** dcramer_ has joined #openstack-infra | 16:26 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fix misspellings in storyboard https://review.openstack.org/72053 | 16:27 |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack-infra/config: Added requirements gates for Murano repositories https://review.openstack.org/75451 | 16:28 |
| *** oubiwann has quit IRC | 16:28 | |
| anteaya | something christmas treed in the gate | 16:29 |
| *** IvanBerezovskiy has left #openstack-infra | 16:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Retry jobs after gear disconnect https://review.openstack.org/75453 | 16:30 |
| fungi | similar problems in hpcloud az1? | 16:30 |
| *** CaptTofu has joined #openstack-infra | 16:30 | |
| banix | After getting "LOST" on all tests at the gate, I did a recheck no bug and now I get the following error ERROR: invocation failed, logfile: /home/jenkins/workspace/gate-neutron-pep8/.tox/pep8/log/pep8-2.log ; I don't seem to find any reference to the code being tested. Am I missing something or this error may be not related to the patch. | 16:31 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:31 | |
| anteaya | fungi: yes I saw one in az1 and here is one in hpcloud az3: https://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron/1108/console | 16:32 |
| fungi | maybe there were more failed image builds in the log than SergeyLukjanov noticed | 16:33 |
| fungi | er, s/failed/broken/ | 16:33 |
| jeblair | banix: it's probably related to what fungi is working on; probably best to wait about 30 mins and then recheck | 16:33 |
| banix | jeblair: no worries. thanks for letting me know. | 16:33 |
| jeblair | #status alert Some builds are failing due to errors in worker images; fix eta 1700 UTC. | 16:34 |
| openstackstatus | NOTICE: Some builds are failing due to errors in worker images; fix eta 1700 UTC. | 16:34 |
| *** ChanServ changes topic to "Some builds are failing due to errors in worker images; fix eta 1700 UTC." | 16:34 | |
| fungi | hopefully the thunderstorms here don't knock out my power. if i vanish from irc you'll know why | 16:34 |
| jeblair | fungi: i'm keeping a list of reviews we need to do: https://etherpad.openstack.org/p/infra-2014-02-21 | 16:35 |
| jeblair | once we get to that point | 16:35 |
| *** pmathews has joined #openstack-infra | 16:36 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Switch Zuul geard logging to DEBUG level https://review.openstack.org/75455 | 16:37 |
| fungi | okay, i've confirmed that the nova and neutron repos, which were the ones spotted with clone failures in the image build logs, are able to be cloned cleanly on each of the 4 git servers | 16:37 |
| fungi | i'll try remote clones for them next just to rule out something funky with apache | 16:38 |
| *** lcheng has joined #openstack-infra | 16:38 | |
| *** beagles is now known as beagles_brb | 16:38 | |
| *** skraynev is now known as skraynev_afk | 16:38 | |
| anteaya | do you want me to file a bug? | 16:38 |
| fungi | there are already a couple duplicate bugs open | 16:39 |
| *** DinaBelova is now known as DinaBelova_ | 16:40 | |
| anteaya | k | 16:40 |
| fungi | no reported network problems in rackspace today according to their status page | 16:43 |
| jeblair | fungi: could the clones have happened during a repack? | 16:43 |
| *** jcooley_ has quit IRC | 16:44 | |
| *** jcooley_ has joined #openstack-infra | 16:44 | |
| fungi | jeblair: maybe, but i got it again just a few minutes ago rebuilding the image for bare-centos6-rax-iad | 16:44 |
| *** gokrokve has quit IRC | 16:44 | |
| jeblair | fungi: the load is rather high | 16:44 |
| *** gokrokve has joined #openstack-infra | 16:45 | |
| *** therve_ has quit IRC | 16:45 | |
| jeblair | current on git01 is 11, it was recently above 20 | 16:45 |
| fungi | i suspect that's a snowball effect from so many jobs retrying to clone nova and neutron | 16:46 |
| fungi | since it seems to have picked up after the image rebuilds | 16:46 |
| *** dkliban is now known as dkliban_afk | 16:46 | |
| fungi | i see similar patterns on the rest of the git servers too | 16:47 |
| jeblair | fungi: do we have images in production with broken caches? | 16:47 |
| fungi | it seems like we do. there are now similar failures spotted for hpcloud-az1 and az3 | 16:48 |
| jeblair | fungi: if so, we should remove them, let the load subside, and only then attempt a rebuild. preferably after your patches to ensure that we don't get broken images land. | 16:48 |
| fungi | and i don't know yet (haven't checked back) whether the images i rebuilt in iad and dfw failed to clone again | 16:48 |
| fungi | okay, agreed | 16:48 |
| jeblair | i'll let you do that | 16:49 |
| *** gokrokve has quit IRC | 16:49 | |
| fungi | yeah, i'll just go through and manually check nodes built from all the existing images to see if any are okay | 16:49 |
| fungi | and delete the corresponding images if not | 16:50 |
| jeblair | fungi: or just delete all of today's images. your call. | 16:50 |
| *** gokrokve has joined #openstack-infra | 16:51 | |
| *** oubiwann has joined #openstack-infra | 16:51 | |
| *** geekinutah has joined #openstack-infra | 16:51 | |
| geekinutah | are there problems with git.openstack.org? | 16:52 |
| geekinutah | seeing a few failures like this: https://jenkins07.openstack.org/job/gate-nova-pep8/819/console | 16:52 |
| jeblair | geekinutah: yes; we updated the channel topic with inf | 16:52 |
| jeblair | o | 16:52 |
| jeblair | geekinutah: (we cache git repos on our worker images; that failed so we are ddosing ourselves) | 16:53 |
| fungi | recent devstack-precise-rax-iad nodes look okay at the moment, so i think that image update was a success | 16:53 |
| geekinutah | jeblair: got it | 16:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Clone git repos for images via git protocol https://review.openstack.org/73073 | 16:53 |
| *** max_lobur_afk is now known as max_lobur | 16:54 | |
| fungi | devstack-precise-rax-dfw seems good now too | 16:55 |
| openstackgerrit | A change was merged to openstack-infra/config: Stop separately caching repos for devstack images https://review.openstack.org/73076 | 16:55 |
| *** gokrokve has quit IRC | 16:56 | |
| *** eharney has quit IRC | 16:56 | |
| jeblair | fungi: your robustify change lgtm; i think we can merge it whenever, but i'll hold off approving in case another core shows up soon | 16:56 |
| *** sahid has joined #openstack-infra | 16:56 | |
| *** comstud is now known as bearhands | 16:56 | |
| fungi | devstack-precise-rax-ord looks like it was okay already | 16:56 |
| *** starmer has joined #openstack-infra | 16:56 | |
| anteaya | yes I never saw an error from rax-ord | 16:56 |
| anteaya | or iad | 16:56 |
| fungi | iad was previously broken | 16:57 |
| anteaya | ah | 16:57 |
| *** jcooley_ has quit IRC | 16:57 | |
| *** basha has joined #openstack-infra | 16:57 | |
| fungi | looks like devstack-precise-hpcloud-az1 is broken, so i'm deleting it | 16:57 |
| *** Ajaeger has joined #openstack-infra | 16:57 | |
| *** jcooley_ has joined #openstack-infra | 16:58 | |
| jeblair | fungi: i have to run an errand biab. | 16:58 |
| fungi | jeblair: cool. thanks | 16:58 |
| ttx | fungi: do you have a bug number for the issue you are working to solve ? | 16:58 |
| fungi | ttx: i think there are a couple in scrollback | 16:58 |
| ttx | Trying to see if it matches http://logs.openstack.org/93/74993/1/gate/gate-grenade-dsvm/ec70c94/console.html | 16:58 |
| Ajaeger | How often is the requirements job run? It seems to me that changes only get propagated to a few projects but not to all. | 16:59 |
| *** sarob has joined #openstack-infra | 16:59 | |
| fungi | ttx: if you see in the setup workspace it's trying to clone nova or neutron via https, then yes | 17:00 |
| fungi | Ajaeger: that job is broken (or at least was). there may be a change proposed to fix it... seems like clarkb worked on one a few weeks ago | 17:00 |
| Ajaeger | fungi: In that case still broken - ok, will check the proposed patches | 17:01 |
| ttx | for reference: Bug 1282880 or bug Bug 1282876 | 17:01 |
| *** jcooley_ has quit IRC | 17:02 | |
| fungi | devstack-precise-hpcloud-az3 is broken too, so deleting that image | 17:02 |
| fungi | i think any devstack jobs starting in the next few minutes should be safe from the current riot | 17:03 |
| *** derekh has quit IRC | 17:04 | |
| *** rossella-s has quit IRC | 17:04 | |
| anteaya | fungi: does protocol.slave.o.o use the same images? | 17:04 |
| anteaya | it is showing a failed job | 17:04 |
| fungi | anteaya: i have no idea what you're asking | 17:04 |
| *** gokrokve has joined #openstack-infra | 17:04 | |
| Ajaeger | fungi, I found nothing, will file a bug. | 17:04 |
| fungi | we don't have a protocol slave | 17:04 |
| anteaya | https://jenkins.openstack.org/job/manuals-upstream-translation-update/1825/console | 17:04 |
| anteaya | https://jenkins.openstack.org/computer/proposal.slave.openstack.org/ | 17:05 |
| fungi | anteaya: sometimes those jobs are just broken for some projects | 17:05 |
| anteaya | proposal | 17:05 |
| *** sarob has quit IRC | 17:05 | |
| *** rossella-s has joined #openstack-infra | 17:06 | |
| *** masayukig has joined #openstack-infra | 17:06 | |
| fungi | anteaya: that log claims that openstack-manuals-i18n.cli-reference is not a project in transifex, which is quite possibly true | 17:06 |
| *** tjones has joined #openstack-infra | 17:06 | |
| *** jp_at_hp has quit IRC | 17:07 | |
| anteaya | okay | 17:07 |
| Ajaeger | fungi, anteaya known issue. | 17:07 |
| anteaya | Ajaeger: alright thanks | 17:07 |
| Ajaeger | anteaya: https://bugs.launchpad.net/openstack-manuals/+bug/1275599 | 17:07 |
| anteaya | trying to separate the actual brokeness from the everything brokeness | 17:07 |
| *** rossella-s has quit IRC | 17:07 | |
| Ajaeger | anteaya: but according to the bug, it should work now - so if it still fails, please reopen the bug. | 17:08 |
| dkranz | fungi: I need to reverify a patch where one of the jobs just never got started. I vaguely remember there being some catch-all infra bug but can't find it. | 17:09 |
| Ajaeger | anteaya: looking at jenkins, it worked an hour ago, so might really be fixed | 17:09 |
| fungi | dkranz: at the moment it's probably a case of 1282880/1282876 | 17:09 |
| dkranz | fungi: k, thanks | 17:10 |
| *** masayukig has quit IRC | 17:10 | |
| *** eharney has joined #openstack-infra | 17:11 | |
| fungi | Ajaeger: the most recent commit merged to openstack/requirements seems to be 77ed652f5216e37e71632aa4e9e407f0badbabb5 and the log for the associated reqs proposal job is therefore http://logs.openstack.org/77/77ed652f5216e37e71632aa4e9e407f0badbabb5/post/propose-requirements-updates/f1cf7c8/console.html | 17:11 |
| dkranz | fungi: Doesn't look like those to me http://logs.openstack.org/94/52994/16/gate/gate-tempest-dsvm-large-ops/b6db454/console.html | 17:11 |
| anteaya | Ajaeger: I added a comment to the bug report | 17:12 |
| fungi | Ajaeger: that's the same issue i recall clarkb working on (rebasing problem presumed due to someone uploading a changed patchset for that review) | 17:12 |
| fungi | dkranz: http://logs.openstack.org/94/52994/16/gate/gate-tempest-dsvm-large-ops/b6db454/logs/devstack-gate-setup-workspace-new.txt "fatal: Not a git repository (or any parent up to mount parent )" | 17:13 |
| fungi | same issue we're currently dealing with | 17:13 |
| dkranz | fungi: Oh, sorry. I see this is not in the console. | 17:13 |
| *** basha has quit IRC | 17:13 | |
| Ajaeger | anteaya: so did I - thanks! | 17:14 |
| anteaya | np | 17:14 |
| fungi | dkranz: nope. image builds overnight ran into issues cloning nova and neutron in some places, so the local cache for those is missing on the nodes built from them. then devstack clones them over the network instead and the ensuing load kills our git server farm | 17:14 |
| *** hogepodge has quit IRC | 17:14 | |
| dkranz | fungi: ouch | 17:15 |
| clarkb | morning | 17:15 |
| fungi | which is why the error pattern is a subtle one and not hitting every run | 17:15 |
| *** gokrokve has quit IRC | 17:15 | |
| fungi | morning clarkb | 17:15 |
| clarkb | I worked on a thing? | 17:15 |
| * clarkb catches up | 17:15 | |
| clarkb | also I managed to get an earlier start today \o/ | 17:15 |
| *** gokrokve has joined #openstack-infra | 17:15 | |
| fungi | clarkb: you had a patch for the reqs proposal job's rebasing issues, right? | 17:15 |
| anteaya | morning clarkb | 17:16 |
| fungi | clarkb: did it not work, or did it rot on the vine? | 17:16 |
| clarkb | oh yes, I thought we merged the fix | 17:16 |
| * clarkb checks what happened | 17:16 | |
| *** relaxdiego has quit IRC | 17:16 | |
| fungi | clarkb: anyway, latest recommended reading is unrelated to that. https://review.openstack.org/75450 | 17:16 |
| *** jcooley_ has joined #openstack-infra | 17:17 | |
| clarkb | 1059461c6c11d6745f1d5890eb2c15194746e378 is the infra change that should have fixed rebase issues for requirements proposals | 17:18 |
| clarkb | Ajaeger: ^ | 17:18 |
| *** eharney has quit IRC | 17:18 | |
| fungi | clarkb: does http://logs.openstack.org/77/77ed652f5216e37e71632aa4e9e407f0badbabb5/post/propose-requirements-updates/f1cf7c8/console.html look to you like a continuation of the same reqs proposal issue then, or a new one? | 17:18 |
| *** DinaBelova_ is now known as DinaBelova | 17:19 | |
| jeblair | fungi: re | 17:19 |
| *** e0ne_ has quit IRC | 17:19 | |
| *** e0ne has joined #openstack-infra | 17:19 | |
| jeblair | clarkb: when you have a sec, these reviews are important. especially the top one. https://etherpad.openstack.org/p/infra-2014-02-21 | 17:20 |
| *** nicedice has joined #openstack-infra | 17:20 | |
| clarkb | jeblair: ok | 17:20 |
| *** gokrokve has quit IRC | 17:20 | |
| anteaya | right now the only patch passing in the gate is a neutron patach | 17:21 |
| *** jcooley_ has quit IRC | 17:21 | |
| anteaya | but all the jobs are fresh, so let's see what happens | 17:21 |
| *** jcooley_ has joined #openstack-infra | 17:22 | |
| clarkb | fungi: I think that is a new related problem. possibly a race | 17:22 |
| clarkb | fungi: if say cinder were to propose a change that bumps the reqs manually and get that behind the requirements change in the gate | 17:23 |
| clarkb | and that merges before the propose requirements job runs | 17:23 |
| clarkb | actually wait | 17:23 |
| clarkb | that is still doing a git review -d which shouldn't happen anymore | 17:23 |
| clarkb | maybe the script needs to be updated on the proposal host? | 17:23 |
| fungi | maybe the proposal host has hung puppet | 17:24 |
| clarkb | anyways I am going to review that list of changes now but checking puppet and the version of the script on that host is what I would look at next | 17:24 |
| *** e0ne has quit IRC | 17:24 | |
| *** jcooley_ has quit IRC | 17:24 | |
| *** beagles_brb is now known as beagles | 17:25 | |
| fungi | puppet agent is configured to start on boot, not run as a cron job, but it's not actually running | 17:25 |
| fungi | no idea how long it's been that way, maybe a very, very long time | 17:25 |
| *** hogepodge has joined #openstack-infra | 17:25 | |
| clarkb | puppet agent --test --noop will give you a diff if you are worreid | 17:25 |
| *** markmc has quit IRC | 17:25 | |
| fungi | already running it | 17:25 |
| fungi | yeah, crazy huge diff | 17:26 |
| clarkb | jeblair: what is with all of the failures on the zuul changes? | 17:26 |
| clarkb | syntax errors and pep8 unhappy | 17:27 |
| jeblair | whoops | 17:27 |
| fungi | clarkb: see my inline comment on the first one | 17:27 |
| clarkb | oh ok, I thought I might've been missing something | 17:27 |
| clarkb | fungi: ah thnaks I was on a child change | 17:27 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Add gear statsd support https://review.openstack.org/75441 | 17:28 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Retry jobs after gear disconnect https://review.openstack.org/75453 | 17:28 |
| openstackgerrit | A change was merged to openstack-infra/config: Make nodepool git repo caching more robust https://review.openstack.org/75450 | 17:29 |
| *** dangers_` is now known as dangers | 17:30 | |
| fungi | okay, i've caught puppet up on proposal.s.o.o and started the agent back running again | 17:30 |
| clarkb | fungi: ^ merged | 17:30 |
| fungi | clarkb: saw, thanks | 17:30 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/gear: Geard: Report packet timing to statsd https://review.openstack.org/75449 | 17:30 |
| *** jog0 is now known as flashgordon | 17:30 | |
| fungi | i'll kick off an image update shortly once it's in place, to make sure it works as intended | 17:30 |
| *** jcooley_ has joined #openstack-infra | 17:31 | |
| *** jcooley_ has quit IRC | 17:32 | |
| jeblair | anteaya: what's the bug for the git/image issues? | 17:32 |
| jeblair | fungi: are we free of broken images now? | 17:32 |
| fungi | jeblair: as best as i can tell, yes | 17:33 |
| clarkb | jeblair: fungi: this is probably not worth worrying about too much but client id is supplied by a remote worker/client to geard. That value is then possibly used by pythonlogging for filesystem related things. Any concern about that being a security issue? | 17:33 |
| clarkb | jeblair: fungi: mostly worried about the naive logging config | 17:33 |
| fungi | jeblair: bug 1282880/1282876 (dupes) | 17:33 |
| anteaya | 1282876 | 17:33 |
| jeblair | clarkb: we can take our time on that change if it needs more work | 17:33 |
| jeblair | clarkb: the others don't depend on it | 17:33 |
| Ajaeger | clarkb: that was merged on the 6th of February but seems it still happens | 17:33 |
| fungi | Ajaeger: we discovered that puppet agent hasn't been running on that slave, possibly for a very long time. i just got it caught up and restarted | 17:34 |
| clarkb | jeblair: ok, I will drop a comment with the current thought process then move on to the others | 17:34 |
| Ajaeger | fungi: Ah, I see. Great! Thanks. | 17:34 |
| jeblair | #status alert Git-related build issues should be resolved. If your job failed with no build output, use "recheck bug 1282876". | 17:36 |
| openstackstatus | NOTICE: Git-related build issues should be resolved. If your job failed with no build output, use "recheck bug 1282876". | 17:36 |
| *** ChanServ changes topic to "Git-related build issues should be resolved. If your job failed with no build output, use "recheck bug 1282876"." | 17:36 | |
| *** relaxdiego has joined #openstack-infra | 17:37 | |
| *** gyee has joined #openstack-infra | 17:38 | |
| *** cadenzajon has joined #openstack-infra | 17:39 | |
| *** vrovachev has left #openstack-infra | 17:40 | |
| clarkb | jeblair: ok commented on geard logging chnage | 17:40 |
| *** gokrokve has joined #openstack-infra | 17:40 | |
| *** atiwari has quit IRC | 17:40 | |
| fungi | looks like cpu utilization is finally dropping off for the git servers | 17:40 |
| *** vkozhukalov has quit IRC | 17:40 | |
| fungi | and load averages have taken a sharp nosedive | 17:41 |
| *** jgallard has quit IRC | 17:41 | |
| anteaya | yay | 17:41 |
| clarkb | fungi: were they being hammered for some reason leading to the bad images? | 17:42 |
| fungi | clarkb: the other way around | 17:42 |
| anteaya | the bad images didn't have a full git cache | 17:42 |
| anteaya | so the images were going to the gits for the repos | 17:42 |
| clarkb | oh fun | 17:43 |
| anteaya | in bundles | 17:43 |
| *** oubiwann has quit IRC | 17:44 | |
| fungi | we were ddos'ing our git servers with many hundreds of devstack slaves | 17:44 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 17:44 |
| clarkb | fungi: nice | 17:45 |
| clarkb | fungi: any chance we know what caused the image builds to need retries and all that? | 17:45 |
| *** hogepodge has quit IRC | 17:45 | |
| clarkb | jeblair: I am surprised at how simple the retry job after LOST zuul change is but maybe I shouldn't be :) | 17:45 |
| fungi | clarkb: error: Unable to get pack index https://git.openstack.org/openstack/nova | 17:45 |
| *** blitzrage has quit IRC | 17:46 | |
| fungi | clarkb: might have tried to clone during a repack | 17:46 |
| fungi | but that was what we had in the image logs | 17:46 |
| dhellmann | jeblair: for "no build output" do you mean a job that couldn't fetch the code at all, or is there literally no output in the log? | 17:46 |
| jeblair | clarkb: yeah, it was a one liner. i added other stuff to fluff it out a bit. ;) | 17:46 |
| *** BobBall has quit IRC | 17:47 | |
| fungi | jeblair: so just not including a job status on complete causes it to retry? | 17:47 |
| jeblair | dhellmann: couldn't fetch the code. i'm not sure how to say that in an irc-subject length way. am open to suggestions. output looks like this http://logs.openstack.org/94/52994/16/gate/gate-tempest-dsvm-large-ops/b6db454/console.html | 17:47 |
| jeblair | fungi: yep. | 17:47 |
| *** hogepodge has joined #openstack-infra | 17:47 | |
| jeblair | fungi: well, it should. i haven't tested it. | 17:47 |
| clarkb | jeblair: fungi: yeah I had to read through onBuildCompleted to see that | 17:47 |
| dhellmann | jeblair: ok, I had a "Temporary failure in name resolution" so that sounds like a different issue | 17:48 |
| clarkb | jeblair: it sets the retry flag to true which is used in other cases | 17:48 |
| fungi | clarkb: heh. i did as well | 17:48 |
| clarkb | so I think it should work, but ++ to tests | 17:48 |
| jeblair | fungi: but it should look just like what happens when jenkins returns a work_fail. | 17:48 |
| fungi | got it | 17:48 |
| anteaya | dhellmann: that might be the dns failure bug | 17:49 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Removed unneeded lstrip https://review.openstack.org/75472 | 17:49 |
| anteaya | we have been seeing it on rax-dfw | 17:49 |
| dhellmann | anteaya: yep, I just found that one :-) | 17:49 |
| anteaya | k | 17:49 |
| * dhellmann crosses his fingers and tries again :-) | 17:49 | |
| fungi | anteaya: what dns failure bug? | 17:50 |
| anteaya | https://bugs.launchpad.net/openstack-ci/+bug/1270382 | 17:50 |
| fungi | anteaya: oh. that one. i missed dhellmann's "Temporary failure in name resolution" above | 17:51 |
| anteaya | dhellmann: was in on a rax-dfw slave? | 17:51 |
| dhellmann | anteaya: I'm not sure how to tell, let me find that log again | 17:51 |
| dhellmann | anteaya: http://logs.openstack.org/08/74408/4/check/gate-oslo.test-python33/8347716/ | 17:52 |
| anteaya | dhellmann: in the console log, go to the very top | 17:52 |
| anteaya | about the 3rd line | 17:52 |
| *** dansmith is now known as damnsmith | 17:52 | |
| dhellmann | ah, yeah, "Building remotely on py3k-precise-rax-dfw-1499503" | 17:52 |
| anteaya | yes | 17:52 |
| anteaya | so same bug | 17:52 |
| *** mgagne1 is now known as mgagne | 17:53 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add new oslo libs to ATC stats program https://review.openstack.org/75437 | 17:55 |
| *** max_lobur is now known as max_lobur_afk | 17:55 | |
| jeblair | fungi: oh, wasn't there a change to pull that from governance ^ ? | 17:56 |
| *** harlowja_away is now known as harlowja | 17:56 | |
| jeblair | https://review.openstack.org/#/c/73348/ | 17:56 |
| jeblair | perhaps we should review that to save other people some work | 17:56 |
| fungi | jeblair: yes | 17:56 |
| flashgordon | is git.o.o still having issues? or is it safe to recheck | 17:57 |
| flashgordon | http://logs.openstack.org/21/74621/3/check/gate-nova-pep8/5936597/console.html | 17:57 |
| jeblair | flashgordon: safe (see channel topic) | 17:57 |
| *** wendar has quit IRC | 17:57 | |
| flashgordon | jeblair: derp | 17:57 |
| flashgordon | thanks | 17:57 |
| *** dizquierdo has quit IRC | 17:57 | |
| clarkb | I am going to be holding a bare-precise slave to help zaro debug buck build issues | 17:58 |
| *** blitzrage has joined #openstack-infra | 17:59 | |
| jeblair | fungi: i think it would be better if the email stats program parsed the yaml instead of bash... would you be receptive to that? | 17:59 |
| fungi | jeblair: see the comment i just added | 17:59 |
| fungi | jeblair: but sure, i'm happy to spend time reworking it. that was just my quick-n-dirty hack i used when i generated the last set of stats | 18:00 |
| *** blitzrage is now known as leifmadsen | 18:00 | |
| fungi | so wanted it to get recorded rather than losing track of it | 18:00 |
| *** leifmadsen has quit IRC | 18:00 | |
| *** leifmadsen has joined #openstack-infra | 18:00 | |
| jeblair | fungi: ok, left comment. | 18:01 |
| *** hogepodge_ has joined #openstack-infra | 18:02 | |
| fungi | jeblair: thanks--totally agree, obviously | 18:02 |
| *** hogepodge has quit IRC | 18:02 | |
| *** hogepodge_ is now known as hogepodge | 18:02 | |
| *** prad_ has joined #openstack-infra | 18:02 | |
| *** sarob has joined #openstack-infra | 18:03 | |
| clarkb | our bare precise slaves use java6 by default for some reason | 18:04 |
| *** prad__ has quit IRC | 18:05 | |
| fungi | yay! OverLimit: OverLimit Retry... (HTTP 413) | 18:05 |
| fungi | or whatever the opposite of yay is | 18:06 |
| jeblair | fungi: rate limit or ram limit or what? | 18:06 |
| *** masayukig has joined #openstack-infra | 18:07 | |
| *** sarob has quit IRC | 18:07 | |
| fungi | hard to know... novaclient reported that from a createImage action | 18:08 |
| jeblair | that's weird. yay indeed. | 18:09 |
| *** hemnafk is now known as hemna | 18:09 | |
| anteaya | fungi: which zone? | 18:10 |
| fungi | i'm deleting the old devstack-precise image from rax-iad since the new one is good | 18:10 |
| jeblair | fungi: er? | 18:10 |
| jeblair | fungi: why? | 18:10 |
| fungi | maybe they're frowning on too many images there | 18:10 |
| jeblair | fungi: i don't think that should be the case. | 18:11 |
| *** bobba has joined #openstack-infra | 18:11 | |
| fungi | jeblair: this morning when issues first became apparent, i rebuilt devstack-precise in rax-iad and rax-dfw, but hadn't deleted the old ones | 18:11 |
| jeblair | fungi: yeah, but nodepool is supposed to take care of that itself. the ideal is to have the current image and the previous good one online at all times | 18:11 |
| *** masayukig has quit IRC | 18:11 | |
| jeblair | fungi: so that we can roll back easily | 18:12 |
| fungi | well, the previous good one was not good | 18:12 |
| *** marun has joined #openstack-infra | 18:12 | |
| fungi | but it doesn't seem to be keeping more than one of most of them | 18:12 |
| *** melwitt has joined #openstack-infra | 18:12 | |
| jeblair | fungi: that's a bug then.... what image was being used when you deleted the current images earlier to fix the problems? | 18:12 |
| fungi | jeblair: none--there were no older images for them | 18:13 |
| dkranz | fungi: should I be doing reverify at this point? | 18:14 |
| jeblair | fungi: ah. so the expected procedure for this is "if the current image is bad, delete it so that nodepool will use the next most recent image; ie, the one it was using yesterday" | 18:14 |
| jeblair | fungi: this explains why you did not execute that procedure immediately for all of the images this morning. :) | 18:14 |
| jeblair | fungi: let's fix that bug, because it's a pretty important part of how the system is supposed to work | 18:14 |
| fungi | well for whatever reason it seems to get rid of the old one when the new one completes building, unless you run image-update from the client | 18:15 |
| fungi | so my guess would be an off-by-one error | 18:15 |
| jeblair | fungi: in the meanwhile, you shouldn't need to delete any images | 18:15 |
| anteaya | dkranz: recheck bug 1282876 | 18:15 |
| fungi | okay, then we're probably over either ram or machine quota in iad for some reason. i'll try to figure out which | 18:15 |
| anteaya | or reverify bug 1282876 | 18:15 |
| anteaya | and yes, you should be good | 18:15 |
| jeblair | fungi: if there is an image (or image storage) quota, we need to identify and correct that too | 18:16 |
| fungi | jeblair: oh! this time when i retried, i got a more descriptive error from novaclient | 18:16 |
| fungi | OverLimit: Quota exceeded for instances,ram: Requested 1, but already used 192 of 192 instances (HTTP 413) (Request-ID: req-95e6c100-07ac-48a4-94bf-01a986e139db) | 18:16 |
| fungi | i'll check the alien list | 18:16 |
| fungi | we probably have some strays in iad | 18:16 |
| *** dkliban_afk is now known as dkliban | 18:16 | |
| fungi | 7 in fact. i'll nova delete those | 18:17 |
| *** damnsmith is now known as busymany | 18:18 | |
| *** busymany is now known as damnsmith | 18:18 | |
| *** hogepodge has quit IRC | 18:19 | |
| *** hogepodge has joined #openstack-infra | 18:19 | |
| *** dpyzhov has joined #openstack-infra | 18:20 | |
| jeblair | fungi: most of those nodepool thought it deleted. :( | 18:21 |
| jeblair | fungi: can you nova show them and see what state it says they are in? | 18:21 |
| *** cadenzajon has quit IRC | 18:21 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added REST API for tasks https://review.openstack.org/74004 | 18:21 |
| jeblair | fungi: especially the 150... ones | 18:22 |
| clarkb | figured out why bare precise slaves are using java 6. The ant package on precise depends on java6 | 18:22 |
| clarkb | and doesnt OR java7 as a dependency | 18:22 |
| fungi | jeblair: ahh, yep i think we may have some lag there. one of them was deleted by the time i tried to nova delete it | 18:22 |
| fungi | jeblair: but devstack-precise-rax-iad-1487023 says active | 18:22 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add gear statsd support https://review.openstack.org/75441 | 18:23 |
| *** dolphm is now known as dolphm_503 | 18:23 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Retry jobs after gear disconnect https://review.openstack.org/75453 | 18:23 |
| fungi | bare-precise-rax-iad-1483046 too | 18:23 |
| *** khyati has joined #openstack-infra | 18:23 | |
| jeblair | 2014-02-21 02:01:08,204 INFO nodepool.NodePool: Deleted node id: 1487023 | 18:23 |
| fungi | bare-precise-rax-iad-1482880 says active as well | 18:24 |
| *** morganfainberg_Z is now known as morganfainberg | 18:24 | |
| fungi | so it does seem to be losing track of at least some of these | 18:24 |
| *** mrmartin has joined #openstack-infra | 18:24 | |
| *** sandywalsh has quit IRC | 18:24 | |
| *** UtahDave has joined #openstack-infra | 18:24 | |
| anteaya | we have a failure on a centos 6 slave: https://jenkins01.openstack.org/job/gate-nova-python26/21247/console | 18:24 |
| anteaya | it doesn't look like 1282876 to me | 18:25 |
| jeblair | fungi: ok. i think that's all the debugging info we can get; feel free to proceed | 18:25 |
| *** tjones has quit IRC | 18:25 | |
| anteaya | nor 1270382 | 18:25 |
| fungi | anteaya: yeah, that's another example of one of the broken images. i was trying to rebuild that one but it was taking too long, so i aborted it and am retrying with the new patches added | 18:26 |
| fungi | anteaya: but there are far fewer repositories cloned by py26 tests, so the impact of those on the git servers is much lower | 18:27 |
| anteaya | okay, I wondered if that was the one | 18:27 |
| anteaya | so still 1282876 for that one? | 18:27 |
| fungi | anteaya: and we don't have slaves of that type on as many providers, so blowing away the current images for them would leave us a lot more starved | 18:27 |
| fungi | sure | 18:27 |
| anteaya | k | 18:27 |
| *** che-arne has quit IRC | 18:27 | |
| fungi | the one in iad is in a similar situation, which is why i was trying to diagnose my inability to update images there | 18:28 |
| *** cadenzajon has joined #openstack-infra | 18:28 | |
| anteaya | damnsmith: if https://review.openstack.org/#/c/75186/ fails on that py26 failure ^^ | 18:28 |
| anteaya | makes sense | 18:29 |
| jeblair | fungi: oh, i understand the 1487023 leak | 18:29 |
| anteaya | I don't want to comment prematurely in case the gate is reset | 18:29 |
| damnsmith | anteaya: sorry, what? | 18:29 |
| anteaya | currently 75186 is failing on py26 in the gate | 18:30 |
| jeblair | fungi: nodepool got a 503 back from the create server call | 18:30 |
| jeblair | ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 503) | 18:30 |
| jeblair | fungi: so it never got the id of the server | 18:30 |
| anteaya | if it doesn't get reset and fails use bug 1282876 | 18:30 |
| damnsmith | anteaya: you're just prepping me for a reverify? | 18:30 |
| jeblair | fungi: basically, i have no idea how to handle that case. nova sent us a 503 and created the server anyway. | 18:30 |
| anteaya | yes | 18:30 |
| anteaya | if you need it | 18:30 |
| fungi | jeblair: oh, that makes sense. can't delete it if you can't identify it | 18:30 |
| damnsmith | anteaya: heh, okay, thanks | 18:30 |
| *** cody-somerville has joined #openstack-infra | 18:30 | |
| *** cody-somerville has quit IRC | 18:30 | |
| *** cody-somerville has joined #openstack-infra | 18:30 | |
| anteaya | np | 18:30 |
| clarkb | jeblair: that is an awesome failure mode | 18:31 |
| clarkb | I failed, (just kidding did it behind your back anyways) | 18:31 |
| *** oubiwann has joined #openstack-infra | 18:31 | |
| fungi | jeblair: one suggestion BobBall had earlier was that nodepool could rediscover nodes at configured providers based on specific name patterns | 18:31 |
| jeblair | flashgordon: ^ any thoughts? nova gave us a 503 on server creation, created the server anyway, but since we didn't get an id, we lose track of it and it is leaked. | 18:31 |
| fungi | which might at least allow it to notice those at some later stage | 18:31 |
| jeblair | fungi: yeah. if we do it carefully, that's probably ok. especially since these accounts are now (almost) all-dynamic. | 18:32 |
| *** atiwari has joined #openstack-infra | 18:32 | |
| jeblair | flashgordon: full traceback http://paste.openstack.org/show/68094/ | 18:33 |
| flashgordon | jeblair: looking | 18:33 |
| flashgordon | which cloud is it? and do you have the req-id? | 18:33 |
| jeblair | flashgordon: rax-iad and no | 18:33 |
| *** medieval1 has joined #openstack-infra | 18:34 | |
| jeblair | flashgordon: reading ClientException, i believe that means that there was no request id | 18:35 |
| flashgordon | jeblair: hmm yeah | 18:36 |
| *** tjones has joined #openstack-infra | 18:37 | |
| clarkb | fungi: was https://review.openstack.org/#/c/75449/ not approved because we raced to the +2? | 18:37 |
| clarkb | I can approve it if that was the only reason | 18:37 |
| jeblair | flashgordon: should we ask a racker to try to track this error down? | 18:37 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: make ant use java 7 https://review.openstack.org/75478 | 18:37 |
| *** sandywalsh has joined #openstack-infra | 18:38 | |
| *** markwash_ has joined #openstack-infra | 18:38 | |
| flashgordon | I think this is also a novaclient bug | 18:39 |
| flashgordon | that is hidding the actual error message | 18:39 |
| clarkb | jeblair: https://review.openstack.org/#/c/73118/ I think I need lower latency communication to talk about that. Does here work? | 18:40 |
| flashgordon | the attribute 'message' of 'exceptions.BaseException' part | 18:40 |
| jeblair | flashgordon: i read "<attribute 'message' of 'exceptions.BaseException' objects>" as indicating that there was no message supplied to the constructor based on the 'message or ...' part of __init__ | 18:40 |
| flashgordon | but there should be a message | 18:40 |
| jeblair | clarkb: yep | 18:40 |
| jeblair | flashgordon: ok | 18:40 |
| *** sarob has joined #openstack-infra | 18:40 | |
| jeblair | flashgordon: a 503 should come with a message? | 18:40 |
| flashgordon | jeblair: I think so | 18:41 |
| jeblair | flashgordon: makes sense then | 18:41 |
| *** markwash has quit IRC | 18:41 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Clean up dead code in image_update https://review.openstack.org/75479 | 18:41 |
| clarkb | jeblair: for my first question, it sounds like that invalidates any concern of the second? | 18:41 |
| clarkb | jeblair: since all of the values logged are locally set? | 18:41 |
| *** markwash_ is now known as markwash | 18:41 | |
| flashgordon | jeblair: although not sure what in nova does a 503 | 18:42 |
| clarkb | rather than supplied by remote ends of the connection | 18:42 |
| flashgordon | jeblair: I don't think anything does a 503 in nova | 18:42 |
| flashgordon | (os_tenant_networks and fping do) | 18:42 |
| jeblair | flashgordon: maybe a proxy in front of it? | 18:42 |
| flashgordon | jeblair: yeah | 18:42 |
| flashgordon | which explains why no req-id | 18:42 |
| flashgordon | or message | 18:43 |
| flashgordon | jeblair: perhaps ask comstud | 18:43 |
| flashgordon | aka bearhands: | 18:43 |
| jeblair | clarkb: i think that's the case. | 18:44 |
| clarkb | jeblair: great thanks | 18:45 |
| jeblair | clarkb: however, when looking at the server logs, we probably would like to know which client it's dealing with. but perhaps that's better handled in the message itself? | 18:45 |
| clarkb | jeblair: that is what I am thinking | 18:45 |
| jeblair | clarkb: basically, looking at client logs, you know who the server is so we're just trying to identify which client in a situation where there are multiple ones | 18:45 |
| jeblair | clarkb: for server logs, well, you probably know what the server is, and you want to know who the clients are... | 18:46 |
| *** dprince has quit IRC | 18:46 | |
| *** markwash has quit IRC | 18:46 | |
| jeblair | clarkb: but there are some programs (logstash client) where you might have clients and servers in the same process | 18:46 |
| jeblair | clarkb: so then it's probably useful to distinguish clienta clientb and server if you have them all funneling through logging | 18:47 |
| *** dpyzhov has quit IRC | 18:47 | |
| jeblair | clarkb: so i was thinking that putting the clientid in the logging prefix would make it so you could write a logging config that put your gear server logs in one place, your clienta logs in another, and clientb in a third | 18:47 |
| dims | clarkb, who do we ping about the missing libvirt patch in precise-proposed? (https://bugs.launchpad.net/nova/+bug/1228977/comments/37) | 18:48 |
| clarkb | dims: zul | 18:48 |
| jeblair | clarkb: so maybe this is a good way to go, and we just add missing connection name info to individual log entries on the server side? | 18:48 |
| *** markwash has joined #openstack-infra | 18:48 | |
| dims | zul, ping! :) | 18:48 |
| clarkb | jeblair: yes that si what I am thinking | 18:48 |
| dims | thanks clarkb | 18:48 |
| zul | dims: whats up | 18:48 |
| jeblair | clarkb: (because, i'm guessing you would never want to separate logging streams by client id for a server) | 18:48 |
| dims | zul, plz see https://bugs.launchpad.net/nova/+bug/1228977/comments/37 | 18:49 |
| jeblair | clarkb: so putting the client id in the logging prefix for a server is not that useful. | 18:49 |
| jeblair | clarkb: this talk has been helpful for me. i hope it has been for you. :) | 18:49 |
| clarkb | jeblair: it has | 18:49 |
| zul | dims: yeah i think i might be missing something in that patch, we should be getting a new version of libvirt next week from usptream | 18:49 |
| dims | zul, popped up again in another run - " error : virCPUDefUpdateFeatureInternal:679 : internal error: CPU feature `tsc-deadline' specified more than once" | 18:49 |
| dims | zul, k, so we wait till then and try again? | 18:50 |
| zul | dims: yes please | 18:50 |
| dims | zul, sounds good. thx | 18:50 |
| dims | zul, another thing i noticed was that i had to nuke /usr/lib/libvirt/connection-driver/libvirt_driver_libxl.so as it stopped the driver from initializing | 18:51 |
| *** dolphm_503 is now known as dolphm | 18:51 | |
| *** markmcclain has quit IRC | 18:52 | |
| zul | dims: really? you still dont have the log file for it do you? | 18:52 |
| anteaya | but did it make a dolphm in the background? | 18:52 |
| dims | zul, will find it | 18:52 |
| zul | dims: cool thanks | 18:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Enable statsd for jenkins-log-client https://review.openstack.org/75481 | 18:53 |
| dolphm | anteaya: o/ | 18:53 |
| jeblair | clarkb: ^ to finish what we started the other day | 18:53 |
| anteaya | o/ | 18:53 |
| *** pmathews has quit IRC | 18:55 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add zm0{1,2} to cacti https://review.openstack.org/75482 | 18:56 |
| *** tjones has quit IRC | 18:57 | |
| *** SumitNaiksatam has quit IRC | 18:57 | |
| mordred | jeblair: dammit. I had a brilliant idea while driving, and I almost called you to tell you about it - but then I thought Id just find you on IRC ... | 18:57 |
| mordred | jeblair: and now I've forgotten what it was | 18:57 |
| anteaya | glad you don't text and drive | 18:58 |
| anteaya | you will remember it | 18:58 |
| mordred | I do - but this was too long for text | 18:58 |
| anteaya | don't text and drive | 18:58 |
| clarkb | logstash workers are still getting into unhappy places. I am going to reduce to 2 workers per node since that seemed to have helped previously (and accept that there will be backlog) | 19:00 |
| *** pmathews has joined #openstack-infra | 19:00 | |
| *** Ajaeger has quit IRC | 19:01 | |
| SergeyLukjanov | I'm sorry, was away... it was an unexpectedly long way to home today... | 19:01 |
| anteaya | you are allowed to be away | 19:02 |
| anteaya | also allowed to sleep and eat | 19:02 |
| SergeyLukjanov | how is the gate? | 19:02 |
| anteaya | and sometimes leave the house | 19:02 |
| anteaya | right now, all is well | 19:02 |
| anteaya | right now | 19:02 |
| SergeyLukjanov | heh, it's not bad at least ;) | 19:02 |
| anteaya | much cleanup due to images created with bad git caches that ddos'd our git | 19:03 |
| anteaya | using https://bugs.launchpad.net/openstack-ci/+bug/1282876 for cleanup | 19:03 |
| anteaya | the git caches were incomplete so all the slaves were hitting git.o.o to get teh missing repos | 19:04 |
| anteaya | fungi deleted the bad images to fix | 19:04 |
| *** tjones has joined #openstack-infra | 19:04 | |
| fungi | well, most of them. the replacement centos6 images are going to be ready soon | 19:04 |
| *** khyati has quit IRC | 19:06 | |
| flashgordon | http://logs.openstack.org/88/75188/1/check/check-tempest-dsvm-full/cafeca7/console.html | 19:08 |
| flashgordon | do we havea bug for that | 19:08 |
| anteaya | woohooo <-- hockey | 19:08 |
| *** johnthetubaguy has quit IRC | 19:08 | |
| flashgordon | hmm looks like git issue | 19:08 |
| kevinbenton | Hi, I have a question about gerrit. I need to set a dependency to another review, which is normally not a problem because I rebase against it. However, the review is missing a newer commit in master that I'm also depending on. If I cherry-pick the newer one from master as well, gerrit thinks I'm trying to update that change as well since the commit-id is now different and then fails. Is having a review dependen | 19:09 |
| kevinbenton | two commits possible? | 19:09 |
| fungi | flashgordon: yes, that one's still being worked on. the problem ran deeper than devstack images, but devstack was the hardest-hit so that was dealt with first | 19:10 |
| dims | zul, here you go - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1283179 | 19:10 |
| jeblair | fungi: i didn't know there was another problem; what's that? | 19:10 |
| fungi | jeblair: the centos6 images in iad and dfw had similar issues with their local clones | 19:11 |
| *** wendar has joined #openstack-infra | 19:11 | |
| fungi | jeblair: i ended up having to abort the image updates for them earlier because it was taking too long, was before we merged my changed, and the git servers were still under heavy load at the time. didn't want to delete the images for those because it would leave us with basically no workers for py25 jobs | 19:12 |
| *** locke105 has joined #openstack-infra | 19:12 | |
| fungi | jeblair: iad is rebuilding now, as a means of debugging the new nodepool script updates | 19:12 |
| jeblair | fungi: what flashgordon linked to looks like a missing repo on an devstack image problem... | 19:12 |
| zul | dims: thanks | 19:12 |
| anteaya | devstack-precise-rax-iad-1496017 | 19:12 |
| dhellmann | hmm, what does a failure of "post-mirror-python33" mean didn't work at a high level? | 19:12 |
| fungi | jeblair: oh, i missed the url and thought he was talking about https://jenkins01.openstack.org/job/gate-nova-python26/21247/ | 19:12 |
| jeblair | ah. ok. all caught up. | 19:13 |
| anteaya | dhellmann: uncertain, have you logs? | 19:13 |
| dhellmann | it says it can't find a file, but doesn't say which one: https://jenkins.openstack.org/job/post-mirror-python33/373/console | 19:13 |
| dhellmann | I'm not sure what that job is supposed to be doing, so I don't know if it matters that it didn't work | 19:13 |
| openstackgerrit | Jay Pipes proposed a change to openstack-infra/config: Adds ! defined() guards around a2mod declarations https://review.openstack.org/74443 | 19:13 |
| dhellmann | is it rebuilding a mirror, or testing that the new package can work in the mirror somehow? | 19:13 |
| fungi | flashgordon: yes, sorry, i was looking at the wrong log. that should be fixed now, bug number coming... | 19:13 |
| jeblair | dhellmann: it is actually rebuilding the production mirror | 19:14 |
| jeblair | so moderately important | 19:14 |
| fungi | flashgordon: 1282876 | 19:14 |
| dhellmann | jeblair: that's what I was afraid of | 19:14 |
| dhellmann | jeblair: the python 2.6 and 2.7 jobs appear to have passed ok | 19:14 |
| flashgordon | fungi: thanks | 19:15 |
| *** hogepodge has quit IRC | 19:15 | |
| jeblair | dhellmann: previous run had same failure https://jenkins.openstack.org/job/post-mirror-python33/372/console | 19:15 |
| jeblair | dhellmann: last success was 23 days ago https://jenkins.openstack.org/job/post-mirror-python33/293/ | 19:15 |
| clarkb | anteaya: do you get the game live? | 19:15 |
| anteaya | yes | 19:16 |
| jeblair | dhellmann: the periodic job runs nightly https://jenkins.openstack.org/job/periodic-mirror-python33/ | 19:16 |
| clarkb | anteaya: we have silly tape delays in this country | 19:16 |
| anteaya | no | 19:16 |
| jeblair | dhellmann: and it has the same situation | 19:16 |
| *** hogepodge has joined #openstack-infra | 19:16 | |
| jeblair | dhellmann: so this broke on jan 30. | 19:16 |
| clarkb | anteaya: is it over yet? iirc it started a couple hours ago | 19:16 |
| anteaya | it was 1-0 from the first period, ran downstairs twice to see some plays | 19:16 |
| mordred | I think I briefly looked at mirror fails a little while ago - but I can't remember what the issue was | 19:16 |
| dhellmann | jeblair: I believe it's trying to run virtualenv, but that's as far as I got | 19:16 |
| anteaya | the clock ran out, US pulled their goalie I think- was upstairs for it | 19:16 |
| anteaya | clarkb: yes over, 1-0 Canada | 19:16 |
| jeblair | mordred: cool. this one is all yours then. :) | 19:16 |
| mordred | oh wow - well... | 19:17 |
| mordred | https://jenkins.openstack.org/job/periodic-mirror-python33/203/console | 19:17 |
| mordred | is pretty spectacular | 19:17 |
| *** leifmadsen has quit IRC | 19:17 | |
| mordred | OSError: [Errno 2] No such file or directory | 19:17 |
| mordred | what file?? | 19:17 |
| anteaya | clarkb: some very good hockey from what I saw | 19:17 |
| anteaya | olympic hockey is the best | 19:18 |
| anteaya | mordred: yeah, that is doug's question | 19:18 |
| dhellmann | mordred: it looks like it's trying to run virtualenv? line 255 | 19:18 |
| clarkb | anteaya: you guys beat both our teams :( | 19:19 |
| anteaya | yeah | 19:19 |
| anteaya | only in hockey | 19:19 |
| clarkb | we'll let you have it :) | 19:19 |
| anteaya | US beats us lots in most every thing else | 19:20 |
| anteaya | thanks | 19:20 |
| anteaya | you get to keep beiber | 19:20 |
| anteaya | you saw that chicago billboard in reddit? | 19:20 |
| clarkb | down to 2 worker process on 01-16 now. going to leave it that way so that we can get stable numbers regardless of backlog | 19:20 |
| anteaya | loser keeps bieber? | 19:20 |
| clarkb | anteaya: I did see that | 19:20 |
| anteaya | we were motivated for the win | 19:21 |
| mordred | dhellmann: yeah | 19:21 |
| mordred | dhellmann, jeblair: there is no virtualenv on that box | 19:21 |
| anteaya | gold medal game against sweden 8am Sunday EST | 19:21 |
| anteaya | good hockey that game | 19:22 |
| *** SumitNaiksatam has joined #openstack-infra | 19:22 | |
| anteaya | eww figure skating, tv is off again | 19:22 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Add oslotest library https://review.openstack.org/75487 | 19:23 |
| *** khyati has joined #openstack-infra | 19:24 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Nodepool should clone the config repo from git.o.o https://review.openstack.org/75488 | 19:25 |
| fungi | that's ^ hand-patched on nodepool.o.o with puppet stopped so that it will work | 19:25 |
| fungi | (minor oversight! gerrit apparently doesn't do plain git protocol) | 19:25 |
| *** vkozhukalov has joined #openstack-infra | 19:25 | |
| clarkb | fungi: why not do https for all of it? | 19:26 |
| *** jergerber has joined #openstack-infra | 19:26 | |
| *** dpyzhov has joined #openstack-infra | 19:26 | |
| clarkb | with a shallow clone that should be dirt cheap | 19:26 |
| fungi | clarkb: performance on centos6, among other reasons | 19:26 |
| clarkb | oh right | 19:26 |
| clarkb | :( | 19:26 |
| clarkb | fungi: it just seems weird to willingly switch https to http | 19:27 |
| jeblair | fungi: why the https->http change for install puppet? | 19:27 |
| fungi | we could do https for all of it prepare_node.sh, but the other caching is happening via git:// | 19:27 |
| clarkb | fungi: I think that is more reasonable | 19:27 |
| clarkb | especially since the shallow clone shouldn't be too bad even over https (watch me regret saying that) | 19:28 |
| fungi | jeblair: i'll happily switch it to https. we changed the git repo caching to not https, but i suppose we can encrypt/validate the bootstrap/config retrieval | 19:28 |
| jeblair | i'm okay with the git change | 19:28 |
| fungi | doing | 19:28 |
| jeblair | i'm jut wondering about the wget change | 19:28 |
| jeblair | to be perfectly clear, i think we should "wget https" and "git clone git://" | 19:28 |
| fungi | seemed unnecessary to retrieve it via https when it's going to then run other stuff as root retrieved via a nonencrypted/validated protocol, but i can put that back | 19:29 |
| jraim | sdague ping | 19:29 |
| jeblair | fungi: if we don't now, we'll just end up doing it later, or someone else will | 19:29 |
| *** UtahDave has quit IRC | 19:29 | |
| jeblair | fungi: plus, if that gets copied, etc. | 19:29 |
| fungi | just trying not to present any false sense of security there by implying we've protected ourselves from anything | 19:30 |
| jeblair | fungi: someday we won't have to run on centos6 and we can change everything to https | 19:30 |
| fungi | sure, i can see that reasoning | 19:30 |
| *** leifmadsen has joined #openstack-infra | 19:30 | |
| *** khyati has quit IRC | 19:30 | |
| *** leifmadsen has quit IRC | 19:31 | |
| *** leifmadsen has joined #openstack-infra | 19:31 | |
| anteaya | the gate is all happy again and some of check is happy again | 19:31 |
| anteaya | will take a while for all the older patches in check to finish up | 19:31 |
| fungi | the bare-centos6 image in rax-iad rebuild succesfully. fighting with dfw now because i think we've actually overallocated our quota... calculations neglected to take into account that we have other long-lived (trusted) slaves in that region besides just the four precisepy3k slaves | 19:33 |
| anteaya | and I was wrong Canada's goal came in the 2nd period, not the 1st | 19:33 |
| fungi | i'll fix that in a bit | 19:33 |
| *** khyati has joined #openstack-infra | 19:33 | |
| anteaya | how are you going to fix that we are overallocated in rax-dfw? | 19:34 |
| fungi | anteaya: by reducing the nodepool limit for that provider | 19:34 |
| anteaya | ah | 19:35 |
| *** masayukig has joined #openstack-infra | 19:35 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Nodepool should clone the config repo from git.o.o https://review.openstack.org/75488 | 19:37 |
| *** ociuhandu has quit IRC | 19:37 | |
| *** ArxCruz has quit IRC | 19:38 | |
| *** masayukig has quit IRC | 19:39 | |
| *** mrodden has joined #openstack-infra | 19:40 | |
| *** dpyzhov has quit IRC | 19:42 | |
| *** ArxCruz has joined #openstack-infra | 19:42 | |
| *** sarob has quit IRC | 19:43 | |
| fungi | is there any way to tell what our actual quotas are in a given provider/region via novaclient? | 19:46 |
| fungi | nevermind. nova quota-show... duh | 19:47 |
| fungi | oh, except that must not be implemented | 19:48 |
| fungi | "ERROR: Not found (HTTP 404)" | 19:48 |
| clarkb | fungi: try with keystone client | 19:50 |
| clarkb | I believe keystone tracks that data too | 19:51 |
| dolphm | clarkb: keystone does not (yet) | 19:51 |
| fungi | yeah, neither rs or hp seem to implement quota-show. must be too new | 19:52 |
| fungi | anyway, i have reason to believe that our quota in dfw is way, way, way lower than we thought | 19:52 |
| jeblair | fungi: absolute-limits | 19:52 |
| jeblair | nova absolute-limits | 19:52 |
| fungi | jeblair: yep--woeks great. thanks! | 19:53 |
| fungi | maxServerMeta | 40 | 19:53 |
| fungi | patch coming | 19:53 |
| *** sarob has joined #openstack-infra | 19:54 | |
| clarkb | dolphm: :( I thought it did | 19:54 |
| dolphm | clarkb: quotas are still distributed among the services, and keystone keeps none of its own | 19:55 |
| dolphm | clarkb: there's been some discussion to centralize quota data, but keep distributed enforcement | 19:55 |
| *** lnxnut has quit IRC | 19:56 | |
| *** protux_ has joined #openstack-infra | 19:57 | |
| *** protux_ has quit IRC | 19:57 | |
| *** lnxnut has joined #openstack-infra | 19:57 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Fix pip on py3k/pypy nodes https://review.openstack.org/75213 | 19:57 |
| *** protux has quit IRC | 19:57 | |
| *** protux_ has joined #openstack-infra | 19:57 | |
| fungi | oh, wait, i need to be looking at maxTotalInstances | 19:57 |
| clarkb | fungi: https://review.openstack.org/#/c/75449/ did you asnwer about why that wasn't approved earlier? I assume just that you didn't see the second +2 because it happened just before yours but want ot make sure there wasn't a more important reason | 19:57 |
| clarkb | fungi: you actually have to look at all the values | 19:58 |
| clarkb | since hitting any one of them stops the presses | 19:58 |
| *** harlowja is now known as harlowja_away | 19:59 | |
| *** sarob has quit IRC | 19:59 | |
| *** sarob has joined #openstack-infra | 20:00 | |
| openstackgerrit | A change was merged to openstack-infra/config: Nodepool should clone the config repo from git.o.o https://review.openstack.org/75488 | 20:00 |
| *** lnxnut has quit IRC | 20:01 | |
| *** markmcclain has joined #openstack-infra | 20:02 | |
| *** markwash_ has joined #openstack-infra | 20:03 | |
| openstackgerrit | A change was merged to openstack-infra/gear: Geard: Report packet timing to statsd https://review.openstack.org/75449 | 20:04 |
| *** sarob has quit IRC | 20:04 | |
| *** mgagne has quit IRC | 20:04 | |
| *** markwash has quit IRC | 20:05 | |
| *** markwash_ is now known as markwash | 20:05 | |
| *** ok_delta has joined #openstack-infra | 20:06 | |
| *** harlowja_away is now known as harlowja | 20:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add zm0{1,2} to cacti https://review.openstack.org/75482 | 20:10 |
| *** ok_delta has quit IRC | 20:11 | |
| *** hashar has joined #openstack-infra | 20:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: make ant use java 7 https://review.openstack.org/75478 | 20:12 |
| * clarkb grabs lunch | 20:12 | |
| *** tjones has quit IRC | 20:12 | |
| fungi | hmm, yep, so i can't find anything obvious in our quotas we're exceeding, but still getting a vague "OverLimit: OverLimit Retry... (HTTP 413)" in dfw | 20:12 |
| fungi | maybe nova api call frequency? | 20:13 |
| *** mrmartin has quit IRC | 20:14 | |
| *** afrittoli has quit IRC | 20:18 | |
| *** lnxnut has joined #openstack-infra | 20:21 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct nodepool max-servers in rax-dfw https://review.openstack.org/75500 | 20:28 |
| fungi | anyway, there's ^ a more realistic max-servers for nodepool in rax-dfw | 20:29 |
| *** cadenzajon has quit IRC | 20:29 | |
| sdague | jraim: what's up? | 20:29 |
| fungi | actually, i guess i should reduce it by the number of images we might build simultaneously | 20:30 |
| jraim | sdague: was wondering if you had a sec to look at our newest devstack-gate patch and let me know what you think | 20:30 |
| jraim | sdague: https://review.openstack.org/#/c/74530 | 20:30 |
| sdague | jraim: can barbican run without keystone? | 20:31 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Correct nodepool max-servers in rax-dfw https://review.openstack.org/75500 | 20:31 |
| jraim | sdague: it could I guess, but not in a useful way | 20:32 |
| jraim | all auth and auth is keystone based | 20:32 |
| sdague | ok, so you'll need to add keystone to the services list | 20:32 |
| jraim | so solum didn't have it, do they not need keystone? | 20:32 |
| *** starmer has quit IRC | 20:32 | |
| anteaya | fungi: you are happy with 172? | 20:33 |
| fungi | jraim: i think solum is only using devstack servers to get the benefit of some of the machinery and repositories, but doesn't actually run devstack | 20:33 |
| sdague | jraim: apparently not | 20:33 |
| jraim | ahh, that makes more sense. I was wondering if it was being added somewhere else. Okay, I'll add keystone for enabled services | 20:33 |
| sdague | https://review.openstack.org/#/c/74530/ I put in very specific feedback | 20:33 |
| sdague | jraim: if there are any other openstack services you need or want, add those to the ENABLED_SERVICES list as well | 20:34 |
| jraim | okay. I think keystone would be it | 20:34 |
| sdague | but that's basically correct otherwise | 20:34 |
| *** starmer has joined #openstack-infra | 20:34 | |
| jraim | okay, we'll go make the changes | 20:34 |
| jraim | thanks for the time | 20:34 |
| sdague | yep | 20:34 |
| fungi | anteaya: yes, 192 quota - 5 trusted slaves - 2 dev slaves - 4 remaining py3k slaves - 5 images which might build at once - 1 more for insurance | 20:34 |
| fungi | (and that assumes our image builds all kick off at a point when we're maxxed out on our capacity in dfw) | 20:35 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 20:35 | |
| *** masayukig has joined #openstack-infra | 20:36 | |
| *** alexpilotti has quit IRC | 20:36 | |
| *** andreaf has quit IRC | 20:37 | |
| *** andreaf has joined #openstack-infra | 20:37 | |
| anteaya | okay that is 17 slaves and a buffer of 3, for a difference of 20 between nodepool settings and max quota | 20:38 |
| anteaya | unless I can't count anymore, which is possible | 20:38 |
| *** dkliban has quit IRC | 20:39 | |
| *** ociuhandu has joined #openstack-infra | 20:40 | |
| *** masayukig has quit IRC | 20:41 | |
| fungi | yeah. breathing room of a few | 20:41 |
| anteaya | sounds good | 20:41 |
| anteaya | I like breathing room | 20:41 |
| *** pballand has joined #openstack-infra | 20:43 | |
| pballand | I'm trying to get 'make' to fire as part of tox/PBR - can anyone point me at some applicable docs/examples? | 20:45 |
| anteaya | make? | 20:45 |
| anteaya | can you share what instructions you are using for tox/PBR that make is a suggested command? | 20:46 |
| *** thomasem has quit IRC | 20:46 | |
| mordred | pballand: what are you trying to accomplish? I'm not sure we've ever had anyone want to do that before | 20:46 |
| *** skraynev_afk is now known as skraynev | 20:47 | |
| fungi | sounds like maybe using pbr in something which isn't a pure-python package | 20:47 |
| pballand | anteaya: actually, 'make' is optional, but I need to generate some code based on a metadata file | 20:47 |
| anteaya | pballand: I think we are keen to hear the context | 20:47 |
| anteaya | you have us perplexed | 20:47 |
| pballand | mordred: actually we discussed this a few weeks (months?) ago, but it got pushed to my backburner | 20:47 |
| mordred | pballand: ah. I think I might remember something about this | 20:48 |
| mordred | you're going to be much better off just writing a pbr hook in python | 20:48 |
| pballand | the project (stackforge/congress) has a grammar file, which uses a java program to generate a python parser | 20:48 |
| mordred | there's an example of using a local hook in the neutron source tree | 20:48 |
| mordred | in fact, I think I might have made an example patch for you at some point/ | 20:49 |
| pballand | mordred: yeah you said you were going to, but I never heard back | 20:49 |
| mordred | or - maybe I didn't | 20:50 |
| mordred | :) | 20:50 |
| openstackgerrit | Chad Lung proposed a change to openstack-infra/config: Add DevStack job for Barbican https://review.openstack.org/74530 | 20:50 |
| *** ekarlso has quit IRC | 20:50 | |
| *** ekarlso has joined #openstack-infra | 20:50 | |
| pballand | where would I find the example in neutron? | 20:50 |
| mordred | pballand: http://git.openstack.org/cgit/openstack/neutron/tree/setup.cfg#n78 | 20:50 |
| mordred | and | 20:50 |
| *** atiwari has quit IRC | 20:50 | |
| mordred | http://git.openstack.org/cgit/openstack/neutron/tree/neutron/hooks.py | 20:51 |
| *** oubiwann has quit IRC | 20:52 | |
| openstackgerrit | Chad Lung proposed a change to openstack-infra/config: Add DevStack job for Barbican https://review.openstack.org/74530 | 20:52 |
| pballand | ok, so it looks like calling os.popen from setup_hook is "reasonable" ? | 20:52 |
| *** oubiwann has joined #openstack-infra | 20:52 | |
| *** starmer has quit IRC | 20:53 | |
| mordred | sure | 20:54 |
| mordred | just beware that it's going to run on every invocation of setup.py - you may eventually want to dig in to figure out how to only run sometimes - or maybe put a freshness check in | 20:55 |
| mordred | or something | 20:55 |
| mordred | :) | 20:55 |
| mordred | I'd give more direct help, but I'm a bit slammeded | 20:55 |
| pballand | thanks, I think I can take it from there | 20:55 |
| pballand | I really appreciate the pointer | 20:56 |
| *** esker has quit IRC | 20:56 | |
| *** esker has joined #openstack-infra | 20:57 | |
| *** e0ne has joined #openstack-infra | 20:59 | |
| *** ociuhandu has quit IRC | 21:01 | |
| *** esker has quit IRC | 21:01 | |
| *** atiwari has joined #openstack-infra | 21:03 | |
| davidlenwell | mordred is running into human scaling problems.. | 21:07 |
| *** rfolco has quit IRC | 21:07 | |
| fungi | i've been having no problem scaling out. i need to scale back a bit | 21:07 |
| openstackgerrit | A change was merged to openstack-infra/config: Correct nodepool max-servers in rax-dfw https://review.openstack.org/75500 | 21:09 |
| fungi | speaking of scaling, i strongly suspect we're overrunning the api rate limit for our openstackjenkins account in rax-dfw... how can i tell for sure? right now we seem to be able to build no new nodepool nodes there (the count has basically dwindled to 0) | 21:09 |
| fungi | every call to create a node or an image gets back an overlimit response | 21:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Keep current and previous snapshot images https://review.openstack.org/75510 | 21:10 |
| jeblair | fungi: we should have an enormous api rate | 21:10 |
| jeblair | fungi: however, you are correct | 21:12 |
| jeblair | fungi: 'nova rate-limits' | 21:12 |
| fungi | aha | 21:12 |
| jeblair | | POST | /servers | 5000 | 0 | DAY | 2014-02-21T23:21:07.394Z | | 21:12 |
| mordred | I usually fix that by mentioning p-v-o in channel | 21:12 |
| mordred | WOW | 21:12 |
| mordred | 5000 a day? that's not enough | 21:12 |
| jeblair | fungi: apparently we are only allowed to create 5000 servers in a day | 21:12 |
| jeblair | mordred: i think we need to remove those hyphens | 21:13 |
| *** amcrn has quit IRC | 21:13 | |
| mordred | hey pvo ... you around my friend? | 21:13 |
| *** cadenzajon has joined #openstack-infra | 21:13 | |
| jeblair | ord and iad both have values like | POST | /servers | 5000 | 3768 | DAY | 2014-02-22T00:12:54.065Z | | 21:13 |
| jeblair | so quite a bit left | 21:13 |
| fungi | interesting | 21:14 |
| *** zarric has quit IRC | 21:14 | |
| fungi | i wonder what happened | 21:14 |
| jeblair | fungi: why is ord at max-servers=60? | 21:15 |
| fungi | well, one of the problem devstack nodes was in dfw, so maybe the fast failing those jobs were doing tapped us out | 21:15 |
| fungi | jeblair: good question--we seem to have upped that yeah? | 21:15 |
| fungi | maxTotalInstances | 100 | 21:16 |
| *** e0ne has quit IRC | 21:16 | |
| fungi | so we can bump that safely to 92. patch coming | 21:16 |
| *** oubiwann has quit IRC | 21:16 | |
| clarkb | lunch is slow... | 21:17 |
| *** e0ne has joined #openstack-infra | 21:18 | |
| anteaya | clarkb: have you tried rebooting? | 21:19 |
| *** pdmars has quit IRC | 21:19 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Nodepool should use more of the rax-ord quota https://review.openstack.org/75513 | 21:20 |
| *** thomasem has joined #openstack-infra | 21:20 | |
| *** tjones has joined #openstack-infra | 21:20 | |
| *** e0ne has quit IRC | 21:22 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added parallelization options https://review.openstack.org/75514 | 21:22 |
| jeblair | fungi: oh, we fixed the same problem in https://review.openstack.org/#/c/75479/1 and https://review.openstack.org/#/c/75510/ | 21:23 |
| fungi | jeblair: yeah, i just abandoned mine | 21:23 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Keep current and previous snapshot images https://review.openstack.org/75510 | 21:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Nodepool should use more of the rax-ord quota https://review.openstack.org/75513 | 21:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable statsd for jenkins-log-client https://review.openstack.org/75481 | 21:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch Zuul geard logging to DEBUG level https://review.openstack.org/75455 | 21:27 |
| openstackgerrit | A change was merged to openstack-infra/gear: Use client_id as part of the logger name https://review.openstack.org/73118 | 21:29 |
| jeblair | the zuul gear statsd change is going to explode the graphite metrics a bit. (+ 3 * num_jobs) == +6000 roughly | 21:29 |
| *** melwitt has quit IRC | 21:29 | |
| *** smarcet has quit IRC | 21:30 | |
| fungi | and /var/lib/graphite/storage is at 53% used. do we need to add a second pv? | 21:30 |
| *** melwitt has joined #openstack-infra | 21:30 | |
| jeblair | so that's about another 19G, but we have 472 available so we're probably ok | 21:30 |
| *** markmcclain has quit IRC | 21:31 | |
| *** melwitt has quit IRC | 21:32 | |
| fungi | ahh, okay. a bit sounded like a possible understatement. yeah we've got >200k inodes on there, so assuming most are whisper files 6k more of them is a fairly small percentage increase overall | 21:32 |
| *** lcheng has quit IRC | 21:32 | |
| *** melwitt has joined #openstack-infra | 21:32 | |
| jeblair | fungi: i didn't know whether it was under or overstatement at the time i made it; just thinking out loud. :) | 21:33 |
| fungi | also, forgot to mention, but puppet agent is running on nodepool.o.o again for the past little while, so it should in theory get those changes we merged | 21:33 |
| *** lcheng has joined #openstack-infra | 21:33 | |
| jeblair | tagged gear 0.5.2 | 21:33 |
| *** lcostantino has quit IRC | 21:34 | |
| * anteaya goes for a walk | 21:36 | |
| *** masayukig has joined #openstack-infra | 21:36 | |
| *** dolphm is now known as dolphm_503 | 21:38 | |
| *** skraynev is now known as skraynev_afk | 21:41 | |
| *** masayukig has quit IRC | 21:41 | |
| fungi | i see a recent uptick in building node count, so nodepoold probably picked up the max-servers increase for ord | 21:44 |
| *** starmer has joined #openstack-infra | 21:44 | |
| *** jeblair has quit IRC | 21:45 | |
| *** markmcclain has joined #openstack-infra | 21:46 | |
| *** DinaBelova is now known as DinaBelova_ | 21:48 | |
| *** mgagne has joined #openstack-infra | 21:49 | |
| *** jeblair has joined #openstack-infra | 21:50 | |
| *** jcooley_ has joined #openstack-infra | 21:50 | |
| * jeblair discovers new and interesting things about the 'server' command in irssi. | 21:50 | |
| *** hartbot has joined #openstack-infra | 21:51 | |
| *** skraynev_afk is now known as skraynev | 21:52 | |
| hartbot | Hey folks. I have a weird log on my patch: http://logs.openstack.org/62/75262/3/check/gate-nova-docs/4866f14/console.html … and it looks like something really died a horrible death: "[SCP] ‘doc/build/html/**’ doesn’t match anything, but ‘**’ does. Perhaps that’s what you mean?" | 21:52 |
| jeblair | hartbot: "recheck bug 1282876" (see channel topic) | 21:53 |
| *** yassine has quit IRC | 21:54 | |
| hartbot | okay sorry | 21:55 |
| *** mfer has quit IRC | 21:56 | |
| jeblair | np | 21:56 |
| *** lyxus has joined #openstack-infra | 21:56 | |
| *** hartbot has left #openstack-infra | 21:57 | |
| lyxus | Hello Folks, I was wondering what was the difference. If I am running something like testr run tempest.api.network.test_networks.NetworksIpV6TestJSON.test_list_networks, the test will be run without problems. However if the [gate,smoke] is appended at the end. The test won't be run. Any ideas/pointers? | 21:58 |
| *** melwitt has quit IRC | 21:59 | |
| *** melwitt has joined #openstack-infra | 22:00 | |
| clarkb | ok back | 22:03 |
| clarkb | ramen acquired would be slow again | 22:04 |
| clarkb | lyxus: you may have better luck getting an answer over in #openstack-qa | 22:05 |
| clarkb | not sure how tempest is configured around that | 22:05 |
| lyxus | clarkb, thanks will ask | 22:05 |
| *** sarob has joined #openstack-infra | 22:06 | |
| fungi | okay, since the broken has subsided somewhat, i'm going to head out and grab dinner... back in a while | 22:06 |
| clarkb | fungi: have fun, I suppose we can pick up on ES on monday | 22:07 |
| fungi | clarkb: yeah, today was a bit of a bust | 22:07 |
| clarkb | which shouldn't be a problem since ES is least busy over the weekend | 22:07 |
| clarkb | fungi: ya :/ | 22:07 |
| *** hashar has quit IRC | 22:09 | |
| *** jnoller has quit IRC | 22:09 | |
| jeblair | clarkb: i upgraded gear on logstash, so whenever the queue gets sufficiently low, we should be good to restart that and pick up the statsd changes | 22:09 |
| clarkb | jeblair: will do, probably sometiume tomorrow when it is caught up | 22:10 |
| jeblair | same thing for zuul | 22:10 |
| *** CaptTofu has quit IRC | 22:10 | |
| *** sabari has joined #openstack-infra | 22:10 | |
| *** sabari_ has joined #openstack-infra | 22:10 | |
| *** sabari has quit IRC | 22:11 | |
| *** sabari has joined #openstack-infra | 22:11 | |
| *** CaptTofu has joined #openstack-infra | 22:12 | |
| openstackgerrit | gordon chung proposed a change to openstack/requirements: bump pycadf requirement to 0.4.1 https://review.openstack.org/75201 | 22:12 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 22:14 | |
| openstackgerrit | gordon chung proposed a change to openstack/requirements: bump pycadf requirement to 0.4.1 https://review.openstack.org/75201 | 22:15 |
| *** sarob has quit IRC | 22:15 | |
| *** sarob has joined #openstack-infra | 22:16 | |
| *** wenlock has quit IRC | 22:16 | |
| openstackgerrit | Matt Ray proposed a change to openstack-infra/config: Create new Chef cookbook-openstack-integration-test for Tempest support. https://review.openstack.org/68791 | 22:17 |
| *** jgrimm has quit IRC | 22:17 | |
| *** yolanda has quit IRC | 22:19 | |
| *** westmaas has quit IRC | 22:19 | |
| *** westmaas has joined #openstack-infra | 22:20 | |
| *** oubiwann-home is now known as oubiwann-lamba | 22:21 | |
| *** skraynev is now known as skraynev_afk | 22:21 | |
| *** masayukig has joined #openstack-infra | 22:21 | |
| *** gokrokve has quit IRC | 22:22 | |
| *** sarob has quit IRC | 22:22 | |
| *** gokrokve has joined #openstack-infra | 22:22 | |
| *** sarob has joined #openstack-infra | 22:23 | |
| *** fbo is now known as fbo_away | 22:24 | |
| *** dolphm_503 is now known as dolphm | 22:24 | |
| flashgordon | so looks like bug 1282876 was seen in gate | 22:24 |
| flashgordon | so working on a e-r query for it | 22:25 |
| jeblair | flashgordon: is that worthwhile? i think it's going to be difficult, and the problem should be fixed. | 22:25 |
| flashgordon | but not sure what to fingerrint in http://logs.openstack.org/94/67994/3/gate/gate-tempest-dsvm-large-ops/29583fb/console.html | 22:25 |
| flashgordon | jeblair: if we can find a fingerprint, yes its worth it | 22:26 |
| jeblair | flashgordon: the only possibly fingerprintable stuff is going to be in the setup logs. you'll need to add them to logstash. | 22:26 |
| jeblair | flashgordon: why? | 22:26 |
| flashgordon | it helps give us a better picture of the unkown failures http://status.openstack.org/elastic-recheck/data/uncategorized.html | 22:26 |
| openstackgerrit | David Lenwell proposed a change to openstack-infra/config: Adding refstack into stackforge.. yay! https://review.openstack.org/75226 | 22:26 |
| jeblair | i see | 22:26 |
| *** gokrokve has quit IRC | 22:27 | |
| *** sarob has quit IRC | 22:27 | |
| flashgordon | jeblair: but we don't ahve the right logs in ES | 22:28 |
| *** hogepodge_ has joined #openstack-infra | 22:28 | |
| *** mbacchi has quit IRC | 22:29 | |
| flashgordon | so never mind | 22:29 |
| *** hogepodge has quit IRC | 22:29 | |
| *** hogepodge_ is now known as hogepodge | 22:29 | |
| *** sabari_ has quit IRC | 22:29 | |
| jeblair | flashgordon: still worth doing for the future, but we should probably add timestamps to them first | 22:29 |
| flashgordon | jeblair: yeah | 22:29 |
| jeblair | here's an old change that started on that: https://review.openstack.org/#/c/57770/ | 22:30 |
| flashgordon | jeblair: I would offer to help but maybe next week | 22:31 |
| flashgordon | too much on my plate | 22:31 |
| flashgordon | not that you don't have too much also | 22:31 |
| jeblair | flashgordon: yep. i'll rebase that and restore it and we'll see how it goes. | 22:32 |
| flashgordon | so another gate issue | 22:32 |
| flashgordon | http://logs.openstack.org/59/66959/9/gate/gate-tempest-dsvm-large-ops/63fbc81/console.html#_2014-02-18_22_42_12_594 | 22:32 |
| flashgordon | some jobs seem to have not collected the openstack logs | 22:32 |
| clarkb | they weren't there to be collected | 22:33 |
| flashgordon | clarkb: but ifyou scroll up the error detector in log files ruan | 22:33 |
| flashgordon | ran and worked | 22:34 |
| clarkb | flashgordon: right so possible permissions issue | 22:34 |
| flashgordon | http://logs.openstack.org/59/66959/9/gate/gate-tempest-dsvm-large-ops/63fbc81/console.html#_2014-02-18_22_41_56_964 | 22:34 |
| clarkb | or similar | 22:34 |
| *** sarob has joined #openstack-infra | 22:34 | |
| flashgordon | clarkb: ohhh | 22:34 |
| clarkb | jenkins did not see any files | 22:34 |
| flashgordon | that makes sense | 22:34 |
| clarkb | and there were other jobs iwth permissions issues | 22:34 |
| flashgordon | any good way to fingerprint taht | 22:35 |
| jeblair | clarkb: why no cleanup-host log? | 22:35 |
| clarkb | jeblair: oh good point | 22:35 |
| clarkb | flashgordon: not without figuring out more of what happened | 22:35 |
| clarkb | cleanup didn't run for some reason and that didn't copy the logs? | 22:36 |
| *** sarob_ has joined #openstack-infra | 22:36 | |
| *** julim has quit IRC | 22:36 | |
| flashgordon | ahh | 22:37 |
| jeblair | clarkb: that or something happened to the cleanup log | 22:37 |
| clarkb | btw, am I a horrible person for wanting to remove the log checker thing? | 22:37 |
| clarkb | its noise | 22:37 |
| *** SumitNaiksatam has quit IRC | 22:37 | |
| jeblair | clarkb: it will be useful when it fails | 22:37 |
| clarkb | makes it really hard to see actual failures in console.html | 22:38 |
| clarkb | jeblair: I disagree, I want to see the high level fail in console.html | 22:38 |
| flashgordon | clarkb: I use the testr htm output for that usually | 22:38 |
| jeblair | clarkb: i don't think we disagree about that, but that's no reason to remove it. perhaps it should output to its own log file. | 22:38 |
| clarkb | jeblair: that I would be happy with | 22:38 |
| clarkb | I am fine grepping the data out, I just odn't think it belongs in the console log | 22:39 |
| jeblair | clarkb: also, the purpose is that there should be no errors, so there should be no output. | 22:39 |
| *** sarob has quit IRC | 22:39 | |
| clarkb | jeblair: right but then we didn't gate on it so that never happened | 22:39 |
| jeblair | clarkb: again, this is only an issue because it's not actually failing tests | 22:39 |
| jeblair | i assume we will do so at some point. | 22:39 |
| clarkb | flashgordon: the problem with thta is in cases like ^ the testr output isn't helpful | 22:40 |
| clarkb | flashgordon: it is an incomplete high level picture :) | 22:40 |
| jeblair | # TODO(sdague): post icehouse-2 we can talk about turning | 22:40 |
| jeblair | # this back on, but right now it is violating the do no harm | 22:40 |
| jeblair | # principle. | 22:40 |
| jeblair | it is post icehouse-2. | 22:40 |
| *** SumitNaiksatam has joined #openstack-infra | 22:40 | |
| jeblair | perhaps post icehouse-3 we should talk about it again. | 22:40 |
| sdague | yeh, so our biggest issue is getting the whitelist accurate again | 22:41 |
| clarkb | s/again// | 22:42 |
| jeblair | sdague, clarkb: to tackle that, what if we searched ES for "*** Not Whitelisted ***" and built it from that | 22:43 |
| clarkb | doesn't look like all of the console output is saying not whitelisted (not sure why that is) but I think having something liek that to key off of is a great idea | 22:43 |
| sdague | jeblair: yeh, honestly what we really need is a tool that builds that list from ES | 22:44 |
| jeblair | sdague: should be pretty easy to do. | 22:44 |
| sdague | well, yuo have to build regexes | 22:44 |
| jeblair | clarkb: i think the ones that aren't already whitelisted have it | 22:44 |
| sdague | because we're not enforcing on ES | 22:44 |
| sdague | we're enforcing on the logs | 22:44 |
| clarkb | jeblair: oh I see | 22:45 |
| sdague | it's not rocket science, just a couple of days probably | 22:45 |
| flashgordon | clarkb: for http://logs.openstack.org/59/66959/9/gate/gate-tempest-dsvm-large-ops/63fbc81/console.html#_2014-02-18_22_42_12_531 | 22:45 |
| flashgordon | the best fingerprint I can find is message:"ERROR nova.wsgi [-] Could not bind to 0.0.0.0:8773" AND filename:"console.html" | 22:45 |
| clarkb | flashgordon: I don't think that is related | 22:45 |
| flashgordon | there is a spike for that for a few hours | 22:45 |
| clarkb | but it might be | 22:46 |
| clarkb | oh you know what | 22:46 |
| *** gokrokve has joined #openstack-infra | 22:46 | |
| jeblair | sdague: yeah, get all the non whitelisted logs from es; have a human regexify them, add them to whitelist, gate. probably a day or two of work. if it's more than that we should just give up and start over. ;) | 22:46 |
| clarkb | I wonder, if that was a test that ran on a host that was previously used | 22:46 |
| flashgordon | clarkb: figure it could be permissions based or something as well | 22:46 |
| clarkb | would explain permission trouble | 22:46 |
| clarkb | and why 8773 wasn't available | 22:46 |
| flashgordon | clarkb: ahh | 22:46 |
| flashgordon | does the timeframe fit? | 22:46 |
| flashgordon | also is there abug for that? | 22:46 |
| clarkb | fungi: ^ | 22:47 |
| clarkb | fungi is at dinner, I thought there was a bug but he had it sorted by the time I was awake | 22:47 |
| clarkb | which doesn't quite match that fingerprint oh wait this was why we jenkins downgraded | 22:48 |
| clarkb | flashgordon: I think it does match | 22:48 |
| jeblair | flashgordon: oh i see what that timestamp change didn't work before | 22:50 |
| jeblair | 2013-11-21 22:02:51.830 | /opt/stack/new/devstack-gate/devstack-vm-gate-wrap.sh: line 41: gawk: command not found | 22:50 |
| *** hogepodge_ has joined #openstack-infra | 22:50 | |
| *** sarob has joined #openstack-infra | 22:51 | |
| *** hogepodge has quit IRC | 22:51 | |
| *** hogepodge_ is now known as hogepodge | 22:51 | |
| *** basha has joined #openstack-infra | 22:51 | |
| jeblair | hrm. gawk is installed now. | 22:51 |
| flashgordon | clarkb: what was the bug for downgrading jenkins? | 22:52 |
| flashgordon | bug number | 22:52 |
| *** starmer has quit IRC | 22:52 | |
| clarkb | flashgordon: no clue, I was driving between states when that happened | 22:52 |
| clarkb | flashgordon: I can dig it up probably though | 22:52 |
| *** sarob_ has quit IRC | 22:52 | |
| flashgordon | jeblair: ^ | 22:52 |
| *** masayukig has quit IRC | 22:52 | |
| *** masayukig has joined #openstack-infra | 22:53 | |
| jeblair | dunno. it didn't end up in the log. would have to search launchpad | 22:53 |
| *** wenlock has joined #openstack-infra | 22:54 | |
| *** CaptTofu has quit IRC | 22:54 | |
| flashgordon | took a peak at launchpad wasn't sure waht to search for | 22:55 |
| flashgordon | jenkins downgrade came up short | 22:56 |
| *** masayukig has quit IRC | 22:57 | |
| *** sarob has quit IRC | 22:58 | |
| *** jcooley_ has quit IRC | 22:59 | |
| jeblair | flashgordon: i don't see one | 22:59 |
| * flashgordon files a resolved bug | 23:00 | |
| fungi | clarkb: yeah, that was the cause of the jenkins downgrade from most recent bleeding series to most recent oozing wound series | 23:01 |
| *** prad_ has quit IRC | 23:01 | |
| fungi | flashgordon: and yeah, i probably didn't open a bug--we caught it pretty quickly | 23:02 |
| flashgordon | https://bugs.launchpad.net/openstack-ci/+bug/1283283 | 23:02 |
| flashgordon | I put in all the info I know ... which is almost nothing | 23:02 |
| *** sarob has joined #openstack-infra | 23:03 | |
| anteaya | clarkb: http://i.imgur.com/9b7l5Kb.jpg | 23:04 |
| fungi | basically as soon as i was done upgrading jenkins masters, i started downgrading them | 23:05 |
| flashgordon | fungi: haha | 23:05 |
| *** basha has quit IRC | 23:05 | |
| anteaya | no good work goes unpunished | 23:05 |
| fungi | where jenkins security updates are concerned, that is basically true | 23:07 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1283283 https://review.openstack.org/75533 | 23:07 |
| flashgordon | fungi: ^ | 23:07 |
| Mithrandir | it's almost as if somebody should have a jenkins-stable-but-not-ancient series. | 23:07 |
| flashgordon | I got 150 hits or so | 23:07 |
| fungi | Mithrandir: these days the lts is actually not ancient, so it's what we switched to | 23:08 |
| *** gokrokve has quit IRC | 23:08 | |
| fungi | last time this came up, it was ancient | 23:08 |
| *** gokrokve has joined #openstack-infra | 23:08 | |
| Mithrandir | fungi: surely the LTS is from 2012, which is from when licen first appearet? | 23:08 |
| *** thomasem has quit IRC | 23:09 | |
| fungi | Mithrandir: jenkins long-term stable is actually called "hudson" ;) | 23:10 |
| jeblair | zing | 23:10 |
| flashgordon | with that patch we should get close to a 80% classification rate again | 23:10 |
| sdague | jeblair: so I was thinking about devstack gate log runs again. It seems like in the normal case we only care about the Tempest run, as that's the test results | 23:10 |
| fungi | flashgordon: looks like a good proxy for the fallout from reusing those workers, agreed | 23:10 |
| Mithrandir | fungi: heh | 23:10 |
| *** banix has quit IRC | 23:10 | |
| flashgordon | fungi: what to record that opinion as a +A | 23:11 |
| sdague | so I think I could be persuaded that if we are going to keep the setup for d-g in separate logs, we should also probably put the devstack setup in separate log as well, then the console log would load a ton faster, and be mostly the Tempest results and cleanup | 23:11 |
| clarkb | sdague: this would make me so happy :) | 23:12 |
| jeblair | sdague: wfm. i think it currently goes to stdout and a log file, and i think we do capture the devstack log file (but maybe don't logstash it) | 23:12 |
| fungi | the current jenkins lts release is new enough to support our tools yet old enough to be less of a sharp pain than the current jenkins releases... more like just a dull throbbing pain | 23:12 |
| clarkb | yes we do capture the file but don't logstash it | 23:12 |
| jeblair | sdague: does devestack maybe just have a flag we can set to not do stdout? | 23:12 |
| clarkb | because it is redundant | 23:12 |
| sdague | not that I know of | 23:13 |
| *** gokrokve has quit IRC | 23:13 | |
| sdague | I guess we could just bury stdout | 23:13 |
| flashgordon | thanks | 23:13 |
| sdague | flashgordon: while you are looking at er - https://review.openstack.org/#/c/73741/ - I rebased that | 23:13 |
| anteaya | I'm wondering if we should log all of fungi's characteriziations of pain and wounding this week | 23:14 |
| anteaya | y'know as a comparison to other weeks | 23:14 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Add timestamps to devstack-gate output https://review.openstack.org/75534 | 23:14 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1283283 https://review.openstack.org/75533 | 23:14 |
| jeblair | sdague, flashgordon: there's the updated version of my old 'add timestamps to d-g' patch; if it works, it should include the d-g sub-logs. | 23:15 |
| sdague | ok, will take a look | 23:16 |
| *** jergerber has quit IRC | 23:16 | |
| flashgordon | sdague: so for that patch Iam fine with it but want jeblair's opinion too | 23:16 |
| sdague | sure | 23:17 |
| jeblair | https://review.openstack.org/#/c/73741/ ? | 23:17 |
| clarkb | jeblair: I am doing a small experiment on logstash-worker16. I have disabled the crm114 workers there to get a baseline for the difference in throughput between with crm114 and without | 23:17 |
| jeblair | clarkb: ok. expect large (like 20x) | 23:17 |
| clarkb | jeblair: ok | 23:17 |
| jeblair | clarkb: crm114 usage is configurable in the config file as a switch we can flip in situations like this. | 23:18 |
| clarkb | jeblair: yup | 23:18 |
| clarkb | jeblair: I have just noticed that some crm processes keep going for minutes and I am wondering if that is a backup in crm or in logstash | 23:18 |
| *** mriedem has quit IRC | 23:18 | |
| clarkb | so building a sample on 16 | 23:18 |
| sdague | jeblair: yes 73741 | 23:19 |
| jeblair | sdague: so basically, my thought was why add two dependencies just to know that "Z" == "+00:00"? | 23:19 |
| *** brosenberg has left #openstack-infra | 23:21 | |
| jeblair | sdague: i don't care enough to make it an issue though. not by far. :) | 23:22 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add new oslo libs to ATC stats program https://review.openstack.org/75437 | 23:22 |
| sdague | because adding real time parsing isn't a bad thing, and it means that we're future proofed | 23:24 |
| *** sarob has quit IRC | 23:24 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Suppress bug 1269940 from the graphs page https://review.openstack.org/72731 | 23:24 |
| sdague | because I'm sure something else will break us in the future :) | 23:24 |
| jeblair | sdague: i dunno about future proof. a future where we log something in non-utc is not going to come to pass if i have anything to say about it. they're more likely to change to an unparsable tz suffix like "4/32". :) | 23:25 |
| *** persia has quit IRC | 23:26 | |
| jeblair | it's not bad, but in my mind it's just not worth going outside the stdlib | 23:26 |
| lifeless | UTC 4 eva. | 23:27 |
| lifeless | really. | 23:27 |
| lifeless | Its bad enough having translated log messages to support. [I *totally* support translated UIs. But logs? :(] | 23:27 |
| jeblair | lifeless: i almost read that as UTC+4 eva. :) | 23:27 |
| sdague | sure, I've just done too much date parsing logic over the years, would rather future proof it. battle scars | 23:27 |
| sdague | UTC+4... nice | 23:28 |
| sdague | moscow? | 23:28 |
| sdague | or is that +3 | 23:28 |
| *** gokrokve has joined #openstack-infra | 23:28 | |
| fungi | jeblair: when the real future arrives, i'll fight to switch from base-24/60 utc to metric time | 23:28 |
| jeblair | +4 | 23:28 |
| fungi | and a notion of time tracking not rooted in our antiquated notions of the rotation and orbits of stellar bodies | 23:30 |
| fungi | or special relativity | 23:30 |
| *** medberry has joined #openstack-infra | 23:31 | |
| sdague | on the TZ front, this was super cool find today - http://i0.wp.com/poisson.phc.unipi.it/~maggiolo/wp-content/uploads/2014/01/SolarTimeVsStandardTime.png | 23:31 |
| *** medberry is now known as med_ | 23:32 | |
| *** Ryan_Lane2 has quit IRC | 23:33 | |
| anteaya | fungi: what notion do you propose for time tracking? heartbeats? | 23:33 |
| fungi | i've always loved how mainland china is all one timezone, centered on beijing | 23:34 |
| jeblair | fungi: though it looks like it's more centered on shanghai | 23:34 |
| fungi | anteaya: good question. maybe normalize it to planck time and make everyone keep their own relative mapping for purposes of dilation | 23:34 |
| fungi | jeblair: ahh, maybe it was shanghai. anyway east end of the country | 23:35 |
| sdague | fungi: actually, read the source for tzdata some time. The source for that has some spectacular comments about the history of time and timezones | 23:36 |
| *** rustlebee is now known as russellb | 23:36 | |
| anteaya | planck time, that is a new one for me | 23:36 |
| anteaya | okay what event is 000 in planck time? | 23:37 |
| anteaya | this will be a fun one | 23:37 |
| fungi | i'm guessing that's the time our universe buds from its parent universe | 23:38 |
| anteaya | that leaves out most of the southern states | 23:38 |
| *** gokrokve has quit IRC | 23:40 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update java alternatives to java 7 https://review.openstack.org/75538 | 23:40 |
| anteaya | foamy spacetime: https://en.wikipedia.org/wiki/Planck_time | 23:40 |
| *** gokrokve has joined #openstack-infra | 23:40 | |
| *** zehicle_at_dell has joined #openstack-infra | 23:40 | |
| *** gokrokve_ has joined #openstack-infra | 23:41 | |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add python jobs for compass-core project https://review.openstack.org/75198 | 23:41 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 23:41 |
| openstackgerrit | Jerry Zhao proposed a change to openstack-infra/config: Add python jobs for compass-core project https://review.openstack.org/75198 | 23:42 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update java alternatives to java 7 https://review.openstack.org/75538 | 23:44 |
| clarkb | jeblair: fungi I self approved https://review.openstack.org/#/c/75278/ since you had both +2'd it | 23:44 |
| *** gokrokve has quit IRC | 23:44 | |
| *** Ryan_Lane1 has joined #openstack-infra | 23:45 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1395&rra_id=all is beginning to illustrate the differene between crm vs no crm | 23:45 |
| *** sdake_ has joined #openstack-infra | 23:45 | |
| *** sdake_ has quit IRC | 23:45 | |
| *** sdake_ has joined #openstack-infra | 23:45 | |
| clarkb | no crm is a very steady outbound data flow | 23:45 |
| anteaya | we have a nova pep8 failure in the gate: https://jenkins04.openstack.org/job/gate-nova-pep8/3077/console | 23:46 |
| clarkb | crm is a lot more laggy as it has to grab data, spin cpus to categorize, then ship to elasticsearch | 23:46 |
| anteaya | is this 1282876 again? | 23:46 |
| clarkb | certainly looks like a git failures | 23:46 |
| *** gokrokve_ has quit IRC | 23:46 | |
| clarkb | *failure | 23:46 |
| anteaya | yes | 23:46 |
| clarkb | oh | 23:46 |
| *** damnsmith is now known as dansmith | 23:46 | |
| *** gokrokve has joined #openstack-infra | 23:47 | |
| clarkb | you know what we don't have cached repos in jenkins workspaces on the single use non devstack images do we? | 23:47 |
| clarkb | do we need to git clone/rsync from the /opt cache as the first step in g-g-p? | 23:47 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Preserve HOLD state when job starts. https://review.openstack.org/75278 | 23:47 |
| anteaya | g-g-p? | 23:48 |
| *** dstanek is now known as dstanek_dinner | 23:48 | |
| clarkb | gerrit git prep | 23:49 |
| flashgordon | clarkb: message:"fatal: index-pack failed" AND filename:"console.html" | 23:49 |
| flashgordon | how is that for a fingerprint for 1282876 | 23:49 |
| anteaya | ah | 23:49 |
| *** sandywalsh has quit IRC | 23:49 | |
| clarkb | flashgordon: sure | 23:50 |
| anteaya | flashgordon: does index-pack failed show up in all cases of 1282876? | 23:50 |
| flashgordon | anteaya: actaully I don't think so | 23:50 |
| flashgordon | just most | 23:50 |
| flashgordon | message:"fatal: The remote end hung up unexpectedly" AND filename:"console.html" | 23:50 |
| anteaya | I don't see it in the log i posted in comment #4 | 23:50 |
| anteaya | fungi likes this line: fatal: Not a git repository (or any parent up to mount parent ) | 23:51 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: update java alternatives to java 7 https://review.openstack.org/75538 | 23:51 |
| flashgordon | anteaya: oh right | 23:51 |
| flashgordon | it doesn't cover all cases | 23:51 |
| flashgordon | we don't have a good way to fingerprint those | 23:51 |
| *** gokrokve has quit IRC | 23:51 | |
| flashgordon | the right files aren't in logstash for that | 23:51 |
| anteaya | though that doesn't always show up either | 23:52 |
| anteaya | okay | 23:52 |
| anteaya | index-pack failed showed up in this one | 23:52 |
| anteaya | so that is part of it, I agree | 23:52 |
| anteaya | what is the solve for the current nova pep8 failure? I had thought 1282876 was resolved | 23:53 |
| *** dcramer_ has quit IRC | 23:53 | |
| *** masayukig has joined #openstack-infra | 23:53 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1282876 https://review.openstack.org/75542 | 23:55 |
| flashgordon | clarkb: ^ | 23:55 |
| fungi | anteaya: i'm guessing maybe we have a bare-precise image which suffered the same fate as some of the devstack-precise and bare-centos6 images had | 23:55 |
| jeblair | if [ -d /opt/git/$ZUUL_PROJECT/.git ] | 23:55 |
| jeblair | then | 23:55 |
| jeblair | git clone file:///opt/git/$ZUUL_PROJECT . | 23:55 |
| jeblair | clarkb: ^ ggp | 23:56 |
| fungi | checking bare-precise in rax-iad now | 23:56 |
| clarkb | jeblair: does bash -x not show us the test? /me looks at log again | 23:56 |
| clarkb | oh there it is, I am blind | 23:56 |
| fungi | yep, no nova cloned on the bare-precise image in rax-iad | 23:57 |
| jeblair | fungi: before you build an image | 23:58 |
| *** masayukig has quit IRC | 23:58 | |
| jeblair | fungi: maybe let's go ahead and restart nodepool? | 23:58 |
| fungi | sounds good | 23:58 |
| fungi | i was about to suggest that anyway as nodepool's coming up on its daily scheduled image update | 23:58 |
| fungi | do we want a graceful shutdown? | 23:59 |
| jeblair | fungi: no, i haven't fixed that yet, it'd take hours | 23:59 |
| jeblair | fungi: i have restarted it | 23:59 |
| fungi | okay, wfm | 23:59 |
| * anteaya has company afk | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
