| *** obondarev_ has quit IRC | 00:00 | |
| *** obondarev_ has joined #openstack-infra | 00:02 | |
| *** jcooley_ has joined #openstack-infra | 00:03 | |
| *** obondarev_ has quit IRC | 00:07 | |
| *** jcooley_ has quit IRC | 00:08 | |
| openstackgerrit | A change was merged to openstack-infra/config: provide time in queue in zuul ui https://review.openstack.org/65993 | 00:10 |
|---|---|---|
| openstackgerrit | A change was merged to openstack-infra/config: make merge conflict changes black https://review.openstack.org/66056 | 00:15 |
| openstackgerrit | A change was merged to openstack-infra/config: clean up possible js incompatibilities https://review.openstack.org/66057 | 00:16 |
| *** CaptTofu has joined #openstack-infra | 00:18 | |
| *** oubiwann_ has joined #openstack-infra | 00:18 | |
| *** praneshp has quit IRC | 00:21 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard-webclient: Allow use of node from packages https://review.openstack.org/67604 | 00:22 |
| *** sdake has joined #openstack-infra | 00:22 | |
| *** sdake has quit IRC | 00:22 | |
| *** sdake has joined #openstack-infra | 00:22 | |
| *** reed__ has joined #openstack-infra | 00:25 | |
| *** sdake has quit IRC | 00:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add jenkins master name to the logstash records https://review.openstack.org/67508 | 00:26 |
| *** UtahDave has quit IRC | 00:26 | |
| *** hogepodge has quit IRC | 00:29 | |
| *** jcooley_ has joined #openstack-infra | 00:30 | |
| *** soleblaze has quit IRC | 00:31 | |
| *** soleblaze has joined #openstack-infra | 00:31 | |
| *** soleblaze has joined #openstack-infra | 00:31 | |
| *** soleblaze has quit IRC | 00:31 | |
| *** soleblaze has joined #openstack-infra | 00:35 | |
| *** CaptTofu has quit IRC | 00:40 | |
| *** CaptTofu has joined #openstack-infra | 00:40 | |
| *** soleblaze has left #openstack-infra | 00:42 | |
| clarkb | sdague: the black circle for merge fails isn't rendering for me | 00:44 |
| *** CaptTofu has quit IRC | 00:45 | |
| clarkb | I bet the black.js file isn't copied by puppet properly | 00:46 |
| clarkb | er black.png | 00:46 |
| *** wenlock has quit IRC | 00:47 | |
| clarkb | yup that is the problem | 00:47 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Copy black.png for the zuul status page. https://review.openstack.org/67605 | 00:49 |
| clarkb | I am going to self approve that after check tests | 00:49 |
| *** sarob has quit IRC | 00:50 | |
| pleia2 | looks good | 00:50 |
| *** sarob has joined #openstack-infra | 00:50 | |
| clarkb | pleia2: thanks | 00:51 |
| *** sarob has quit IRC | 00:55 | |
| *** gokrokve has quit IRC | 00:56 | |
| *** gokrokve has joined #openstack-infra | 00:56 | |
| *** smurugesan has quit IRC | 00:58 | |
| *** reed__ has quit IRC | 01:01 | |
| *** sarob has joined #openstack-infra | 01:01 | |
| *** krotscheck has quit IRC | 01:01 | |
| *** gokrokve has quit IRC | 01:01 | |
| clarkb | there was just a gate reset, did anyone have a list of changes that should be promoted? | 01:04 |
| *** markwash has quit IRC | 01:04 | |
| mordred | clarkb: I don't want to know why we're not using recursive there do I? | 01:05 |
| clarkb | recursive? | 01:05 |
| mordred | recursive copy of the zuul folder | 01:05 |
| mordred | instead of listing individual resources | 01:05 |
| clarkb | mordred: I have no idea why that decision is made. possible because different files go in differnt places? | 01:06 |
| *** sarob has quit IRC | 01:06 | |
| *** jhesketh_ has joined #openstack-infra | 01:07 | |
| mordred | yay. black circles work for me now | 01:09 |
| clarkb | I applied my change by hand to status.o.o using puppet agent --test | 01:09 |
| mordred | my god there are so many changes in the queue | 01:09 |
| clarkb | mordred: the check queue is way down now, hopefuly that will free up resources to the gate | 01:09 |
| clarkb | but the head of the gate just reset so meh | 01:10 |
| clarkb | also damnsmith is patch bombing us | 01:10 |
| clarkb | mordred: can you review my black.png change real quick and I can approve | 01:12 |
| mordred | +A | 01:13 |
| clarkb | sdague: btw zaro has a build for scp plugin that should fix our console log problems. I don't want to restart jenkinses until gate is happier but that will go in as soon as it won't be horrible to do so | 01:13 |
| mordred | ++ | 01:13 |
| mordred | I'm going to go drink some beer now | 01:13 |
| clarkb | mordred: me too | 01:13 |
| clarkb | https://github.com/jenkinsci/scp-plugin/pull/8 is the scp plugin pull requests, comments welcome | 01:15 |
| clarkb | I think it includes a fix for the job abort but marked as failure in scp plugin problem too | 01:15 |
| mikal | Wow, so much fail at the top of the check queue | 01:16 |
| *** thuc_ has quit IRC | 01:16 | |
| damnsmith | clarkb: sorry | 01:16 |
| damnsmith | clarkb: that's only about 2/3 of that queue, if that helps at all :/ | 01:16 |
| *** thuc has joined #openstack-infra | 01:16 | |
| clarkb | damnsmith: its fine. I only noticed beacuse pep8 made large sections of it red but I think you fixed those problems ;) | 01:19 |
| *** salv-orlando has quit IRC | 01:19 | |
| *** salv-orlando_ has joined #openstack-infra | 01:19 | |
| *** thuc has quit IRC | 01:21 | |
| *** CaptTofu has joined #openstack-infra | 01:23 | |
| clarkb | ok I found a list of changes, looks like all but one is approved | 01:25 |
| clarkb | I am constructing the list and we can promote during the next large gate reset | 01:26 |
| clarkb | nope nevermind there are jenkins -2's and all I see in response is a recheck no bug | 01:27 |
| *** dimsum has quit IRC | 01:28 | |
| *** smurugesan has joined #openstack-infra | 01:28 | |
| clarkb | I am just going to let normal gate action steer a course since I don't have energy to untangle what happened there | 01:29 |
| *** ok_delta has joined #openstack-infra | 01:33 | |
| *** thuc has joined #openstack-infra | 01:33 | |
| mikal | How does the global requirements sync work? | 01:41 |
| mikal | I assume there's a config file somewhere? | 01:41 |
| mikal | I have an urge to explain https://review.openstack.org/#/c/67048/1/test-requirements.txt | 01:41 |
| *** rakhmerov has quit IRC | 01:46 | |
| *** dimsum has joined #openstack-infra | 01:46 | |
| fungi | mikal: when a change is merged to openstack/requirements, a post job is triggered which checks out any projects listed in projects.txt in the requirements repo, sees what entries they have in their requirements.txt and test-requirements.txt, looks to see whether they match entries in the global-requirements.txt and if not, proposes changes to update them | 01:50 |
| *** thuc_ has joined #openstack-infra | 01:50 | |
| *** yamahata has quit IRC | 01:50 | |
| fungi | i can provide links to the job definition and script it runs too if you need | 01:50 |
| *** markmcclain has joined #openstack-infra | 01:52 | |
| *** thuc has quit IRC | 01:53 | |
| *** markmcclain1 has joined #openstack-infra | 01:54 | |
| *** jerryz has quit IRC | 01:54 | |
| *** thuc_ has quit IRC | 01:54 | |
| *** markmcclain has quit IRC | 01:56 | |
| mikal | fungi: so... is it possible that test-requirements in python novaclient wasn't updated because the update happened in global-requirements? | 01:57 |
| mikal | Where's the script so I can have a poke? | 01:57 |
| fungi | mikal: no, this came up a day or two ago. the job has a bug. i dug up and linked the log but i don't think anyone's looked into it | 01:58 |
| fungi | i'll dredge it back out of irc logs real quick | 01:58 |
| mikal | Ta | 01:59 |
| mikal | Is someone looking at the bug? | 01:59 |
| mikal | Should I just approve that manual test-requirements update? | 01:59 |
| fungi | yeah, no harm now that the requirements integration jobs work again now | 02:00 |
| fungi | er, don't mind me, the requirements integration job has nothing to do with it | 02:01 |
| fungi | but yes, safe to approve manual syncs of reqs | 02:01 |
| *** tma996 has joined #openstack-infra | 02:02 | |
| fungi | mikal: http://logs.openstack.org/df/df444f161d085478bc32ac529062afeabcfc3faa/post/propose-requirements-updates/dfeac29/console.html.gz | 02:02 |
| *** jhesketh_ has quit IRC | 02:05 | |
| openstackgerrit | A change was merged to openstack-infra/config: Copy black.png for the zuul status page. https://review.openstack.org/67605 | 02:05 |
| *** praneshp has joined #openstack-infra | 02:06 | |
| *** praneshp_ has joined #openstack-infra | 02:07 | |
| *** praneshp has quit IRC | 02:11 | |
| *** praneshp_ is now known as praneshp | 02:11 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Enable Neutron metering service plugin https://review.openstack.org/66142 | 02:17 |
| dimsum | clarkb, fungi do we build our own hudson jars/war? i threw a try/catch around that wait() in jenkins remoting - https://github.com/jenkinsci/remoting/pull/20 | 02:18 |
| *** tma996 has quit IRC | 02:20 | |
| *** arosen has quit IRC | 02:22 | |
| fungi | dimsum: we don't, though if we wanted to zaro might have ideas around that | 02:23 |
| *** CaptTofu has quit IRC | 02:24 | |
| *** gothicmindfood has quit IRC | 02:26 | |
| fungi | we normally install the debs provided by upstream | 02:27 |
| dimsum | i see. thx | 02:28 |
| *** melwitt has quit IRC | 02:29 | |
| *** yamahata has joined #openstack-infra | 02:32 | |
| *** markmcclain1 has quit IRC | 02:42 | |
| *** smurugesan has quit IRC | 02:46 | |
| *** jhesketh has joined #openstack-infra | 02:47 | |
| dimsum | rechecks page is awol - http://status.openstack.org/rechecks/ | 02:48 |
| *** coolsvap_away has joined #openstack-infra | 02:52 | |
| *** thuc has joined #openstack-infra | 02:52 | |
| *** thuc_ has joined #openstack-infra | 02:53 | |
| *** coolsvap has quit IRC | 02:54 | |
| *** thuc has quit IRC | 02:57 | |
| *** AaronGr is now known as AaronGr_Zzz | 02:57 | |
| *** derekh has quit IRC | 02:59 | |
| *** emagana has quit IRC | 03:01 | |
| fungi | dimsum: oh, i know why... fixing | 03:02 |
| *** fifieldt has joined #openstack-infra | 03:09 | |
| *** oubiwann_ has quit IRC | 03:11 | |
| *** oubiwann_ has joined #openstack-infra | 03:12 | |
| *** coolsvap_away has quit IRC | 03:15 | |
| fungi | dimsum: fixes | 03:17 |
| fungi | fixed i mean | 03:17 |
| *** SumitNaiksatam has quit IRC | 03:18 | |
| *** thuc_ has quit IRC | 03:26 | |
| *** thuc has joined #openstack-infra | 03:26 | |
| *** mrda has joined #openstack-infra | 03:27 | |
| *** sdake has joined #openstack-infra | 03:28 | |
| *** talluri has quit IRC | 03:31 | |
| *** thuc has quit IRC | 03:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:33 | |
| *** mrda has quit IRC | 03:54 | |
| *** carl_baldwin has joined #openstack-infra | 03:56 | |
| *** thuc has joined #openstack-infra | 03:57 | |
| *** thuc has quit IRC | 04:04 | |
| jog0 | I don't think any console logs are getting into elasticsearch | 04:06 |
| jog0 | http://logstash.openstack.org/#eyJzZWFyY2giOiIgYnVpbGRfcGF0Y2hzZXQ6XCIxMVwiIEFORCBidWlsZF9jaGFuZ2U6XCI0NDgzNlwiIEFORCBidWlsZF91dWlkOjllNzkzNGQqIEFORCBidWlsZF9uYW1lOlwiY2hlY2stdGVtcGVzdC1kc3ZtLW5ldXRyb25cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDMyMDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiJ0ZXJtcyIsImFuYWx5emVfZmllbGQiOiJmaWxlbmFtZSIsInN0YW1wIjoxMzkwMDE3OTk0Mzk5fQ== | 04:07 |
| jog0 | which means we don't have any valid elastic-recheck hits | 04:07 |
| portante | jog0: so if I see an elastic-recheck post on a review, it is not valid? Or are you saying that I won't see one when I should? | 04:08 |
| jog0 | portante: that its not valid right now. and that you shouldn't see one | 04:09 |
| jog0 | so both | 04:09 |
| jog0 | portante: you shouldn't see hits but there is a bug that ttps://review.openstack.org/#/c/67596/ should fix | 04:10 |
| jog0 | but there is no console logs in new logstash data | 04:11 |
| jog0 | fungi clarkb ^ | 04:11 |
| *** krotscheck has joined #openstack-infra | 04:18 | |
| portante | bummer. Thanks. | 04:19 |
| *** adam_g has quit IRC | 04:20 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Storyboard API Interface and basic project management https://review.openstack.org/67582 | 04:21 |
| *** adam_g has joined #openstack-infra | 04:22 | |
| *** adam_g has quit IRC | 04:22 | |
| *** adam_g has joined #openstack-infra | 04:22 | |
| *** ok_delta has quit IRC | 04:26 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 04:29 | |
| *** nati_ueno has joined #openstack-infra | 04:37 | |
| *** CaptTofu has joined #openstack-infra | 04:42 | |
| *** dstanek has joined #openstack-infra | 04:50 | |
| *** smurugesan has joined #openstack-infra | 04:53 | |
| *** krotscheck has quit IRC | 04:57 | |
| *** starmer has joined #openstack-infra | 04:58 | |
| zaro | dimsum: my idea is that we shouldn't. | 04:59 |
| *** harlowja is now known as harlowja_away | 05:00 | |
| *** gyee has quit IRC | 05:00 | |
| *** starmer has quit IRC | 05:01 | |
| *** starmer has joined #openstack-infra | 05:03 | |
| zaro | jog0: this is probably the same issue we've been having with the jenkins scp plugin. | 05:08 |
| *** CaptTofu has quit IRC | 05:15 | |
| *** dkliban has quit IRC | 05:15 | |
| clarkb | jog0: ? | 05:17 |
| clarkb | its a bug zaro has a fix requires jenkins restart will happen when sane to do so | 05:17 |
| sdague | clarkb: reading scrollback, the scp fix is fine to land during quiet time, like Sunday or something. We at least have identified that problem | 05:18 |
| clarkb | yup I wasthinking maybe early monday since it is US holiday | 05:19 |
| sdague | sure | 05:19 |
| clarkb | if queues clear I will do one host first to test more | 05:22 |
| clarkb | zaro tested locally and on -dev so should be good | 05:22 |
| *** emagana has joined #openstack-infra | 05:23 | |
| *** sarob has joined #openstack-infra | 05:25 | |
| sdague | cool | 05:27 |
| sdague | we're now displaying in -qa everything we loose a log | 05:28 |
| sdague | you can see that's happening quite a bit | 05:28 |
| clarkb | thats what that is | 05:29 |
| clarkb | ya it should be about 3/4 of all tests | 05:29 |
| clarkb | :( but fix soon | 05:29 |
| sdague | yep | 05:30 |
| *** vkozhukalov has joined #openstack-infra | 05:30 | |
| sdague | honestly next week I probably won't get to spend a lot of time on the data analysis bits anyway, so if it's fixed early, at least the week after I should have a more complete data set | 05:30 |
| sdague | so monday is good for me | 05:30 |
| clarkb | ok. I may try putting the jpi on 04 or 03 early tomorrow to let it burn in, then if all looks good get more aggressive | 05:32 |
| clarkb | at the latest I figure I can have all nodes on that version early monday | 05:32 |
| *** obondarev_ has joined #openstack-infra | 05:33 | |
| sdague | sure | 05:34 |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 05:34 | |
| clarkb | sdague: I like the new status page | 05:34 |
| sdague | I wonder if it's worth thinking about the early pep8 fail again. Given that so many folks are pushing bad code to the check queue, it would save us on dsvm nodes | 05:35 |
| clarkb | it is both informative and a bit scary | 05:35 |
| sdague | clarkb: cool, thanks | 05:35 |
| sdague | yep :) | 05:35 |
| clarkb | it shows that the check queue is currently flowing pretty well | 05:35 |
| clarkb | gate is still bogged down | 05:35 |
| clarkb | sdague: yeah, early pep8 may help | 05:36 |
| sdague | at least on check | 05:36 |
| sdague | just a thought | 05:36 |
| sdague | anyway, need to sleep. Have a good weekend. Will probably check in on sunday at some point. | 05:36 |
| clarkb | ok have a good weekend yourself | 05:37 |
| *** pcrews has quit IRC | 05:37 | |
| clarkb | I will have spotty connectivity as I will be traveling tio portland for the long weekend, but should have decent chunks of time to do scp plugin and so on | 05:37 |
| clarkb | https://jenkins02.openstack.org/job/gate-tempest-dsvm-large-ops/12168/console is an interesting fail | 05:43 |
| clarkb | fungi: I am going to get up early so we can assess the viability of a project rename but it isn't looking so great right now | 05:44 |
| clarkb | fungi: I think priority is to the openstack gate and not stackforge project rename. If renaming may be troublesome due to gate state we should postpone | 05:45 |
| *** _ruhe is now known as ruhe | 05:45 | |
| clarkb | I hate saying this but I think the error logs check at the end of d-g is causing a bit of thrash... | 05:46 |
| clarkb | sdague: dkranz: we should probably do a quick sanity check on it and see if we missed any white listable errors | 05:48 |
| *** carl_baldwin has quit IRC | 05:53 | |
| *** talluri has joined #openstack-infra | 05:56 | |
| *** thuc has joined #openstack-infra | 06:01 | |
| *** ruhe is now known as _ruhe | 06:01 | |
| *** talluri has quit IRC | 06:01 | |
| *** talluri has joined #openstack-infra | 06:02 | |
| *** thuc has quit IRC | 06:05 | |
| *** sarob has quit IRC | 06:12 | |
| *** sarob has joined #openstack-infra | 06:12 | |
| * mordred throws a fish at error logs | 06:16 | |
| *** sarob has quit IRC | 06:17 | |
| *** ryanpetrello has quit IRC | 06:25 | |
| *** obondarev_ has quit IRC | 06:32 | |
| *** oubiwann_ has quit IRC | 06:36 | |
| *** changbl has quit IRC | 06:36 | |
| * SpamapS watches in delight as the fish rebounds back at mordred | 06:38 | |
| *** starmer has quit IRC | 06:41 | |
| *** nati_ueno has quit IRC | 06:42 | |
| *** kraman has left #openstack-infra | 06:48 | |
| *** sarob has joined #openstack-infra | 07:13 | |
| *** smurugesan has quit IRC | 07:15 | |
| *** CaptTofu has joined #openstack-infra | 07:16 | |
| *** boris-42 has joined #openstack-infra | 07:16 | |
| *** sarob has quit IRC | 07:18 | |
| *** denis_makogon has joined #openstack-infra | 07:18 | |
| *** rwsu has quit IRC | 07:19 | |
| *** CaptTofu has quit IRC | 07:20 | |
| *** denis_makogon has quit IRC | 07:28 | |
| *** starmer has joined #openstack-infra | 07:40 | |
| anteaya | nice addition of the wait time for gate patches on the zuul page | 07:59 |
| clarkb | anteaya: I was going to promote changes but the head of the neutron chain failed check tests | 08:00 |
| anteaya | thanks | 08:00 |
| anteaya | good call | 08:00 |
| anteaya | I've slept my version of the night and now I am awake for the day | 08:00 |
| anteaya | woohoo | 08:00 |
| clarkb | but it is 3am | 08:01 |
| * anteaya checks out the reference neutron patches | 08:01 | |
| anteaya | yeah | 08:01 |
| anteaya | fun times | 08:01 |
| anteaya | been waking up at 1:30am | 08:01 |
| anteaya | so 3 is an improvement | 08:01 |
| clarkb | I have the opposite problem with my jet lag, I can't fall asleep until after 1am | 08:01 |
| anteaya | so lots of coverage in here then | 08:02 |
| anteaya | was this the patch that was failing check? https://review.openstack.org/#/c/67537/ | 08:02 |
| clarkb | no that one has no dependencies | 08:03 |
| anteaya | I don't see that it even had logs reported | 08:03 |
| clarkb | anteaya: it was the first one in the list on the etherpad | 08:03 |
| clarkb | anteaya: there are no logs because it is still in the gate | 08:03 |
| anteaya | markmcclain approved it from the airport he left before the conversation we had to ensure it passed check first | 08:03 |
| anteaya | right | 08:04 |
| anteaya | blast: https://review.openstack.org/#/c/61964/ got approved without ever passing the check queue | 08:05 |
| anteaya | rarareabhgadlkjsdgklsdgd | 08:05 |
| anteaya | I thought I had addressed the social issue behind this | 08:05 |
| * anteaya notes another chat with markmcclain is in order | 08:06 | |
| clarkb | well if you want me to get started on that change... the change owner approved it, and it was recheck no bugged so had no context as to whether or not the failure was a preexisting thing | 08:06 |
| clarkb | >_> | 08:06 |
| anteaya | yeah | 08:06 |
| anteaya | this is a cultural thing | 08:07 |
| clarkb | and, recheck no bug is not what the rechecker wanted I think, I believe the intention was to gate it not check it | 08:07 |
| anteaya | which I had mistakenly assumed I had addressed | 08:07 |
| anteaya | and obviously I had not | 08:07 |
| anteaya | I will ensure it is addressed this time | 08:07 |
| anteaya | I don't know where the idea came from that patches can be approved without passing check first but it seems to be a prevalent idea | 08:08 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add build job for Japanese Install Guide https://review.openstack.org/67481 | 08:08 |
| * anteaya relucantly retrives the nazi hat from the attic where she stored it just 8 short hours ago | 08:09 | |
| *** crank has quit IRC | 08:09 | |
| anteaya | clarkb: sorry I am mis-understanding what you are offering to do on that patch/with that patch | 08:11 |
| clarkb | anteaya: I was going to promote it and its children, but then saw it doesn't even pass tests so gave up | 08:12 |
| clarkb | but I had no context to determine if I could promote that patch because it was recheck no bugged | 08:12 |
| anteaya | right | 08:13 |
| anteaya | okay yes, let it be | 08:13 |
| anteaya | I will follow up with mark and address the fact check needs to pass as a prerequiste to +A | 08:14 |
| *** sarob has joined #openstack-infra | 08:14 | |
| anteaya | and then we will look at the patch and see what needs to happen to ensure it passes check | 08:14 |
| anteaya | I should comment on the patch so someone else doesn't try to just reverify it | 08:15 |
| *** crank has joined #openstack-infra | 08:16 | |
| *** sarob has quit IRC | 08:19 | |
| *** starmer has quit IRC | 08:22 | |
| anteaya | clarkb: I have commented: https://review.openstack.org/#/c/61964/ | 08:23 |
| *** elasticio has joined #openstack-infra | 08:32 | |
| *** bknudson has quit IRC | 08:39 | |
| *** bknudson has joined #openstack-infra | 08:40 | |
| *** yolanda has joined #openstack-infra | 08:42 | |
| *** odyssey4me has joined #openstack-infra | 08:42 | |
| *** rakhmerov has joined #openstack-infra | 08:48 | |
| *** yolanda has quit IRC | 08:50 | |
| *** yolanda has joined #openstack-infra | 08:51 | |
| *** odyssey4me has quit IRC | 08:55 | |
| *** odyssey4me has joined #openstack-infra | 09:03 | |
| *** emagana has quit IRC | 09:08 | |
| *** praneshp has quit IRC | 09:11 | |
| *** sarob has joined #openstack-infra | 09:13 | |
| *** CaptTofu has joined #openstack-infra | 09:16 | |
| *** sarob has quit IRC | 09:18 | |
| *** CaptTofu has quit IRC | 09:21 | |
| *** crank has quit IRC | 09:28 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 09:32 |
| *** crank has joined #openstack-infra | 09:35 | |
| lifeless | how do you tell nodepool that it's actually not building an image it thinks it's building ? | 09:50 |
| clarkb | lifeless: the image build died, but the client didn't pick up on that? | 09:54 |
| clarkb | not sure I understand the question, but it is also very late at night for me. I should sleep | 09:55 |
| lifeless | clarkb: nodepool starts a build | 09:55 |
| lifeless | nodepool and the cloud end up on nonspeaking terms | 09:56 |
| lifeless | nodepool is killed hard | 09:56 |
| lifeless | nodepool is started | 09:56 |
| lifeless | lifeless goes 'wtf why isn't it building images' ? | 09:56 |
| clarkb | oh I see, state left over in the DB | 09:56 |
| clarkb | nodepool image-delete $ID iirc | 09:56 |
| *** Ryan_Lane has quit IRC | 09:58 | |
| *** Ryan_Lane has joined #openstack-infra | 09:58 | |
| clarkb | might be nodepool image delete | 09:59 |
| clarkb | there is also a corresponding list command if you need to find the id | 09:59 |
| lifeless | clarkb: I might pick your brains when we're not both past bedtime about how to debug nodepool build failures as non-openstack-admin | 10:00 |
| *** saschpe has quit IRC | 10:00 | |
| *** saschpe_ has joined #openstack-infra | 10:00 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Permit using a known keypair when bootstrapping. https://review.openstack.org/67649 | 10:00 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Add some debugging around image checking. https://review.openstack.org/67650 | 10:00 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Ignore vim editor backup and swap files. https://review.openstack.org/67651 | 10:00 |
| *** dkranz has quit IRC | 10:02 | |
| *** loquacities has quit IRC | 10:02 | |
| *** uvirtbot has quit IRC | 10:03 | |
| *** jhesketh has quit IRC | 10:03 | |
| *** dkranz has joined #openstack-infra | 10:03 | |
| *** jhesketh has joined #openstack-infra | 10:04 | |
| *** loquacities has joined #openstack-infra | 10:04 | |
| *** boris-42 has quit IRC | 10:06 | |
| *** dtroyer has quit IRC | 10:08 | |
| *** dtroyer has joined #openstack-infra | 10:11 | |
| *** loquacities has quit IRC | 10:11 | |
| *** sdake has quit IRC | 10:11 | |
| *** nikhil___ has quit IRC | 10:11 | |
| *** nikhil__ has joined #openstack-infra | 10:12 | |
| *** sarob has joined #openstack-infra | 10:13 | |
| *** loquacities has joined #openstack-infra | 10:14 | |
| *** sdake has joined #openstack-infra | 10:14 | |
| *** jcooley_ has quit IRC | 10:14 | |
| anteaya | loquacities: heyo | 10:16 |
| StevenK | openstackrecheck is not a happy bot. | 10:17 |
| *** sarob has quit IRC | 10:18 | |
| anteaya | hi StevenK | 10:19 |
| anteaya | the possibility is high, care to share the url of logs or output? | 10:19 |
| StevenK | anteaya: Hai! I'm just going by its messages in -qa. | 10:20 |
| anteaya | okay | 10:23 |
| anteaya | the output seems to explain itself to me | 10:23 |
| anteaya | I am reading "console logs not available after <timestamp>" to mean that the logs will expire by that time | 10:24 |
| *** thomasem has joined #openstack-infra | 10:24 | |
| StevenK | It sounds to me like "I spent 13 seconds waiting for the logs to appear, and they have not." | 10:24 |
| anteaya | and in the last 3ish hours there have been 3 tempest failures | 10:25 |
| anteaya | I will look again | 10:25 |
| anteaya | ah okay, yes your interpretation makes more sense | 10:26 |
| anteaya | I agree with you | 10:26 |
| anteaya | I think that is less a function of openstackrecheck bot and more a funtion of logging | 10:26 |
| anteaya | there have been logging issues, which jog0 and sdague have been working on addressing, with clarkb's help | 10:27 |
| anteaya | obviously they are still in a state | 10:27 |
| anteaya | http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2014-01-17.log timestamp 2014-01-17T12:11:14 | 10:29 |
| anteaya | that appears to be when they started timing out | 10:29 |
| anteaya | <- afk doing morning packing-to-check-out-of-this-hotel-room type stuff | 10:33 |
| *** boris-42 has joined #openstack-infra | 10:33 | |
| *** mrda has joined #openstack-infra | 10:44 | |
| *** talluri has quit IRC | 10:49 | |
| *** sarob has joined #openstack-infra | 10:53 | |
| *** sarob has quit IRC | 10:57 | |
| *** salv-orlando_ has quit IRC | 10:58 | |
| *** derekh has joined #openstack-infra | 11:00 | |
| *** mrda has quit IRC | 11:02 | |
| *** sarob has joined #openstack-infra | 11:13 | |
| *** sarob has quit IRC | 11:18 | |
| *** CaptTofu has joined #openstack-infra | 11:18 | |
| *** CaptTofu has quit IRC | 11:22 | |
| anteaya | StevenK: the issue is the scp plugin the jenkins are using on jenkins 03 and 04 (the new ones) is not the save version as the jenkinses 00 01 and 02 | 11:35 |
| anteaya | logstash, which elastic recheck accesses for logs, is unable to get the logs due to the scp plugin version | 11:36 |
| anteaya | at least that is my best understanding of the situation | 11:36 |
| anteaya | s/save version/same version | 11:36 |
| loquacities | hey anteaya :) | 11:39 |
| loquacities | did you get home ok? | 11:39 |
| anteaya | not quite yet | 11:41 |
| anteaya | I'm in Canada and should be in my own house on Monday night | 11:41 |
| anteaya | had to do a thing in Montreal this week | 11:41 |
| loquacities | ah, i see | 11:41 |
| anteaya | you don't live in Perth do you? | 11:41 |
| loquacities | well, you're in the right country at least :) | 11:41 |
| loquacities | no, i live in brisbane | 11:41 |
| anteaya | it is a start | 11:41 |
| loquacities | i got home late on Monday night | 11:41 |
| anteaya | yes, you told me that | 11:41 |
| anteaya | great | 11:41 |
| anteaya | hot in Brisbane? | 11:41 |
| loquacities | relatively hot | 11:42 |
| loquacities | not as bad as the southern states right now though | 11:42 |
| anteaya | yeah, wow | 11:42 |
| loquacities | about 30 degrees here | 11:42 |
| anteaya | those poor tennis players | 11:42 |
| loquacities | i know! | 11:42 |
| anteaya | pffft, mild at 30C | 11:42 |
| loquacities | adelaide and victoria are either melting or on fire right now | 11:42 |
| anteaya | wow | 11:42 |
| anteaya | poor cyeoh | 11:42 |
| StevenK | anteaya: Yes, but that's 30C at almost 11pm. | 11:43 |
| anteaya | he is in adelaide | 11:43 |
| anteaya | ah, okay a bit warm for the evening | 11:43 |
| loquacities | actually, it's 24 here noe | 11:43 |
| anteaya | wooohooo | 11:43 |
| loquacities | and only 9:45pm | 11:43 |
| anteaya | nice | 11:43 |
| loquacities | it's quite pleasant outdoors, i only just came in | 11:43 |
| anteaya | bet you are the envy of the rest of the country | 11:44 |
| loquacities | but mrda_ is in adelaide, and mattoliverau is in melbourne | 11:44 |
| loquacities | yep, which is weird! | 11:44 |
| anteaya | ah | 11:44 |
| anteaya | ha ha ha | 11:44 |
| loquacities | it's usually me moaning about the heat ;) | 11:44 |
| anteaya | bake-y | 11:44 |
| *** derekh has quit IRC | 11:44 | |
| loquacities | indeed | 11:44 |
| anteaya | as andrew was saying | 11:44 |
| anteaya | I think it was andrew | 11:44 |
| anteaya | fly to Canada, no more moaning about the heat | 11:45 |
| anteaya | problem solved | 11:45 |
| StevenK | ... then we'll just moan about the plane flight and jet lag? :-P | 11:45 |
| loquacities | lol | 11:45 |
| loquacities | well, andrew lives in wellington! | 11:45 |
| loquacities | he got to fly home and stop complaining about the heat too ;) | 11:46 |
| loquacities | bring on LCA in auckland, i say! | 11:46 |
| StevenK | Yes, lifeless was saying that he left Australia to get away from the stupidily hot days, and then look what Perth did ... | 11:46 |
| loquacities | lol | 11:46 |
| anteaya | StevenK: wfm | 11:47 |
| anteaya | poor lifeless | 11:47 |
| anteaya | loquacities: fly to visit him | 11:47 |
| loquacities | hehe, i might have to do that ;) | 11:47 |
| anteaya | you need to do lots of prep for lca in auckland, right? | 11:47 |
| loquacities | exactly! | 11:47 |
| loquacities | :P | 11:48 |
| loquacities | anyway, i need to go make my bed so that i can fall into it | 11:48 |
| anteaya | loquacities need to do some scouting for lca 2015, mikal, you are good with that right? | 11:48 |
| anteaya | loquacities: happy sleep | 11:48 |
| loquacities | hehe! | 11:48 |
| anteaya | :D | 11:48 |
| loquacities | o/ | 11:48 |
| anteaya | \o | 11:48 |
| *** derekh has joined #openstack-infra | 11:50 | |
| *** obondarev_ has joined #openstack-infra | 12:10 | |
| *** sarob has joined #openstack-infra | 12:13 | |
| *** elasticio has quit IRC | 12:15 | |
| *** derekh has quit IRC | 12:16 | |
| *** sarob has quit IRC | 12:18 | |
| *** elasticio has joined #openstack-infra | 12:20 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add build job for Japanese Install Guide https://review.openstack.org/67481 | 12:21 |
| *** yolanda has quit IRC | 12:27 | |
| *** elasticio has quit IRC | 12:36 | |
| anteaya | I'm off, possibly for the rest of the weekend | 12:50 |
| *** sarob has joined #openstack-infra | 13:13 | |
| *** sarob has quit IRC | 13:18 | |
| *** CaptTofu has joined #openstack-infra | 13:18 | |
| *** rossella_s has joined #openstack-infra | 13:19 | |
| *** CaptTofu has quit IRC | 13:21 | |
| *** CaptTofu has joined #openstack-infra | 13:22 | |
| *** derekh has joined #openstack-infra | 13:26 | |
| *** salv-orlando has joined #openstack-infra | 13:29 | |
| *** obondarev_ has quit IRC | 13:32 | |
| *** derekh has quit IRC | 13:34 | |
| *** rakhmerov has quit IRC | 13:36 | |
| *** obondarev_ has joined #openstack-infra | 13:41 | |
| sdague | clarkb / fungi: I was looking at the runs this morning, and it looks like a few other errors have showed up in logs that are pretty racey | 13:42 |
| sdague | so my suggestion is to actually ninja patch out the enforcement at this point, because at this rate we won't be able to have an i2 | 13:43 |
| sdague | and then revisit after | 13:43 |
| sdague | for whenever folks get up | 13:43 |
| *** elasticio has joined #openstack-infra | 13:43 | |
| sdague | I'm going to be in a car shortly. I might be able to tether in about 2 hrs. So if anyone is able to address, you've got my ok to stop enforcing on errors in logs. | 13:44 |
| *** david-lyle_ has joined #openstack-infra | 13:53 | |
| *** ilyashakhat has quit IRC | 14:01 | |
| *** dstanek has quit IRC | 14:03 | |
| *** ilyashakhat has joined #openstack-infra | 14:03 | |
| *** elasticio has quit IRC | 14:04 | |
| *** sarob has joined #openstack-infra | 14:13 | |
| *** fifieldt has quit IRC | 14:13 | |
| *** obondarev_ has quit IRC | 14:16 | |
| *** DennyZhang has joined #openstack-infra | 14:16 | |
| *** sarob has quit IRC | 14:18 | |
| *** yamahata has quit IRC | 14:27 | |
| *** rossella_s has quit IRC | 14:27 | |
| *** yamahata has joined #openstack-infra | 14:28 | |
| *** DennyZha` has joined #openstack-infra | 14:29 | |
| *** dkliban has joined #openstack-infra | 14:31 | |
| *** DennyZhang has quit IRC | 14:33 | |
| portante | sdague: nice work on the zuul page, thanks | 14:36 |
| *** rakhmerov has joined #openstack-infra | 14:36 | |
| *** yamahata has quit IRC | 14:40 | |
| *** rakhmerov has quit IRC | 14:41 | |
| *** yamahata has joined #openstack-infra | 14:44 | |
| *** dkliban has quit IRC | 14:49 | |
| *** dkliban has joined #openstack-infra | 14:49 | |
| *** dizquierdo has joined #openstack-infra | 14:50 | |
| *** DennyZha` has quit IRC | 14:54 | |
| *** vkozhukalov has quit IRC | 15:01 | |
| *** rakhmerov has joined #openstack-infra | 15:02 | |
| *** rakhmerov has quit IRC | 15:06 | |
| *** flaper87|afk is now known as flaper87 | 15:11 | |
| *** sarob has joined #openstack-infra | 15:13 | |
| sdague | portante: no prob | 15:17 |
| *** sarob has quit IRC | 15:18 | |
| fungi | sdague: ahh, yep, there's another gate reset on stray nova errors in the large-ops job. i'll put together a quick patch | 15:26 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: disable error log enforcement https://review.openstack.org/67656 | 15:26 |
| sdague | fungi: actually, I think that's the page | 15:26 |
| sdague | patch | 15:26 |
| sdague | was just working on it | 15:27 |
| sdague | so if you want to sanity check that one, then ninja merge, that's probably helpful | 15:27 |
| sdague | fungi: it's not actually the large ops job only, this shows up in multiple jobs. I think when we made nova conductor have workers, they sometimes race on setting up the topic | 15:28 |
| sdague | we could white list that, however, I think right now we should give up on this one until the gate gets back to normal | 15:29 |
| fungi | sur | 15:33 |
| fungi | e | 15:33 |
| fungi | sdague: would it help logstash-eyness if we continued to provide the same output in the log but just replaced the exit with a message saying what it would have exited with were we enforcing it? | 15:36 |
| *** CaptTofu has quit IRC | 15:36 | |
| sdague | fungi: maybe, honestly, we can fix that later | 15:36 |
| sdague | this is about damage control | 15:36 |
| fungi | okay, fair enough | 15:36 |
| sdague | I'd rather just get the gate flowing again | 15:37 |
| *** CaptTofu has joined #openstack-infra | 15:37 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: disable error log enforcement https://review.openstack.org/67656 | 15:39 |
| fungi | [the openstacks must flow] | 15:39 |
| *** changbl has joined #openstack-infra | 15:41 | |
| *** DinaBelova_ is now known as DinaBelova | 15:42 | |
| *** CaptTofu has quit IRC | 15:42 | |
| *** vkozhukalov has joined #openstack-infra | 15:43 | |
| sdague | :) | 15:43 |
| sdague | fungi: thanks for being up and abount on a Sat | 15:43 |
| sdague | hopefully things will level off and you can run far away from computers | 15:43 |
| fungi | why ever would i want to do that? i'm just looking for sufficient spare time to finish jacking a computer into my skull | 15:44 |
| fungi | in unrelated news, the debian systems administrators just announced they've set this up for the debian developer community... https://wiki.debian.org/UnifiedCommunications/DebianDevelopers | 15:46 |
| fungi | some interesting ideas there similar to what we've been trying with asterisk | 15:47 |
| jog0 | did I read the backlog right, we expect at least some jenkins nodes to get there console logs into elasticsearch | 15:47 |
| fungi | and also they're open to federating their systems with other free software communities who want to do something similar | 15:47 |
| *** derekh has joined #openstack-infra | 15:47 | |
| jog0 | or are none working now | 15:47 |
| jog0 | I didn't see any overnight | 15:47 |
| fungi | jog0: to clarify, jobs run through jenkins.openstack.org and jenkins01 should be getting their console logs. jobs run through jenkins02/03/04 may or may not depending on a race condition in the log availability notification and upload process | 15:48 |
| *** jhesketh has quit IRC | 15:49 | |
| *** ianw has quit IRC | 15:49 | |
| *** dripton has quit IRC | 15:49 | |
| *** dragondm has quit IRC | 15:49 | |
| *** kpepple has quit IRC | 15:49 | |
| *** mordred has quit IRC | 15:49 | |
| *** Hefeweizen has quit IRC | 15:49 | |
| *** mtreinish has quit IRC | 15:49 | |
| *** emagana has joined #openstack-infra | 15:49 | |
| jog0 | fungi: thanks, I didn't see anything get added last night. but will keep running my test patch | 15:49 |
| *** nati_ueno has joined #openstack-infra | 15:50 | |
| jog0 | running https://review.openstack.org/#/c/67596/ on a VM to test it out | 15:50 |
| fungi | jog0: it's entirely possible something else has cropped up to make that race worse, and/or break it for jenkins01 in other ways | 15:50 |
| sdague | jog0: realistically, we know the fix for the logs, clarkb just wanted to see if we could let the queue move more before dealing with jenkins | 15:50 |
| sdague | I'm supportive of that | 15:50 |
| *** nati_ueno has quit IRC | 15:50 | |
| jog0 | sdague: I agree with that too | 15:51 |
| jog0 | just wanted to know if I should expect any console logs or not | 15:51 |
| *** nati_ueno has joined #openstack-infra | 15:51 | |
| jog0 | sounds like maybe | 15:51 |
| *** jhesketh has joined #openstack-infra | 15:52 | |
| *** ianw has joined #openstack-infra | 15:52 | |
| *** dripton has joined #openstack-infra | 15:52 | |
| *** dragondm has joined #openstack-infra | 15:52 | |
| *** kpepple has joined #openstack-infra | 15:52 | |
| *** mordred has joined #openstack-infra | 15:52 | |
| *** Hefeweizen has joined #openstack-infra | 15:52 | |
| *** mtreinish has joined #openstack-infra | 15:52 | |
| jog0 | 5 merges to openstack/openstack in last 6 hours :/ | 15:53 |
| fungi | however, i think we can do jenkins upgrades and plugin upgrades fairly hitlessly to the gate at this point. we will run at reduced slave capacity for certain jobs at times while we quiesce each master, but unless we actually break something it should be mostly transparent to everyone | 15:53 |
| fungi | jog0: at our current integration test runtime, i take that to mean we have a ~50% ejection rate | 15:54 |
| fungi | hopefully it was mostly those stray benign nova error messages | 15:54 |
| jog0 | fungi: there is no rush from my point of view to fix the console bug I can wait a day or two | 15:55 |
| fungi | jog0: well, it leaves us somewhat blind to the causes of failures in aggregate, so the sooner it's fixed the better a position we're in to take faster corrective actions for other issues impacting gating performance | 15:55 |
| *** jhesketh has quit IRC | 15:56 | |
| *** ianw has quit IRC | 15:56 | |
| *** dripton has quit IRC | 15:56 | |
| *** dragondm has quit IRC | 15:56 | |
| *** kpepple has quit IRC | 15:56 | |
| *** mordred has quit IRC | 15:56 | |
| *** Hefeweizen has quit IRC | 15:56 | |
| *** mtreinish has quit IRC | 15:56 | |
| jog0 | fungi: agreed | 15:56 |
| fungi | and with the load we're expecting next week, we'll need every advantage we can muster | 15:57 |
| *** jhesketh has joined #openstack-infra | 15:58 | |
| *** ianw has joined #openstack-infra | 15:58 | |
| *** dripton has joined #openstack-infra | 15:58 | |
| *** dragondm has joined #openstack-infra | 15:58 | |
| *** kpepple has joined #openstack-infra | 15:58 | |
| *** mordred has joined #openstack-infra | 15:58 | |
| *** Hefeweizen has joined #openstack-infra | 15:58 | |
| *** mtreinish has joined #openstack-infra | 15:58 | |
| *** thomasem has quit IRC | 16:00 | |
| *** dstanek has joined #openstack-infra | 16:01 | |
| *** rakhmerov has joined #openstack-infra | 16:03 | |
| jog0 | yeah next will be a little crazy, I have been so cought up I haven't even done my nova reviews in a week too | 16:04 |
| jog0 | anyway time to go AFK, good luck holding down the fort | 16:04 |
| *** rakhmerov has quit IRC | 16:07 | |
| *** dstanek has quit IRC | 16:09 | |
| fungi | clarkb: i've made a judgement call and canceled the rename (reply sent to -infra ml thread and the devs who requested it) | 16:10 |
| *** sarob has joined #openstack-infra | 16:13 | |
| clarkb | fungi wfm | 16:16 |
| clarkb | I think it is the right call. | 16:16 |
| fungi | hopefully things start moving faster now that the error log checking is disabled again | 16:17 |
| fungi | but basically we merged one patch an hour while i was asleep. not so great | 16:18 |
| *** DinaBelova is now known as DinaBelova_ | 16:20 | |
| *** blitzrage has joined #openstack-infra | 16:22 | |
| clarkb | jog0: there are a lot of console.html files in logstash | 16:22 |
| clarkb | approximately 1/4 of the total | 16:22 |
| *** carl_baldwin has joined #openstack-infra | 16:23 | |
| clarkb | I'm not sure what I did not see any overnight means in this context | 16:23 |
| *** gokrokve has joined #openstack-infra | 16:27 | |
| *** blitzrage has quit IRC | 16:28 | |
| clarkb | fungi: I am going to hold off on updating scp plugins for now as well for the same reasons we held off on the job rename | 16:33 |
| fungi | k | 16:34 |
| *** dcramer_ has joined #openstack-infra | 16:34 | |
| clarkb | if ever the queue gets shorter I will try to take that as an opportunity to restart some jenkinses | 16:34 |
| clarkb | fungi: have you had a chance to look at the code zaro pull requested on github? | 16:34 |
| fungi | clarkb: i have not yet. my eyes tend to glaze over when i start looking at java, but i'll have a peek at it later once i'm done picking up groceries | 16:36 |
| clarkb | fungi: no rush and I am fairly comfortable with it, was just checkign if you had comments | 16:36 |
| clarkb | zaro: I do have one qusetion will ask here because otherwise I will get confused by github. In the catch for InterruptedException, should we log that at least? | 16:36 |
| *** emagana has quit IRC | 16:37 | |
| *** nati_uen_ has joined #openstack-infra | 16:39 | |
| *** sarob has quit IRC | 16:41 | |
| *** nati_ueno has quit IRC | 16:42 | |
| *** derekh has quit IRC | 16:45 | |
| sdague | fungi: if your brain is feeling up to a real review - https://review.openstack.org/#/c/67591/ is something I'd like to figure out how to do right next week, so we could have the unclassified bug page by bug squash day | 16:50 |
| sdague | I am not convinced that is right | 16:50 |
| *** odyssey4me has quit IRC | 16:50 | |
| *** carl_baldwin has quit IRC | 16:54 | |
| *** DinaBelova_ is now known as DinaBelova | 16:54 | |
| *** krtaylor has quit IRC | 16:55 | |
| *** nati_uen_ has quit IRC | 16:56 | |
| clarkb | sdague: it seems sound, I gave it a soft -1 with some suggestions on making it even ebtter | 16:57 |
| clarkb | by 'soft' -1 I mean that if fungi went and approved itright now I would be totally fine with that | 16:57 |
| *** odyssey4me has joined #openstack-infra | 16:58 | |
| *** nati_ueno has joined #openstack-infra | 17:02 | |
| *** elasticio has joined #openstack-infra | 17:02 | |
| *** rakhmerov has joined #openstack-infra | 17:03 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 17:03 | |
| *** dcramer_ has quit IRC | 17:03 | |
| *** nati_uen_ has joined #openstack-infra | 17:06 | |
| *** nati_ueno has quit IRC | 17:07 | |
| *** rakhmerov has quit IRC | 17:08 | |
| *** nati_uen_ has quit IRC | 17:09 | |
| *** dizquierdo has quit IRC | 17:10 | |
| *** nati_ueno has joined #openstack-infra | 17:10 | |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| *** sarob has joined #openstack-infra | 17:13 | |
| *** nati_ueno has quit IRC | 17:13 | |
| *** gokrokve has quit IRC | 17:14 | |
| *** HenryG has joined #openstack-infra | 17:14 | |
| *** sarob has quit IRC | 17:18 | |
| HenryG | Many of the recent gate failures for tempest patches seem to be due to "Process leaked file descriptors." | 17:19 |
| *** starmer has joined #openstack-infra | 17:19 | |
| HenryG | This is new to me so I am trying to understand the problem. | 17:20 |
| *** DinaBelova is now known as DinaBelova_ | 17:25 | |
| clarkb | its not a problem | 17:26 |
| clarkb | that is a benign warning from jenkins and has nothing to do with failures | 17:26 |
| HenryG | clarkb: but jenkins gives a -1 | 17:33 |
| HenryG | clarkb: (sorry for using the term 'gate failure') | 17:34 |
| clarkb | HenryG: jenkins gave the -1 for a different reason | 17:34 |
| *** blitzrage has joined #openstack-infra | 17:35 | |
| clarkb | you need to read more of the test log to find the failure | 17:35 |
| HenryG | clarkb: I see, thanks | 17:35 |
| clarkb | the leaked file descriptors warning happens every test because the tests create files outside the jenkins workspace | 17:35 |
| *** CaptTofu has joined #openstack-infra | 17:37 | |
| *** dstanek has joined #openstack-infra | 17:37 | |
| HenryG | clarkb: got it, thanks. I guess I was misled by the 'FAILURE' message right next to it. | 17:37 |
| *** oubiwann_ has joined #openstack-infra | 17:40 | |
| *** CaptTofu has quit IRC | 17:42 | |
| *** salv-orlando_ has joined #openstack-infra | 17:45 | |
| *** salv-orlando has quit IRC | 17:45 | |
| *** salv-orlando_ is now known as salv-orlando | 17:45 | |
| *** dstanek has quit IRC | 17:53 | |
| *** marun has joined #openstack-infra | 18:01 | |
| *** HenryG has quit IRC | 18:03 | |
| *** rakhmerov has joined #openstack-infra | 18:04 | |
| *** marun has quit IRC | 18:06 | |
| *** rakhmerov has quit IRC | 18:09 | |
| *** DinaBelova_ is now known as DinaBelova | 18:10 | |
| *** sarob has joined #openstack-infra | 18:13 | |
| *** dizquierdo has quit IRC | 18:16 | |
| *** sarob has quit IRC | 18:18 | |
| *** UtahDave has joined #openstack-infra | 18:33 | |
| *** starmer has quit IRC | 18:34 | |
| *** rakhmerov has joined #openstack-infra | 18:36 | |
| notmyname | with the new timings on the zull status page, are they still reset with a manual reorder of the gate? | 18:37 |
| *** rakhmerov1 has joined #openstack-infra | 18:38 | |
| *** rakhmerov has quit IRC | 18:38 | |
| *** rakhmerov1 has quit IRC | 18:43 | |
| sdague | notmyname: no | 18:44 |
| notmyname | ah cool. glad that got fixed too, then :-) | 18:44 |
| fungi | notmyname: well, not any more, but they were set when i loaded all the changes into the gate after the zuul move to a bigger server and tmpfs-backed git repos though | 18:44 |
| sdague | I did a durable enqueue times patch to zuul | 18:44 |
| sdague | yeh, and after that restart we took the new code | 18:44 |
| fungi | there are still just under a dozen in the gate from when i reloaded them into it yesterday | 18:45 |
| notmyname | right. so from now on, the numbers will be accurate | 18:45 |
| fungi | yep | 18:45 |
| *** starmer has joined #openstack-infra | 18:45 | |
| *** rakhmerov has joined #openstack-infra | 18:46 | |
| sdague | fungi: https://jenkins02.openstack.org/job/gate-tempest-dsvm-large-ops/12276/console how did that fail ? Could it have queued previously to the ninja merge? | 18:47 |
| sdague | basically something errored out on the n-cond has errors log thing which is the current reset (keystone, about 8 jobs down) | 18:48 |
| fungi | sdague: if it was already being tested when we disabled the error filter, that wouldn't have taken effect on it | 18:49 |
| sdague | yeh, guess that just seemed long enough ago | 18:49 |
| fungi | so if the log starts prior to 15:39 utc | 18:49 |
| fungi | which it doesn't | 18:50 |
| fungi | log on that run starts at 18:34 | 18:50 |
| sdague | yeh... that's concerning | 18:50 |
| fungi | several hours later | 18:50 |
| sdague | yep | 18:50 |
| *** oubiwann_ has quit IRC | 18:51 | |
| fungi | i bet the check-logs is returning nonzero and the controlling script is under -e | 18:51 |
| fungi | so we need to || true on the end of the command or something | 18:52 |
| sdague | oh... boo | 18:52 |
| sdague | yep, you are correct in suspecting that | 18:52 |
| *** talluri has joined #openstack-infra | 18:52 | |
| sdague | so... all that explicit exit logic was for naught? | 18:53 |
| fungi | apparently it was for show | 18:54 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: disable errexit for the test run phase of the script https://review.openstack.org/67675 | 18:56 |
| sdague | so how about that | 18:56 |
| sdague | turn of errexit before we enter the tempest block | 18:56 |
| sdague | then we have control over it | 18:56 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: disable errexit for the test run phase of the script https://review.openstack.org/67675 | 19:00 |
| *** DinaBelova is now known as DinaBelova_ | 19:00 | |
| fungi | yep, that looks safe | 19:00 |
| fungi | it's inside the top-level conditional | 19:00 |
| sdague | yep | 19:00 |
| morganfainberg | whoever did the merge conflict stuff in zuul... | 19:00 |
| morganfainberg | seriously awesome guys | 19:00 |
| mordred | fungi, clarkb: re: java - krotscheck doesn't hate it (hacks on it for fun in his spare time) - we should point things that need code reviews his way too when they come up | 19:00 |
| fungi | and everything after it passes through to that exit $res | 19:00 |
| fungi | mordred: oh, awesome | 19:01 |
| sdague | fungi: yep, that's why I took it into that scope | 19:01 |
| fungi | sdague: crossing fingers | 19:01 |
| mordred | fungi: which is the pull request that needs looking? | 19:02 |
| morganfainberg | is the number under the "estimated time to completion" how long the item has been in queue? | 19:03 |
| sdague | morganfainberg: yes | 19:03 |
| morganfainberg | sdague, nice feature | 19:03 |
| morganfainberg | sdague, awesome changes to zuul. | 19:04 |
| sdague | morganfainberg: thanks | 19:04 |
| fungi | mordred: zaro linked it earlier. i'll hunt it down in scrollback | 19:06 |
| *** sarob has joined #openstack-infra | 19:07 | |
| mordred | fungi: I found it | 19:08 |
| fungi | mordred: https://github.com/jenkinsci/scp-plugin/pull/8 | 19:09 |
| fungi | oh, you found it | 19:09 |
| *** CaptTofu has joined #openstack-infra | 19:11 | |
| *** sarob has quit IRC | 19:12 | |
| *** sarob has joined #openstack-infra | 19:13 | |
| * dimsum catching up on the scrollback | 19:14 | |
| *** dimsum is now known as dims | 19:14 | |
| sdague | fungi: on the up side, a real neutron race reset the whole gate, so they should get the errexit fix | 19:15 |
| sdague | then we'll just be back to real resets | 19:15 |
| *** sarob has quit IRC | 19:18 | |
| *** morganfainberg is now known as morganfainberg|z | 19:21 | |
| *** morganfainberg|z is now known as morganfainberg | 19:21 | |
| fungi | awesome | 19:22 |
| clarkb | mordred: good to know | 19:24 |
| akscram | guys, please review https://review.openstack.org/#/c/66929/ | 19:25 |
| lifeless | clarkb: pleia2: hey so - can I get a nodepool primer? I have a yaml file pointed at the ci-overcloud | 19:25 |
| lifeless | but it's blowing up on | 19:26 |
| lifeless | File "/home/robertc/work/nodepool/nodepool/nodepool.py", line 511, in bootstrapServer | 19:26 |
| lifeless | ssh_kwargs['password'] = server['admin_pass'] | 19:26 |
| lifeless | nvm | 19:27 |
| lifeless | answering my own question | 19:27 |
| clarkb | akscram: its been a busy week for us, hopefully I/we can catch up on code review enxt week | 19:27 |
| clarkb | lifeless: ? | 19:27 |
| lifeless | bug in my use-an-explicit-key patch | 19:28 |
| *** thomasem has joined #openstack-infra | 19:30 | |
| lifeless | We don't do password injection in ci-overcloud | 19:30 |
| lifeless | so the admin_pass fallback when you don't have an ssh key won't work in nodepool | 19:31 |
| lifeless | I need to override that too... | 19:31 |
| clarkb | what else can you do? | 19:32 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Permit using a known keypair when bootstrapping. https://review.openstack.org/67649 | 19:33 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Ignore vim editor backup and swap files. https://review.openstack.org/67651 | 19:33 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Add some debugging around image checking. https://review.openstack.org/67650 | 19:33 |
| lifeless | ugh, E128, DIAF. | 19:35 |
| *** thomasem has quit IRC | 19:35 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Ignore vim editor backup and swap files. https://review.openstack.org/67651 | 19:36 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Add some debugging around image checking. https://review.openstack.org/67650 | 19:36 |
| *** morganfainberg is now known as morganfainberg|z | 19:38 | |
| lifeless | clarkb: as for what else you can do, ^ just use a regular keypair and turn off the smarts. | 19:50 |
| lifeless | hmm | 19:51 |
| lifeless | host.scp(os.path.join(self.scriptdir, fname), 'scripts/%s' % fname) | 19:51 |
| lifeless | File "/home/robertc/work/nodepool/nodepool/sshclient.py", line 57, in scp | 19:51 |
| lifeless | ftp.put(source, dest) | 19:51 |
| lifeless | File "/home/robertc/.virtualenvs/nodepool-nodepool-dev/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 613, in put | 19:51 |
| lifeless | fl = file(localpath, 'rb') | 19:51 |
| lifeless | IOError: [Errno 21] Is a directory: './.git' | 19:51 |
| *** herndon has joined #openstack-infra | 19:56 | |
| clarkb | lifeless: without telling nova about a particular keypair? | 19:56 |
| lifeless | clarkb: see https://review.openstack.org/67649 | 19:56 |
| *** CaptTofu has quit IRC | 19:57 | |
| clarkb | your comment that said password fallback when you dont have an ssh key confused me because we are talking about ssh keys | 19:58 |
| lifeless | clarkb: see bootstrapServer in nodepool, and the code path down into SSHClient | 19:58 |
| clarkb | right you want to specify a key other than the nodepool managed key(s) | 19:59 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Only attempt to copy files when bootstrapping. https://review.openstack.org/67678 | 20:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Document that fake.yaml isn't usable. https://review.openstack.org/67679 | 20:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/nodepool: Ignore vim editor backup and swap files. https://review.openstack.org/67651 | 20:01 |
| *** senk has joined #openstack-infra | 20:07 | |
| lifeless | clarkb: right, but there was code that said 'oh look, no key, you *must* want a password' | 20:07 |
| lifeless | clarkb: when no password exists | 20:07 |
| clarkb | ya I understand what you meant now. initially read it as "no ssh key no password there is a third way to login that we should support" | 20:09 |
| *** sarob has joined #openstack-infra | 20:13 | |
| *** sarob has quit IRC | 20:18 | |
| *** herndon has quit IRC | 20:19 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Update the geard server for tripleo-gate. https://review.openstack.org/67680 | 20:20 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Improve tripleo nodepool image build efficiency. https://review.openstack.org/67255 | 20:20 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Configure eth1 for DHCP in tripleo-gate instances https://review.openstack.org/67260 | 20:20 |
| *** jaypipes has joined #openstack-infra | 20:23 | |
| *** UtahDave has quit IRC | 20:30 | |
| *** sarob has joined #openstack-infra | 20:37 | |
| *** dcramer_ has joined #openstack-infra | 20:42 | |
| *** sarob has quit IRC | 20:42 | |
| *** senk has quit IRC | 20:53 | |
| *** nati_ueno has joined #openstack-infra | 20:58 | |
| *** nati_ueno has quit IRC | 20:58 | |
| *** nati_ueno has joined #openstack-infra | 20:58 | |
| *** gokrokve has joined #openstack-infra | 21:00 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 21:01 | |
| *** jhesketh has quit IRC | 21:07 | |
| *** sarob has joined #openstack-infra | 21:13 | |
| *** talluri has quit IRC | 21:16 | |
| *** julim has quit IRC | 21:16 | |
| *** rakhmerov has quit IRC | 21:18 | |
| *** sarob has quit IRC | 21:18 | |
| *** boris-42 has quit IRC | 21:19 | |
| *** sgrasley has quit IRC | 21:19 | |
| *** mdenny has quit IRC | 21:19 | |
| *** primeministerp has quit IRC | 21:19 | |
| *** mattoliverau has quit IRC | 21:19 | |
| *** jgriffith has quit IRC | 21:19 | |
| *** moted has quit IRC | 21:19 | |
| *** enikanorov_ has quit IRC | 21:19 | |
| *** luis_ has quit IRC | 21:19 | |
| *** niska has quit IRC | 21:19 | |
| *** _ruhe has quit IRC | 21:19 | |
| *** jhesketh has joined #openstack-infra | 21:19 | |
| *** gokrokve has quit IRC | 21:20 | |
| *** niska has joined #openstack-infra | 21:20 | |
| *** boris-42 has joined #openstack-infra | 21:20 | |
| *** sgrasley has joined #openstack-infra | 21:20 | |
| *** mdenny has joined #openstack-infra | 21:20 | |
| *** primeministerp has joined #openstack-infra | 21:20 | |
| *** mattoliverau has joined #openstack-infra | 21:20 | |
| *** jgriffith has joined #openstack-infra | 21:20 | |
| *** moted has joined #openstack-infra | 21:20 | |
| *** enikanorov_ has joined #openstack-infra | 21:20 | |
| *** luis_ has joined #openstack-infra | 21:20 | |
| *** 20WAAVJ2F has joined #openstack-infra | 21:20 | |
| *** _ruhe has joined #openstack-infra | 21:20 | |
| *** 20WAAVJ2F has quit IRC | 21:20 | |
| *** flaper87 has quit IRC | 21:20 | |
| *** flaper87 has joined #openstack-infra | 21:20 | |
| *** boris-42 has quit IRC | 21:21 | |
| *** boris-42 has joined #openstack-infra | 21:21 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Clean up jshint warnings/errors in status.js https://review.openstack.org/67681 | 21:22 |
| mordred | sdague: ^^ I've been playing with js tools and status.js this morning | 21:23 |
| *** nati_ueno has quit IRC | 21:31 | |
| *** nati_ueno has joined #openstack-infra | 21:33 | |
| *** derekh has joined #openstack-infra | 21:41 | |
| *** vkozhukalov has quit IRC | 21:43 | |
| *** thomasem has joined #openstack-infra | 21:44 | |
| *** talluri has joined #openstack-infra | 21:46 | |
| *** thomasem has quit IRC | 21:47 | |
| *** talluri has quit IRC | 21:48 | |
| *** talluri_ has joined #openstack-infra | 21:48 | |
| *** talluri_ has quit IRC | 21:53 | |
| *** nati_ueno has quit IRC | 21:59 | |
| *** nati_ueno has joined #openstack-infra | 22:00 | |
| *** ok_delta has joined #openstack-infra | 22:02 | |
| *** nati_uen_ has joined #openstack-infra | 22:04 | |
| *** nati_uen_ has quit IRC | 22:04 | |
| *** nati_uen_ has joined #openstack-infra | 22:05 | |
| *** nati_ueno has quit IRC | 22:05 | |
| *** flaper87 is now known as flaper87|afk | 22:07 | |
| *** mrda_ has quit IRC | 22:08 | |
| *** mrda_ has joined #openstack-infra | 22:09 | |
| *** sarob has joined #openstack-infra | 22:13 | |
| *** pballand has joined #openstack-infra | 22:16 | |
| *** nati_uen_ has quit IRC | 22:16 | |
| *** sarob has quit IRC | 22:18 | |
| sdague | mordred: cool | 22:24 |
| sdague | mordred: would be nice to put that into the check queue | 22:30 |
| sdague | clarkb: did something change about building nodes, seems like we're spending a ton more time there in the overall graph | 22:33 |
| fungi | i'm looking at it now | 22:34 |
| *** oubiwann_ has joined #openstack-infra | 22:36 | |
| fungi | yeah, a lot of nodes in a "building" state for up to 3.5 hours | 22:39 |
| clarkb | fungi: do they all belong to a single provider or az? | 22:39 |
| fungi | 190 of them are in hpcloud-az2 | 22:40 |
| fungi | pretty much all of the ones which have been stuck in a "building" state for an hour or more are in that az | 22:41 |
| fungi | the devstack-precise image in that az is several weeks old, and we can't seem to build a bare-precise image there at all, according to nodepool image-list | 22:44 |
| fungi | did we crash hpcloud az2? :/ | 22:44 |
| *** jaypipes has quit IRC | 22:45 | |
| fungi | so basically we have no usable nodes in az2, which is good in a way because we also have no good images there | 22:46 |
| clarkb | fungi: if you log into the account there should be a way to submit tickets or IM with people. I am trying to get set to drive to portland so not very useful rightn ow | 22:46 |
| fungi | understandable | 22:46 |
| clarkb | under the top right portion of the gui iirc | 22:46 |
| *** obondarev_ has joined #openstack-infra | 22:47 | |
| fungi | 2014-01-18 19:04:15,323 ERROR nodepool.ImageUpdater: Exception updating image bare-precise in hpcloud-az2: [...] Exception: Unable to find flavor with min ram: 8192 | 22:49 |
| fungi | checking to see what nova flavor-list gives for that az | 22:49 |
| *** talluri has joined #openstack-infra | 22:49 | |
| fungi | weird. standard.large claims 8192 for Memory_MB | 22:51 |
| *** talluri has quit IRC | 22:55 | |
| fungi | i wonder if maybe we screwed up the ram calculating in nodepool a few weeks back... digging through the commit log to see what's new | 22:56 |
| fungi | nope, nothing new for much longer than that | 22:57 |
| *** rakhmerov has joined #openstack-infra | 22:59 | |
| *** cody-somerville has quit IRC | 23:00 | |
| *** elasticio has quit IRC | 23:06 | |
| *** sarob has joined #openstack-infra | 23:13 | |
| *** rakhmerov has quit IRC | 23:15 | |
| *** sarob has quit IRC | 23:18 | |
| *** pballand has quit IRC | 23:28 | |
| *** cody-somerville has joined #openstack-infra | 23:35 | |
| *** gokrokve has joined #openstack-infra | 23:38 | |
| sdague | fungi: is it possible to just down that AZ and get from others? | 23:40 |
| clarkb | sdague: fungi: sort of, AZ2 is our biggest resource currently | 23:40 |
| sdague | clarkb: ok, our quota doesn't transfer? | 23:41 |
| clarkb | its quota is quite large | 23:41 |
| clarkb | no quota is per az in hpland | 23:41 |
| sdague | ok | 23:41 |
| *** obondarev_ has quit IRC | 23:42 | |
| fungi | yeah, i'm trying to manually launch a vm in it now | 23:42 |
| fungi | their webui shows our account has no vms at all in that az (well, two which are stuck in a deleting state for over a month, but none besides those) | 23:43 |
| sdague | cool. well I'm done for the evening. Sorry couldn't help more. | 23:43 |
| *** talluri has joined #openstack-infra | 23:50 | |
| *** obondarev_ has joined #openstack-infra | 23:50 | |
| *** talluri has quit IRC | 23:55 | |
| *** obondarev_ has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!