Saturday, 2014-03-01

fungiisviridov: yes, they're not synchronized or related in any way. i've added you as the first member for each of those groups00:00
fungiisviridov: https://review.openstack.org/#/admin/groups/279,members https://review.openstack.org/#/admin/groups/278,members https://review.openstack.org/#/admin/groups/280,members00:00
*** cadenzajon has quit IRC00:00
*** valentinbud has joined #openstack-infra00:01
isviridovfungi, awesome. Already adding ))00:01
*** dkliban has joined #openstack-infra00:01
fungiso, i simply re-ran manage-projects for that stackforge/ospurge without doing any cleanup, and got http://paste.openstack.org/show/7080600:05
*** jeckersb is now known as jeckersb_gone00:05
fungithat's very gnarly00:05
*** zns has joined #openstack-infra00:05
fungithen i deleted the repositories for it from /var/lib/jeepyb, /var/lib/git and /home/gerrit2/review_site/git and ran it again, which got me http://paste.openstack.org/show/7080700:06
fungiyet still more and different fun!00:06
*** ok_delta has joined #openstack-infra00:07
*** jcooley_ has quit IRC00:08
fungimordred: i think it's balking at 'git remote remove' in your new patch00:09
fungi"error: Unknown subcommand: remove"00:09
clarkbit needs to be 'remote remove'00:09
clarkboh it is00:09
clarkbhrm00:09
fungimordred: looks like it needs to be "rm" rather than "remove". i'll leave a comment on the review00:09
clarkbremove must be in a newer version of git, I have it00:10
anteayawhat is happening when Failed to add groups file for project is the output?00:11
fungiclarkb: git version 1.7.9.5 on review.o.o00:12
fungianteaya: it's leaving the repository half-configured and not adding the acl00:12
anteayafor reasons unknown?00:12
*** Ryan_Lane2 has quit IRC00:13
fungipresently unknown, yes. that's why i'm investigating00:13
openstackgerritClark Boylan proposed a change to openstack-infra/config: Fix logstash watchdog cron.  https://review.openstack.org/7730600:13
clarkbfungi: ^ I think I derped let me test that really quickly00:13
clarkbthats what I get for copy pasta00:13
anteayafungi: k00:13
* anteaya wonders if anyone else got the Aliens reference00:13
fungiin irc, no one can hear you scream movie quotes00:14
anteayaha ha ha00:15
clarkbI am bad at movie quotes00:15
anteayaI'm pretty good, if I have seen the movie00:15
anteayaI'm really bad at song titles00:15
davidlenwellI do some of my best communication with movie qoutes00:15
*** Ryan_Lane1 has joined #openstack-infra00:16
davidlenwellnow that I see refstack in stackforge I am going to go enjoy a birthday beer .. or 700:16
davidlenwellthanks guys00:17
fungiwow. i keep cleaning up and re-running manage projects and am encountering a beautiful menagerie of seemingly random failures00:17
clarkbfungi: that log you have for manage projects seems to be missing information00:17
clarkbdid we not run it at debug level?00:17
clarkbit should also include what the actual commands are I thought00:17
fungiclarkb: that's at info level and above00:17
fungii can rerun it at debug00:17
clarkbbut I think it is trying to git add groups which failed00:18
fungii'll do a new one at debug00:18
clarkband for the groups file to be written we need to be able to get the group uuids which if the code still works the way I thought it did would short circuit the git add00:18
fungithese were the last two i did (not debug yet)... http://paste.openstack.org/show/70809 http://paste.openstack.org/show/7081000:19
anteayadavidlenwell: happy birthday, enjoy your beer00:19
fungibut those are with remote rm instead of remote remove, so that bug seems solved at least00:19
*** valentinbud has quit IRC00:20
clarkbfungi: are you running some conglomeration of code for jeepyb? is it available anywhere for reading not on review.o.o?00:20
clarkbI want to reread the stuff around group creation00:20
davidlenwellthanks anteaya00:20
fungiclarkb: the only changes are applying 74118, 74130 and 77298 plus my comment in that last patch00:21
fungihere's one at debug level... http://paste.openstack.org/show/7081100:21
clarkbgah that cron is still not working00:22
*** VijayTripathi1 has quit IRC00:24
fungii'm going to go diving in the gerrit ssh and error logs to see if i can spot anything related to group creation00:24
anteayahmmmmm fatal: pathspec 'groups' did not match any files00:24
clarkbanteaya: that means the file didn't exist00:26
clarkbwhich mean jeepyb didn't write one out like it is supposed to00:26
anteayaah ha00:26
fungii find this... http://paste.openstack.org/show/70817/00:27
anteayacreate_groups_file seems to be the function in question: http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n20300:28
clarkbmost likely uuids is empty00:29
anteayaMissingObjectException ?00:29
clarkbwhich may mean the regex isn't matching00:29
clarkbno the object exists, its just an empty dict {}00:30
clarkbnotice that the write out to the groups file is wrapped in if uuids:00:30
fungii'm going to see if anything made it into the db00:30
*** alexpilotti has quit IRC00:31
*** jhesketh__ has joined #openstack-infra00:31
*** valentinbud has joined #openstack-infra00:31
*** gokrokve_ has quit IRC00:32
fungino entries for ospurge-core or ospurge-ptl in teh account_groups table00:32
anteayainteresting00:33
*** cody-somerville has joined #openstack-infra00:34
clarkbfungi: shouldn't that have logged at the error level?00:34
anteayamight it be an issue with gerritlib createGroups: http://git.openstack.org/cgit/openstack-infra/gerritlib/tree/gerritlib/gerrit.py#n18400:34
anteayaor is it manage-projects create_group_file?00:34
fungiclarkb: that last 70811 paste gets debug level and i redirected all stdout/stderr to the file i captured00:35
fungiclarkb: unless the logging config is not conveying to gerritlib for some reason00:35
clarkbfungi: not gerritlib, http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n21700:36
fungiclarkb: paramiko seems to pick it up, so i have a feeling maybe gerritlib's logging isn't quite right00:36
fungioh00:36
mordredfungi: from git remote --help:00:36
mordred       git remote rename <old> <new>00:36
mordred       git remote remove <name>00:36
mordred       git remote rename <old> <new>00:36
mordred       git remote remove <name>00:37
mordredgah00:37
mordreddouble paste00:37
fungiclarkb: unless the re.match() is coming up fail00:37
*** valentinbud has quit IRC00:37
fungiclarkb: crap00:38
fungiwe've been chasing a wild waterfoul00:38
fungifowl00:38
anteayafoul fowl00:39
anteayawhat do you have?00:39
fungifix incoming00:39
*** david-lyle has joined #openstack-infra00:41
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Correct filename for stackforge/ospurge Gerrit ACL  https://review.openstack.org/7730800:41
*** jcooley_ has joined #openstack-infra00:41
*** gokrokve has joined #openstack-infra00:41
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Only add upstream remotes if we track upstream  https://review.openstack.org/7729800:42
*** pmathews has quit IRC00:42
fungisometimes i think i should take up deep sea fishing00:43
clarkbha I know that feel00:44
anteayatempest patch 73789 just failed in the gate on a timeout error00:44
fungianteaya: what's the name of the slave it ran on?00:45
*** yamahata has quit IRC00:45
fungicurious if it's our old friend hpcloud-az2 once more00:45
anteayadevstack-precise-rax-iad-189949200:45
fungioh, not a network timeout, a job timeout00:45
funginm00:45
*** nati_ueno has quit IRC00:45
anteayaa job timeout, sorry00:45
*** jp_at_hp has joined #openstack-infra00:47
*** ok_delta has quit IRC00:47
mordredfungi: if you run again - you may also want to include https://review.openstack.org/#/c/75641/00:48
mordredfungi: or, just approve it :)00:48
fungimordred: yep, approved, though i still have to hand-patch it since puppet is stopped while i test this stuff00:49
mordred++00:49
fungiadded00:50
*** protux has quit IRC00:52
fungiso here's the new error of the moment... http://paste.openstack.org/show/70823/00:52
openstackgerritA change was merged to openstack-infra/config: Correct filename for stackforge/ospurge Gerrit ACL  https://review.openstack.org/7730800:52
clarkbI may have cried wolf about the logstash worker cron failing00:54
clarkbI think its working maybe00:54
clarkbwhen I run the command direclty it works00:54
clarkband when I hand patch for it to log what it does it seems to be run by cron ok00:55
*** gokrokve has quit IRC00:56
*** mriedem has joined #openstack-infra00:56
*** wenlock has quit IRC00:56
fungiran again, and SUCCESS! http://paste.openstack.org/show/7082900:56
anteayaso http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/manage_projects.py#n131 is failing00:57
anteayasometimes00:57
*** jp_at_hp has quit IRC00:57
*** wenlock has joined #openstack-infra00:57
*** Ryan_Lane1 has quit IRC00:57
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Support resetting back to a non-master branch  https://review.openstack.org/7731100:58
mordredclarkb: you'll enoy that one ^^00:58
mordredfungi: woot!00:58
mordredfungi: so, I'm guessing we feel good about landing all of the patches you had applied locally then00:58
fungimordred: so far, yes00:59
*** CaptTofu has joined #openstack-infra00:59
openstackgerritClark Boylan proposed a change to openstack-infra/config: Log logstash watchdog.  https://review.openstack.org/7731200:59
*** bhuvan__ has joined #openstack-infra00:59
*** bhuvan___ has joined #openstack-infra00:59
fungimordred: 77311 is new feature work, not addressing existing bugs, right?01:00
mordredfungi: that's right01:00
clarkbfungi: ^ so I am going to put that in place by hand here on 03 and 04 (my current test nodes) but I think we should approve that to get numbers and logs01:00
mordredfungi: not urgent01:00
mordredfungi: I just put it in because I was in there for the other patch01:00
openstackgerritA change was merged to openstack-infra/jeepyb: Don't split the initial branch list on all whitespace  https://review.openstack.org/7564101:00
fungiclarkb: agreed. silently papering over failures is bad form, of course01:00
clarkbbah, the cron is still wrong new patchset incoming01:00
openstackgerritClark Boylan proposed a change to openstack-infra/config: Log logstash watchdog.  https://review.openstack.org/7731201:01
mordredfungi: all failures should get paper01:02
*** hemna_ has quit IRC01:02
fungithat explains all the toilet paper in my yard01:02
*** rfolco_ has quit IRC01:02
mordred:)01:03
*** bhuvan has quit IRC01:03
*** CaptTofu has quit IRC01:03
*** Ryan_Lane1 has joined #openstack-infra01:04
*** bhuvan_ has quit IRC01:04
*** nati_ueno has joined #openstack-infra01:04
*** prad has quit IRC01:04
*** jcooley_ has quit IRC01:04
clarkbfungi: for that failure about refs/meta/config not fetching I wodner if that is the new race01:04
*** jcooley_ has joined #openstack-infra01:05
fungiclarkb: the pack error?01:05
clarkbfungi: eg gerrit create project returning before there is a git repo with refs/meta/config in it on disk01:05
clarkbfungi: ya01:05
fungiclarkb: i think that matches up with the java stacktrace i pasted earlier too01:05
clarkbso we create project then immediately try to fetch refs/meta/config which may not be synced to disk01:05
clarkbmight be useful to add a poll loop there01:05
anteayaso create_project should be isolated from fetch?01:06
clarkband fail after a few seconds01:06
clarkbanteaya: I think it is, its just we immediately go do the fetch01:06
clarkbit would be so nice if these api commands were synchronous01:06
fungiokay, so i'm moving ahead to the non-upstream new projects. brace yourselves for new and exciting failures01:06
fungi(or not, if we're lucky)01:06
anteayasorry I mean in a separate file, if we were looking to salt it01:06
clarkbanteaya: no I don't think it needs to be a separate file01:07
anteayaclarkb: k01:07
clarkbanteaya: we just need to be careful to wait if it isn't already available01:07
anteayayes01:07
anteayaI'm trying to figure out the best way to wait01:07
*** nati_ueno has quit IRC01:07
anteayaif create was a separate command, that returned something01:07
anteayasalt could wait for the something returned01:08
anteayathen fire a different file01:08
clarkbanteaya: it already is something that we wait for a return on01:08
anteayawithin the file?01:08
clarkbgerrit is just being silly and returning before everything is done (thats my theory at least)01:08
clarkbanteaya: ya01:08
anteayaoh01:08
clarkbwe use gerritlib to create a project and that returns prior to the project being created01:08
anteayathe returned thing is not a response that indicates the actual state01:09
anteayait is worthless returned thing01:09
clarkbrgith at least that is my theory01:09
*** gyee has quit IRC01:09
anteayayou are probably correct01:09
anteayaor gerritlib is silly01:09
anteayahttp://git.openstack.org/cgit/openstack-infra/gerritlib/tree/gerritlib/gerrit.py#n19401:10
anteayawe say return err01:10
anteayaI am not seeing any waiting01:10
*** openstackgerrit has quit IRC01:10
anteayashould there be waiting in gerritlib/createProject?01:11
*** openstackgerrit has joined #openstack-infra01:11
*** yamahata has joined #openstack-infra01:11
clarkbfungi I will have a patch for you really quick if you can wait a minute01:13
fungiclarkb: it's already running01:13
fungibut it may fail spectacularly anyway!01:13
fungiso you could just hold out hope for that ;)01:13
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Poll manage projects config fetches.  https://review.openstack.org/7731401:14
clarkbfungi: ^ that is a bit hacky and could use better logging and a better commit message but I think that should allow us to confirm what is happenin01:14
clarkband if that is indeed happening it may explain all of the trouble we have had01:14
clarkbfungi: mordred https://review.openstack.org/#/c/77312/ it would be great to have that in place before the weekend so that the restarts get logged01:15
clarkbit is running on 03 and 04 and seems happy01:15
openstackgerritRussell Bryant proposed a change to openstack-infra/config: Trim down gantt check/gate jobs  https://review.openstack.org/7164801:15
*** zehicle_at_dell has joined #openstack-infra01:15
anteayafor me, if gerritlib/createProject is the place that is supposed to return something useful when the project is created, I think the fix should go there01:17
SpamapSis there a quick way I can find jenkins jobs running tripleo-* tests?01:17
anteayathen if anything else uses createProject, it won't hit the same error01:17
fungiclarkb: lgtm. feel free to wait for mordred or just approve before you knock off01:17
clarkbanteaya: but as best as createProject can tell the project has been created01:17
clarkbfungi: I will approve because impatient >_>01:18
*** nati_ueno has joined #openstack-infra01:18
fungiclarkb: as for 77314 i'll patch it in on the next run if we need another01:18
clarkbok01:18
*** zehicle has quit IRC01:18
*** CaptTofu has joined #openstack-infra01:18
SpamapSlike, given https://review.openstack.org/#/c/76675/ ... how can I find the in-progress jenkins job to watch?01:19
anteayaSpamapS: I did `grep -R "tripleo" modules/openstack_project/files/zuul/layout.yaml` in config01:19
fungiSpamapS: can you refine what you mean? the names of the jobs or their contents?01:19
anteayayou get more than you need, but it is there01:19
SpamapSWe've been having hardware issues..01:19
SpamapSwe're hoping one is solved..01:19
mordredclarkb: looks good - any reason to not +A?01:19
SpamapSI see VMs spinning up.. but.. how to watch.. ??01:19
*** krotscheck has quit IRC01:20
openstackgerritClark Boylan proposed a change to openstack-infra/jeepyb: Poll manage projects config fetches.  https://review.openstack.org/7731401:20
clarkbthat is a bit cleaner01:20
clarkbmordred: which change? the watchdog logging one? I just approved it01:20
mordredclarkb: the jeepyb poll loop01:20
clarkbmordred: oh, only that it may not actually solve anything01:20
anteayaclarkb: but I thought you said the value createProjects returns is treated as an indication the project has been created01:20
clarkbanteaya: yes as far as gerrit is concerned01:20
anteayaclarkb: when in actual fact it hasn't01:21
fungiSpamapS: jenkins jobs to not spin up vms. nodepoold does that01:21
clarkbanteaya: gerrit is asserting the project is created there01:21
openstackgerritA change was merged to openstack-infra/config: Log logstash watchdog.  https://review.openstack.org/7731201:21
clarkbother code needs to be defensive01:21
anteayaclarkb: how is gerrit asserting the project has been created?01:21
anteayawhat I see is return err01:22
anteayawhich to my eyes isn't indicative of the existance of anything01:22
fungiSpamapS: but if you want to see the progress of jobs which run on them, start from http://status.openstack.org/zuul/ and look in the check-tripleo or experimental-tripleo pipeline sections for changes which have jobs in progress, then click on the name of one which has a progress bar next to it01:22
anteayaunless I don't understand01:22
clarkbanteaya: because gerritlib has said the project is created01:22
SpamapSfungi: well I'm not so much interested in the vms that are spinning up, but just seeing that jobs are in fact going.01:22
mordredanteaya: the gerrit api is returning when it's done01:22
fungiSpamapS: right now i see no signs any are running jobs01:22
mordredanteaya: but under the covers, gerrit is not actually done01:22
mordredanteaya: but the api thinks that it is01:23
SpamapSfungi: hm yeah, I think all of the vms I see are left over zombies01:23
anteayaclarkb: what I am saying is that gerritlib is wrong01:23
clarkbI have cleaned up after myself, puppet is running on 03 and 04 again. going to grab a beer now will touch base later01:23
mordredit's not gerritlib01:23
clarkbanteaya: but it isn't01:23
mordredgerrit is lying to gerritlib01:23
anteayaclarkb: what isn't what01:23
anteayagerrit is lying to gerritlib01:23
clarkbgerritlib isn't wrong. gerrit told it the project was created01:23
mordredbut gerritlib doesn't have any context to know that it's being lied to01:23
anteayawhat is gerrit giving to gerritlib?01:23
fungiSpamapS: i'm trying to delete some whichi have been left in various states for more than 3 hours now... looks like we may be timing out reaching your api endpoint though01:23
clarkbanteaya: a successful return code01:23
mordredso you can't do the check in gerritlib01:24
SpamapSfungi: endpoint should be up now01:24
*** VijayTripathi has joined #openstack-infra01:24
anteayaso gerrit is return success even though there is no success01:24
mordredanteaya: yes01:24
anteayaah okay01:24
fungiSpamapS: ooh, fun, i'm hitting an sqlalchemy lock wait timeout instead at the moment01:24
mordredanteaya: in manage-projects, we have a git repo, so we can do additional git operations that gerritlib does not have the context to perform - so we can loop and check there01:24
fungiSpamapS: so i can't confirm just yet01:24
anteayamordred: so we are assuming that if the git repo is success that gerrit is success01:25
anteayaand ignore gerrit's return code01:25
mordredanteaya: well, kinda01:25
anteayamordred: what part am I missing?01:25
SpamapSfungi: mmk.01:25
mordredanteaya: actually, no. you're right. we're checking the git repo for success01:26
mordredbecause that's the ultimate source of truth for us01:26
anteayaand extrapolating that if the git repo is up that gerrits is up01:26
anteayabecause we can't trust gerrit's return code01:26
fungimordred: so rerunning a full (not specific projects) manage-projects with your proposed fix for upstream tracking, it still looks like it's checking out all the projects. i'll get a new log pasted shortly, when it eventually finishes running01:27
*** lttrl has joined #openstack-infra01:27
mordredfungi: it will check out all of the projects - but it should skip the remote update step01:27
fungimordred: here's the log from this run... http://paste.openstack.org/show/7083801:30
mordredfungi: looking01:30
anteayafungi: so it was successful?01:30
fungianteaya: no idea yet... i'm checking01:31
torandu_hi. i have another q re jjb. should this be possible? http://paste.openstack.org/show/70839/ specifically, setting builders on the jobs. the docs seem to suggest that that would be possible, although it's not working for me01:31
anteayaso far I see warm in there01:32
mordredfungi: you know what ...01:32
anteayaand openstack-sdk-dotnet01:32
anteayaand openstack-cli-powershell01:33
fungiokay, those three projects all seem to have been created successfully01:33
fungitheir acls are added and i can clone them and list branches from all members of our git server farm01:33
anteayaheat-translator01:33
anteayaawesome01:33
*** bhuvan___ has quit IRC01:34
*** bhuvan__ has quit IRC01:34
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Only add upstream remotes if we track upstream  https://review.openstack.org/7729801:34
mordredfungi: updated01:34
openstackgerritMonty Taylor proposed a change to openstack-infra/jeepyb: Support resetting back to a non-master branch  https://review.openstack.org/7731101:34
mordredfungi: moved the remote update into the if track_upstream block01:34
mordredfungi: we don't really need to do a remote update otherwise01:35
anteayaos-cloud-config01:35
mordredfungi: eventually I'd been planning on making an attempt to read setup.cfg in master for description and homepage01:35
SpamapSheat-translator? wha?01:35
fungiSpamapS: it translates heat into something usable--haven't you heard? ;)01:35
anteayaSpamapS: https://review.openstack.org/7598801:36
SpamapSno but... i can haz? such happy. need now.01:36
SpamapSnice01:36
fungilooks like something ibm must be using internally01:36
fungithe readme is succinct... http://git.openstack.org/cgit/stackforge/heat-translator/tree/README.md01:37
fungioh wow. they committed .pyc files in their repo01:37
* fungi should know better than to peek inside a stackforge project01:38
anteayathe name should be heat-tanslator01:38
*** thuc has quit IRC01:39
fungiokay, since "new project friday" is over (yaaay! thanks everyone! flailing kermit arms!) i'm going to turn puppet agent back on for review.openstack.org01:40
anteayanice work fungi01:40
anteayamordred: do we have a way forward?01:40
*** thuc has joined #openstack-infra01:40
clarkbfungi ha01:40
anteayabesides what was addressed tonight?01:40
clarkbdat pyc01:40
* fungi snorfles01:40
torandu_pls disregard my prev question re jjb. i figured it out01:42
fungianteaya: i believe we're continuing "new project fridays" as a weekly exercise until we've worked out the remaining kinks, but yes we identified several actual and theoretical bugs and have patches either proposed or approved so i think we're in good shape01:42
fungiand next week, if there are any pending new projects (at least the several in the "needs work" section from this week's etherpad are likely candidates) we'll see what happens01:43
anteayathat is good01:43
anteayaI just have one week left before I disappear01:43
*** VijayTripathi has quit IRC01:43
anteayaand want to hold up my end of helping mordred to reach the goal of having this autmated by mid-March01:44
*** thuc has quit IRC01:44
anteayaI'm not as good yet as picking up the ball and running with it01:44
*** melwitt has quit IRC01:44
anteayaautomated01:44
fungii'm not good with sports analogies either so don't feel bad01:44
anteayaokay01:45
anteayawhen you and clark and jim are talking01:45
anteayaI can't tell what I should be doing to help01:45
anteayathough it might seem obvious to others01:45
anteayaother than answering the phone01:45
*** gokrokve has joined #openstack-infra01:46
* anteaya goes to bed01:48
*** SumitNaiksatam has quit IRC01:51
*** dcramer__ has joined #openstack-infra01:51
*** Ryan_Lane1 has quit IRC01:53
*** flashgordon is now known as jogo01:54
*** lnxnut has joined #openstack-infra02:10
*** Hunner has quit IRC02:12
*** Hunner has joined #openstack-infra02:18
*** sahumada has quit IRC02:18
*** CaptTofu has quit IRC02:29
*** CaptTofu has joined #openstack-infra02:29
*** SumitNaiksatam has joined #openstack-infra02:31
*** CaptTofu has quit IRC02:33
openstackgerritA change was merged to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks  https://review.openstack.org/7665802:39
*** pcm_ has joined #openstack-infra02:42
*** mriedem has quit IRC02:44
funginodepoold seems to be overlimit in rax-iad (probably the 5000 builds/day again), so it's looping rapidly trying to add nodes there and holding a perpetual lock on the db such that i can't delete the tripleo nodes which have been hung in a building state for over 4 hours02:46
fungii suspect we just need to wait it out02:46
* fungi is going to watch a movie02:47
lifelessfungi: we can delete tripleo nodes02:48
lifelessfungi: will nodepool retry if we delete them from nova ?02:49
lifelessSpamapS: oh, probably want to restart all the n-comutes, we have error-deleting state again02:49
*** thuc has joined #openstack-infra02:51
*** rfolco_ has joined #openstack-infra02:52
*** lnxnut has quit IRC02:52
*** lnxnut has joined #openstack-infra02:55
*** thuc has quit IRC02:55
*** dolphm is now known as dolphm_50303:00
*** rfolco_ has quit IRC03:01
*** jerryz has quit IRC03:01
*** pcrews has quit IRC03:09
*** annegentle has joined #openstack-infra03:10
*** reed has quit IRC03:10
*** talluri has joined #openstack-infra03:13
*** CaptTofu has joined #openstack-infra03:16
*** wenlock has quit IRC03:17
*** talluri has quit IRC03:21
*** pmathews has joined #openstack-infra03:28
*** weshay has quit IRC03:28
*** amcrn has quit IRC03:29
zaropabelanger: i could not repro https://code.google.com/p/gerrit/issues/detail?id=2296  I tested on review-dev.o.o, you can give it a try on there as well.03:30
*** talluri has joined #openstack-infra03:30
zaropabelanger: i'm using git ver 1.8.5.1 that i installed from brew repo.03:32
*** julim has quit IRC03:39
*** dolphm_503 is now known as dolphm03:39
*** CaptTofu has quit IRC03:42
*** dolphm is now known as dolphm_50303:52
*** talluri has quit IRC03:58
*** lttrl has quit IRC04:07
*** lttrl has joined #openstack-infra04:07
*** lnxnut has quit IRC04:18
*** pcm_ has quit IRC04:20
*** lnxnut has joined #openstack-infra04:22
*** harlowja is now known as harlowja_away04:31
*** dolphm_503 is now known as dolphm04:43
*** chandan_kumar has joined #openstack-infra04:48
*** wchrisj has quit IRC04:49
*** jcooley_ has quit IRC04:50
*** jcooley_ has joined #openstack-infra04:50
*** jcooley_ has quit IRC04:55
*** dolphm is now known as dolphm_50304:55
*** wchrisj__ has joined #openstack-infra05:16
*** CaptTofu has joined #openstack-infra05:43
*** VijayTripathi has joined #openstack-infra05:45
*** dolphm_503 is now known as dolphm05:46
*** dcramer__ has quit IRC05:47
*** CaptTofu has quit IRC05:48
*** pmathews has quit IRC05:50
*** dolphm is now known as dolphm_50305:56
*** dcramer__ has joined #openstack-infra06:00
*** chandan_kumar has quit IRC06:01
*** lttrl has quit IRC06:13
*** chandan_kumar has joined #openstack-infra06:13
*** vkozhukalov has joined #openstack-infra06:18
*** jcooley_ has joined #openstack-infra06:19
*** wchrisj__ has quit IRC06:22
*** VijayTripathi has quit IRC06:24
*** chandan_kumar has quit IRC06:28
*** jcooley_ has quit IRC06:34
lifelessfungi: nodepool recovering from rax?06:37
*** marun has quit IRC06:39
*** chandan_kumar has joined #openstack-infra06:40
*** marun has joined #openstack-infra06:40
*** dstanek has quit IRC06:46
*** dolphm_503 is now known as dolphm06:47
*** nati_ueno has quit IRC06:47
*** chandan_kumar has quit IRC06:48
*** nati_ueno has joined #openstack-infra06:48
*** jamespage_ has quit IRC06:50
*** jhesketh__ has quit IRC06:54
*** valentinbud has joined #openstack-infra06:55
*** dolphm is now known as dolphm_50306:56
*** chandan_kumar has joined #openstack-infra06:58
*** nati_ueno has quit IRC07:02
*** jcooley_ has joined #openstack-infra07:08
*** saper has quit IRC07:16
*** chandan_kumar has quit IRC07:22
*** valentinbud has quit IRC07:27
*** yolanda_ has joined #openstack-infra07:32
*** ArxCruz_ has quit IRC07:37
*** CaptTofu has joined #openstack-infra07:44
morganfainberglifeless, now i know you can't claim it's Monday today!07:47
morganfainberglifeless, >.>07:47
*** dolphm_503 is now known as dolphm07:47
*** CaptTofu has quit IRC07:49
lifelessmorganfainberg: Saturday :P07:56
morganfainberglifeless, it's ... almost saturday here07:56
*** dolphm is now known as dolphm_50307:57
*** cody-somerville has quit IRC07:57
*** ianw has quit IRC08:02
openstackgerritA change was merged to openstack-infra/jeepyb: Change manage-projects verbose logging to info  https://review.openstack.org/7411808:05
*** jcooley_ has quit IRC08:10
*** davidhadas has joined #openstack-infra08:17
*** davidhadas_ has quit IRC08:18
*** jdurgin has quit IRC08:18
*** NikitaKonovalov_ is now known as NikitaKonovalov08:27
*** jdurgin has joined #openstack-infra08:32
*** morganfainberg is now known as morganfainberg_Z08:33
*** gokrokve has quit IRC08:38
*** gokrokve_ has joined #openstack-infra08:41
*** gokrokv__ has joined #openstack-infra08:43
*** gokrokve_ has quit IRC08:45
*** dolphm_503 is now known as dolphm08:48
*** sarob_ has joined #openstack-infra08:50
*** sarob_ has quit IRC08:54
*** NikitaKonovalov is now known as NikitaKonovalov_08:55
*** dolphm is now known as dolphm_50308:58
*** Ryan_Lane has quit IRC09:00
*** jcooley_ has joined #openstack-infra09:02
*** vkozhukalov has quit IRC09:08
*** david_lyle_ has joined #openstack-infra09:10
*** david-lyle has quit IRC09:13
*** talluri has joined #openstack-infra09:14
*** david_lyle_ has quit IRC09:15
*** gokrokv__ has quit IRC09:17
*** jcooley_ has quit IRC09:32
*** Ryan_Lane has joined #openstack-infra09:32
*** sdake__ has quit IRC09:36
*** Ryan_Lane has quit IRC09:41
*** talluri has quit IRC09:41
*** talluri has joined #openstack-infra09:42
*** CaptTofu has joined #openstack-infra09:45
*** gokrokve has joined #openstack-infra09:48
*** dolphm_503 is now known as dolphm09:49
*** gokrokve_ has joined #openstack-infra09:49
*** CaptTofu has quit IRC09:50
*** talluri has quit IRC09:50
*** chandan_kumar has joined #openstack-infra09:50
*** gokrokve has quit IRC09:52
*** fifieldt has quit IRC09:56
*** jcooley_ has joined #openstack-infra09:58
*** dolphm is now known as dolphm_50309:59
*** chandan_kumar has quit IRC10:02
*** jhesketh_ has quit IRC10:03
*** jhesketh_ has joined #openstack-infra10:16
*** valentinbud has joined #openstack-infra10:16
*** talluri has joined #openstack-infra10:18
*** chandan_kumar has joined #openstack-infra10:24
*** jcooley_ has quit IRC10:30
*** alexpilotti has joined #openstack-infra10:40
openstackgerritLukasz Jernas proposed a change to openstack-infra/config: Add fonts-takao package on the Debian build slave  https://review.openstack.org/7228710:42
*** davidhadas has quit IRC10:45
*** dolphm_503 is now known as dolphm10:50
*** sarob_ has joined #openstack-infra10:51
*** sarob_ has quit IRC10:56
*** jcooley_ has joined #openstack-infra10:59
*** ildikov_ has joined #openstack-infra10:59
*** dolphm is now known as dolphm_50311:00
*** valentinbud has quit IRC11:04
*** NikitaKonovalov_ is now known as NikitaKonovalov11:25
*** NikitaKonovalov is now known as NikitaKonovalov_11:26
*** jcooley_ has quit IRC11:29
*** ildikov_ has quit IRC11:31
*** chandan_kumar has quit IRC11:36
*** NikitaKonovalov_ is now known as NikitaKonovalov11:36
*** talluri has quit IRC11:37
*** NikitaKonovalov is now known as NikitaKonovalov_11:37
*** CaptTofu has joined #openstack-infra11:43
*** dolphm_503 is now known as dolphm11:50
*** chandan_kumar has joined #openstack-infra11:53
*** alexpilotti has quit IRC11:54
*** jcooley_ has joined #openstack-infra11:55
*** dolphm is now known as dolphm_50312:00
*** ildikov_ has joined #openstack-infra12:03
*** HenryG_ has joined #openstack-infra12:06
*** HenryG has quit IRC12:09
*** lcostantino has joined #openstack-infra12:09
*** ryanpetrello has joined #openstack-infra12:15
*** CaptTofu has quit IRC12:17
*** lcostantino has quit IRC12:18
*** yamahata has quit IRC12:18
*** jhesketh_ has quit IRC12:21
*** e0ne has joined #openstack-infra12:24
*** jcooley_ has quit IRC12:26
*** dizquierdo has joined #openstack-infra12:27
*** ryanpetrello has quit IRC12:30
*** yamahata has joined #openstack-infra12:32
*** valentinbud has joined #openstack-infra12:34
*** ryanpetrello has joined #openstack-infra12:36
*** banix has joined #openstack-infra12:37
*** valentinbud has quit IRC12:37
*** chandan_kumar has quit IRC12:38
*** valentinbud has joined #openstack-infra12:38
*** dizquierdo has quit IRC12:40
*** CaptTofu has joined #openstack-infra12:40
banixNoticed the jobs at the gate for patches being merged takes several hours; Is this how it should be? The tests get run many many times? Just wondering.12:41
*** alexpilotti has joined #openstack-infra12:44
*** yamahata has quit IRC12:45
*** ryanpetrello has quit IRC12:46
*** valentinbud has quit IRC12:48
*** dolphm_503 is now known as dolphm12:51
*** sarob_ has joined #openstack-infra12:52
*** jcooley_ has joined #openstack-infra12:55
*** davidhadas has joined #openstack-infra12:56
*** sarob_ has quit IRC12:57
*** dolphm is now known as dolphm_50313:01
*** yamahata has joined #openstack-infra13:01
*** lcostantino has joined #openstack-infra13:02
*** weshay has joined #openstack-infra13:03
*** lcostantino has quit IRC13:08
*** yamahata has quit IRC13:08
*** yamahata has joined #openstack-infra13:13
*** lcostantino has joined #openstack-infra13:14
*** unicell has joined #openstack-infra13:20
*** lcostantino has quit IRC13:21
*** talluri has joined #openstack-infra13:24
*** jcooley_ has quit IRC13:26
*** alexpilotti has quit IRC13:27
*** yamahata has quit IRC13:28
*** Ryan_Lane has joined #openstack-infra13:28
*** Ryan_Lane has quit IRC13:33
*** dstanek has joined #openstack-infra13:33
*** dolphm_503 is now known as dolphm13:35
dhellmann_good morning13:41
*** dhellmann_ is now known as dhellmann13:41
*** jcooley_ has joined #openstack-infra13:44
*** dhellmann is now known as dhellmann_13:57
*** yamahata has joined #openstack-infra13:57
*** yamahata has quit IRC14:02
*** rfolco has joined #openstack-infra14:03
*** yamahata has joined #openstack-infra14:07
*** jcooley_ has quit IRC14:16
*** dolphm is now known as dolphm_50314:17
*** dstanek has quit IRC14:22
*** yamahata has quit IRC14:24
*** e0ne has quit IRC14:24
*** zehicle_at_dell has quit IRC14:36
*** dstanek has joined #openstack-infra14:36
*** dolphm_503 is now known as dolphm14:36
*** rfolco has quit IRC14:44
*** vkozhukalov has joined #openstack-infra14:47
*** ryanpetrello has joined #openstack-infra14:48
*** Adri2000 has quit IRC14:50
*** ryanpetrello has quit IRC14:51
*** jcooley_ has joined #openstack-infra14:52
*** ryanpetrello has joined #openstack-infra14:52
*** blamar has quit IRC14:52
*** sarob_ has joined #openstack-infra14:54
*** lnxnut has quit IRC14:58
*** sarob_ has quit IRC14:58
*** e0ne has joined #openstack-infra15:03
*** lnxnut has joined #openstack-infra15:04
*** dolphm is now known as dolphm_50315:05
*** davidhadas has quit IRC15:10
*** e0ne has quit IRC15:12
*** Adri2000 has joined #openstack-infra15:14
*** Adri2000 has quit IRC15:14
*** Adri2000 has joined #openstack-infra15:14
*** jcooley_ has quit IRC15:15
*** jcooley_ has joined #openstack-infra15:15
*** ryanpetrello has quit IRC15:16
*** e0ne has joined #openstack-infra15:22
*** jcooley_ has quit IRC15:24
*** dolphm_503 is now known as dolphm15:25
*** e0ne has quit IRC15:26
*** lnxnut has quit IRC15:27
*** dstanek has quit IRC15:28
*** unicell has quit IRC15:34
*** lnxnut has joined #openstack-infra15:35
*** alexpilotti has joined #openstack-infra15:37
*** dolphm is now known as dolphm_50315:37
*** lnxnut has quit IRC15:39
*** jcooley_ has joined #openstack-infra15:40
*** e0ne has joined #openstack-infra15:43
*** unicell has joined #openstack-infra15:43
*** talluri has quit IRC15:43
*** ArxCruz has joined #openstack-infra15:44
*** sdake__ has joined #openstack-infra15:44
*** sdake__ has quit IRC15:47
*** sdake_ has joined #openstack-infra15:48
*** thuc has joined #openstack-infra15:49
*** e0ne has quit IRC15:53
*** mestery_ has quit IRC15:54
*** mestery has joined #openstack-infra15:54
*** lttrl has joined #openstack-infra15:54
*** dolphm_503 is now known as dolphm15:56
*** blamar has joined #openstack-infra15:56
*** e0ne has joined #openstack-infra16:02
*** pcm_ has joined #openstack-infra16:03
*** sarob_ has joined #openstack-infra16:05
*** jcooley_ has quit IRC16:07
*** jcooley_ has joined #openstack-infra16:07
*** sarob_ has quit IRC16:09
*** thuc has quit IRC16:20
*** thuc has joined #openstack-infra16:20
*** davidhadas has joined #openstack-infra16:21
*** e0ne has quit IRC16:21
*** alexpilotti has quit IRC16:22
*** thuc has quit IRC16:24
*** jcooley_ has quit IRC16:25
*** pcrews has joined #openstack-infra16:27
*** dstanek has joined #openstack-infra16:28
*** thuc has joined #openstack-infra16:30
*** thuc has quit IRC16:34
*** pcrews has quit IRC16:34
*** thuc has joined #openstack-infra16:34
*** lnxnut has joined #openstack-infra16:35
SergeyLukjanovdhellmann_, hey, what's up?16:37
*** DinaBelova has joined #openstack-infra16:38
*** NikitaKo- has joined #openstack-infra16:39
*** ruhe- has joined #openstack-infra16:39
*** NikitaKo- is now known as NikitaKonovalov16:39
*** NikitaKonovalov_ has quit IRC16:39
*** SergeyLukjanov2 has joined #openstack-infra16:40
*** ruhe has quit IRC16:40
*** ruhe- is now known as ruhe16:40
*** lnxnut has quit IRC16:40
*** dolphm is now known as dolphm_50316:40
*** SergeyLukjanov has quit IRC16:41
*** SergeyLukjanov2 is now known as SergeyLukjanov16:41
*** DinaBelova_ has quit IRC16:41
*** pmathews has joined #openstack-infra16:49
*** valentinbud has joined #openstack-infra16:49
*** talluri has joined #openstack-infra16:50
*** thuc_ has joined #openstack-infra16:50
openstackgerritClark Boylan proposed a change to openstack-infra/config: Actually restart services in logstash watchdog  https://review.openstack.org/7735616:50
*** valentinbud has quit IRC16:51
*** valentinbud has joined #openstack-infra16:51
*** valentinbud has quit IRC16:52
*** thuc has quit IRC16:54
*** yolanda_ has quit IRC16:54
openstackgerritA change was merged to openstack-infra/config: Actually restart services in logstash watchdog  https://review.openstack.org/7735616:55
*** sarob_ has joined #openstack-infra16:55
mordredbanix: yes. it can take several hours, depending on the situation16:57
clarkbmordred: I self approved ^16:58
clarkbbecause I want reliable indexing before I weekend16:58
*** lnxnut has joined #openstack-infra17:00
*** sarob_ has quit IRC17:00
*** pmathews has quit IRC17:03
banixmordred: Thanks, do you know for neutron patches, how many times the set of tests are run for merges? Just curious.17:05
*** thuc has joined #openstack-infra17:07
clarkbthey run twice if the check results are older than 24 hours old. First the change goes through the check queue and if that passed it goes through the gate queue. If the check results are recent (less than 24 hours old) it is just once through the gate queue. However anytime the gate resets will cause tests to rerun there which is a variable number17:08
*** thuc_ has quit IRC17:11
*** thuc has quit IRC17:12
*** dcramer__ has quit IRC17:12
mordredclarkb: kk17:17
openstackgerritAlex Gaynor proposed a change to openstack-dev/pbr: Declare support for Python versions in setup.cfg  https://review.openstack.org/7586817:18
*** jcooley_ has joined #openstack-infra17:22
*** wchrisj has joined #openstack-infra17:22
*** jcooley_ has quit IRC17:24
*** thuc has joined #openstack-infra17:26
*** wchrisj has quit IRC17:26
sdagueclarkb: I salute your work to get logstash back up to speed. Will be really happy to have that up to date again.17:26
clarkbme too :)17:26
clarkbI think at this point the major issues are addressed or worked around17:26
clarkbthe change above restarted 4 indexers after it went in and we are digging through the backlog once again17:27
clarkbhopefully upstream can address that bug17:27
*** thuc_ has joined #openstack-infra17:27
*** thuc has quit IRC17:28
sdaguecoolio17:28
*** wchrisj has joined #openstack-infra17:28
*** wchrisj has quit IRC17:31
clarkbthen after feature freeze I think we want to grow our index filesystems to 2TB each. 1TB is startin gto feel small17:31
*** e0ne has joined #openstack-infra17:32
*** dolphm_503 is now known as dolphm17:35
*** e0ne has quit IRC17:37
*** dhellmann_ is now known as dhellmann17:41
dhellmannSergeyLukjanov: just saying hi :-)17:42
SergeyLukjanov:)17:42
*** gokrokve_ has quit IRC17:43
*** gokrokve has joined #openstack-infra17:43
*** ryanpetrello has joined #openstack-infra17:47
*** thedodd has joined #openstack-infra17:49
*** talluri has quit IRC17:51
*** lnxnut has quit IRC17:52
*** thuc_ has quit IRC17:52
*** thuc has joined #openstack-infra17:53
*** lnxnut has joined #openstack-infra17:53
*** ociuhandu has quit IRC17:57
*** thuc has quit IRC17:57
*** andre__ has quit IRC17:59
*** ryanpetrello has quit IRC18:00
clarkbfungi: the shard delete cron isn't working anymore. I think that is because cron isn't installed on the pvhvm images by default. I have deleted a couple shards by hand today to give us breathing room, but we shuold fix that monday18:00
*** CaptTofu has quit IRC18:06
*** CaptTofu has joined #openstack-infra18:08
SpamapSso are people still unable to reach the tripleo ci cloud?18:11
SpamapSI don't see nodepooled vms running so I assume yes18:11
*** VijayTripathi has joined #openstack-infra18:16
*** ArxCruz has quit IRC18:19
*** nicedice_ has joined #openstack-infra18:20
*** lnxnut has quit IRC18:20
*** lnxnut has joined #openstack-infra18:21
*** lnxnut has quit IRC18:21
*** lnxnut has joined #openstack-infra18:22
*** nicedice has quit IRC18:22
*** lnxnut has quit IRC18:26
*** ryanpetrello has joined #openstack-infra18:33
*** ryanpetrello has quit IRC18:35
*** e0ne has joined #openstack-infra18:38
*** wchrisj has joined #openstack-infra18:39
*** lttrl has quit IRC18:42
*** dcramer__ has joined #openstack-infra18:43
*** wchrisj has quit IRC18:48
*** andre__ has joined #openstack-infra18:50
*** wchrisj has joined #openstack-infra18:56
*** sarob_ has joined #openstack-infra18:57
*** dolphm is now known as dolphm_50319:00
*** sarob_ has quit IRC19:01
*** wchrisj has quit IRC19:06
*** Ryan_Lane has joined #openstack-infra19:07
*** Ryan_Lane has quit IRC19:14
*** dolphm_503 is now known as dolphm19:20
banixsdague: Could you please revisit: https://review.openstack.org/#/c/71359/ You had given a -2 pending the merge of the plugin. the plugin Just got merged. The link to it is in the commit message.19:20
*** lnxnut has joined #openstack-infra19:23
*** milki has quit IRC19:24
*** Ryan_Lane has joined #openstack-infra19:27
*** lnxnut has quit IRC19:27
*** dolphm is now known as dolphm_50319:29
*** lnxnut has joined #openstack-infra19:35
*** therve` is now known as therve19:37
*** lnxnut has quit IRC19:40
*** banix has quit IRC19:43
*** banix has joined #openstack-infra19:58
*** milki has joined #openstack-infra20:00
*** marun has quit IRC20:03
*** marun has joined #openstack-infra20:05
*** banix has quit IRC20:14
*** dolphm_503 is now known as dolphm20:20
*** yolanda_ has joined #openstack-infra20:26
*** marun has quit IRC20:34
*** lnxnut has joined #openstack-infra20:35
*** boris-42 has quit IRC20:39
*** rwsu has quit IRC20:40
*** lnxnut has quit IRC20:40
*** torandu_ has quit IRC20:43
*** yolanda_ has quit IRC20:48
*** dolphm is now known as dolphm_50320:58
*** sarob_ has joined #openstack-infra20:58
*** sarob_ has quit IRC21:03
*** thedodd has quit IRC21:05
*** boris-42 has joined #openstack-infra21:11
*** ildikov_ has quit IRC21:18
*** markmcclain has joined #openstack-infra21:20
*** pmathews has joined #openstack-infra21:24
*** wchrisj has joined #openstack-infra21:34
*** lnxnut has joined #openstack-infra21:35
*** lnxnut has quit IRC21:40
*** pmathews has quit IRC21:40
*** Sukhdev has joined #openstack-infra21:40
*** cody-somerville has joined #openstack-infra21:48
*** cody-somerville has quit IRC21:48
*** cody-somerville has joined #openstack-infra21:48
*** dolphm_503 is now known as dolphm21:49
*** thuc has joined #openstack-infra21:53
clarkbI have now written and compiled C on an apple lisa running xenix. /weekend21:53
*** wchrisj has quit IRC21:54
*** pmathews has joined #openstack-infra21:54
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: restore bug 1253482 - keystone ephemeral port conflict  https://review.openstack.org/7736721:55
*** CaptTofu has quit IRC21:56
*** thuc has quit IRC21:57
*** dolphm is now known as dolphm_50321:58
sdagueclarkb: nice22:00
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add bug for bad tempest test with ec2 az  https://review.openstack.org/7736822:08
*** dolphm_503 is now known as dolphm22:14
openstackgerritA change was merged to openstack-infra/elastic-recheck: restore bug 1253482 - keystone ephemeral port conflict  https://review.openstack.org/7736722:16
openstackgerritA change was merged to openstack-infra/elastic-recheck: add bug for bad tempest test with ec2 az  https://review.openstack.org/7736822:16
*** dolphm is now known as dolphm_50322:17
mordredclarkb: pvhvm images don't have cron by default?22:20
clarkbcurl22:21
clarkbdid I tyop? i meant curl22:22
*** e0ne has quit IRC22:24
*** e0ne has joined #openstack-infra22:25
*** dolphm_503 is now known as dolphm22:27
sdagueclarkb: you seen this fail in the gate of late - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiV0FSTklORzogVGhlIGZvbGxvd2luZyBwYWNrYWdlcyBjYW5ub3QgYmUgYXV0aGVudGljYXRlZFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzkzNzExNjQxMzMzfQ== ?22:28
sdaguelooks like tons of mirror issues actually22:29
sdaguethere are lots of ways this seems to be failing, I wonder if the mirrors are updating oddly. I've found a couple other bad package issues in the unclassified pile22:29
sdaguetons might be over stating22:30
sdaguebut more package fails than I expect22:30
fungicron isn't installed on pvhvm images by default? and i was surprised to discover curl and lvm2 missing22:33
clarkbno it is22:34
clarkbI meant to say curl22:34
fungiclarkb: apple lisa? i take it you visited the museum22:35
clarkbwhich is used by the cronjob22:35
clarkbI did and am now getting ramen22:35
fungiand yeah, i already discovered curl missing, which is why the bootstrap scripts now try wget when curl fails22:35
*** lnxnut has joined #openstack-infra22:35
clarkbwe should put curl in our base manifrst22:35
clarkband install it everywhere22:35
clarkbfiguring out how to change modes in vi on the lisa keyboard was hard22:36
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: add query for un-authenticated packages  https://review.openstack.org/7737122:36
fungiagreed. in the bootstrap scripts i did a fallback because that's pre-puppet, but yes definitely22:37
sdagueclarkb: I excitedly see the logstash queue is 0. :)22:38
fungithe logstash job queue graph looks... abrupt... though22:38
*** VijayTripathi has quit IRC22:39
clarkbsdague: ya and hopefully it stays down there22:39
clarkbit is much happier with the watchdog working22:39
fungidid it clear out that quickly for natural (no incoming volume) reasons, or did it get dumped?22:39
*** lnxnut has quit IRC22:40
clarkbnatural22:41
anteayaso yesterday at 13:25 utc neutron ryu was emailed that their system was voting -1 due to their ci failing: http://lists.openstack.org/pipermail/openstack-infra/2014-February/000925.html22:45
*** cody-somerville has quit IRC22:45
*** ociuhandu has joined #openstack-infra22:45
anteayathey replied today at 10:40 utc to say they were working on it: http://lists.openstack.org/pipermail/openstack-infra/2014-March/000928.html22:46
anteayabut they haven't said it is fixed and their system is still voting: http://180.37.183.32/jenkins/job/ryuplugin-tempest/9/console22:46
anteayahttps://review.openstack.org/#/dashboard/968122:46
anteayacan I get some input?22:46
anteayaseeing devstack failed. in their logs indicates their is still a problem on their end22:47
anteayato my eyes22:47
openstackgerritA change was merged to openstack-infra/elastic-recheck: add query for un-authenticated packages  https://review.openstack.org/7737122:53
*** alexpilotti has joined #openstack-infra22:55
openstackgerritMonty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls  https://review.openstack.org/7737222:55
mordredclarkb: ^^ that's inspired by a feature in the bot in the solum channel22:56
*** sarob_ has joined #openstack-infra22:59
openstackgerritMonty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls  https://review.openstack.org/7737223:01
anteayamordred: is that just for meetbot?23:04
*** sarob_ has quit IRC23:04
mordredanteaya: yeah. although since meetbot watches all of the channels it's in, it should work for any channel meetbot inhabits23:07
*** vkozhukalov has quit IRC23:08
openstackgerritSean Dague proposed a change to openstack-infra/elastic-recheck: remove the stevedore query, it's almost all success  https://review.openstack.org/7737323:08
anteayareally? is meetbot in here?23:08
mordredopenstack: help23:09
sdaguemordred: any concerns with the external service ?23:09
mordredsdague: I went and read their about page23:09
mordredsdague: they seem somewhat sane23:09
mordredsdague: we coudl also run our own url shortener I suppose23:09
sdagueyeh, more concerned about uptime23:09
sdagueand what happens if it is down23:09
sdagueand explodes23:09
mordredsdague: I believe we just won't get shortened urls23:10
sdaguehow long does requests go before it times out?23:11
mordredgood question ...23:11
* mordred looks23:11
sdagueI'm not actually in the "we always need to run our own camp", just want to make sure there is graceful handling of network outages23:12
sdaguebecause... there are always network outages :)23:12
mordredsdague: configurable23:12
mordredsdague: what do you think is reasonable? 2 seconds?23:12
openstackgerritA change was merged to openstack-infra/elastic-recheck: remove the stevedore query, it's almost all success  https://review.openstack.org/7737323:12
sdague2 seems small. 10 is fine23:12
sdaguejust making sure it wasn't 120 or something23:13
sdagueand not responding until that completed23:13
sdagueor timed out23:13
anteayado we want a message posted that the url couldn't be shortened?23:13
anteayaor silent failing?23:13
StevenKPersonally, silent failing23:13
sdagueyeh, silent failing is fine23:14
sdagueanyway, time to run off for a bit. Enjoy folks.23:14
openstackgerritMonty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls  https://review.openstack.org/7737223:15
mordredoh - well, I just added an error message...23:15
mordredStevenK: you think we should silent fail?23:16
openstackgerritMonty Taylor proposed a change to openstack-infra/meetbot: Reply to lines with urls with shortened urls  https://review.openstack.org/7737223:16
StevenKmordred: Yes23:16
mordredk. done23:17
anteayaneutron ryu ml update: http://lists.openstack.org/pipermail/openstack-infra/2014-March/000931.html23:20
*** pmathews has quit IRC23:26
*** dolphm is now known as dolphm_50323:29
*** dolphm_503 is now known as dolphm23:30
mikalanteaya: would something like http://www.rcbops.com/gerrit/reports/nova-cireport.html for neutron CI systems help you?23:31
anteayamikal: hi23:33
anteayaI'm looking at it23:34
anteayalots of good information there23:34
anteayathe bit I would like to track is how many failures are reported because the system couldn't stand up a devstack?23:34
mikalHmmmm, I'd have to parse logs to get that though right?23:34
mikalThat would appear in gerrit comments23:35
mikals/would/wont/23:35
mikalMore coffee needed23:35
anteayawe are probably looking at a logstash scenario23:35
anteayacoffee up23:35
anteayatoss in some kermit arms23:35
*** lnxnut has joined #openstack-infra23:35
mikalAlthough, you'd see a secondary metric of many many -1 votes in that report23:35
* anteaya looks again23:35
anteayaValid patches in report period: 49523:36
anteayawhat time span is the report period?23:36
mikalYep, and then each CI system gets a summary row23:36
mikal7 days23:36
mikalThis is only for nova23:36
anteayaright23:36
*** CaptTofu has joined #openstack-infra23:36
mikalAnd "valid patches" is a bit complex23:36
anteayayeah23:36
mikalOnly master, only ones that lasted more than three hours before being superceeded, etc23:36
*** che-arne has quit IRC23:36
anteayahow is it being consumed in nova?23:37
mikalThis has only existed for a couple of days23:37
mikalThe plan is to use it to decide who has met the "Group B" requirements for hypervisors23:37
mikali.e. its nice if you're voting, but if you're always voting +1 that doesn't count23:37
*** masayukig has joined #openstack-infra23:37
anteayawhy is hyperv ci and turbo hipster in black ink23:38
anteayaand the others in red ink/print/pixels23:38
mikalYou get red if you have missed too many votes23:38
mikal95% IIRC23:38
anteayahmmmm23:39
anteayawhat would cause them to miss votes?23:39
mikalMy current theory is that the CI systems aren't keeping up with the workload23:39
mikalSo they get backlogged and then flushed23:39
*** dolphm is now known as dolphm_50323:39
anteayathe definition of workload here is difuse23:39
anteayawhich makes it hard to hit the mark23:39
mikalI'd like to add a metric about review latency, but I can't without a gerrit upgrade23:40
anteayathe ones that are keeping up are doing so in an outstanding fashion23:40
*** lnxnut has quit IRC23:40
*** Sukhdev has quit IRC23:40
anteayaah sorry, you were replying to my question 'what would cause them to miss votes' answer being processing the backlog23:41
mikalYeah23:41
*** CaptTofu has quit IRC23:41
anteayanow I am with you23:41
anteayawell certainly having some sort of page to point to would be helpful23:42
mikalOk23:42
mikalI shall have a quick hack tomorrow and see what I can do23:42
anteayaeven if it just manually updated how many devs had come to me to complain about the ci23:42
*** thuc has joined #openstack-infra23:42
mikalWhere can I find a list of the usernames doing neutron CI?23:43
anteayathen I could teach them all to look at it23:43
anteayamikal: here is all the gerrit 3rd party accounts: https://review.openstack.org/#/admin/groups/270,members23:43
anteayayou will see it contains a sublist23:43
anteayathis list is the non-voting accounts and also the sub-list of voting accounts23:44
anteayahttps://review.openstack.org/#/admin/groups/91,members23:44
mikalCool23:44
anteayabut this is everybody, not just neutron23:44
mikalIs there a simple way to tell which ones are neutron?23:44
anteayanot that I know of23:44
mikalI cna just guess worst case23:44
anteayavmware minesweeper is nova and neutron23:44
anteayathey don't have to tell us their testing intentions when they register the account23:45
anteayajust name, ssh key and hopefully contact info23:45
anteayaand thank you23:45
mikalOk, I shall guess23:46
mikalAnd if I miss any we can add them later23:46
anteayasounds great to me23:46
anteayayou have my thanks kind sir23:46
anteayamikal: can you look at this log: http://180.37.183.32/jenkins/job/ryuplugin-tempest/9/console23:47
anteayadoes it appear to you that the ci is still failing to stand up a devstack?23:47
mikalYes, "devstack failed"23:48
anteayalooking again I am in correct23:48
*** che-arne has joined #openstack-infra23:49
*** dolphm_503 is now known as dolphm23:49
anteayamikal: this is the patch: https://review.openstack.org/#/c/74490/23:49
mikalThat's a lot of CI checks23:50
anteayaand the system says it voted on it at 4:36pm https://review.openstack.org/#/dashboard/9681 (which I think is 4:36pm my time) which would be 21:36 utc23:50
anteayathe ci crowd is vast23:51
anteayastack.sh log /var/lib/jenkins/jobs/ryuplugin-tempest/workspace/logs/devstack.2014-02-28-00380623:51
anteayaso what timestamp is that?23:52
anteayaFebruary 28 2014, ?23:52
anteaya00:38 timezone of server?23:52
mikalI'd say so, yes23:52
mikalYep23:52
mikalThat's how I'd read it at least23:52
anteayabut we don't know what time zone this server is using23:52
anteayawe can't assume utc23:53
anteayafungi: are you willing to weigh in?23:53
mikalIf you go to the front page, it says "Page generated: Mar 2, 2014 8:53:19 AM"23:53
mikalWhich means two hours behind me I guess23:53
mikalWhich is approximately the timezone of Hong Kong23:53
anteayaPage generated: Mar 2, 2014 7:43:18 AM23:54
anteayais what i have23:54
mikalhttp://180.37.183.32/jenkins/ is where I got that timestamp from23:54
anteayahttp://180.37.183.32/jenkins/job/ryuplugin-tempest/9/artifact/logs/23:54
anteayaI refreshed the page: Page generated: Mar 2, 2014 8:54:51 AM23:55
anteayaso that is just the timestamp for when your browser last rendered the page, it looks like23:55
mikalRyu is based in Japan?23:55
anteayahttp://lists.openstack.org/pipermail/openstack-infra/2014-March/000928.html23:55
anteayaMotoki-san23:55
anteayaI'd go with Japan23:56
mikalMy point is, its a timestamp of the server time when the page was generated23:56
anteayaah23:56
mikalAnd it matches the current time in Tokyo (8:55am)23:56
anteayanow I am with you23:56
mikali.e. it helps you work out what timezone the log is in23:56
anteayaTokyo time23:56
anteayaabout 33 hours ago23:57
anteayado you calcuate the same?23:58
*** CaptTofu has joined #openstack-infra23:58
*** markmcclain has quit IRC23:58
*** dolphm is now known as dolphm_50323:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!