| openstackgerrit | A change was merged to openstack-infra/config: Remove neutron isolated jobs https://review.openstack.org/71947 | 00:01 |
|---|---|---|
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror https://review.openstack.org/68818 | 00:01 |
| jeblair | wenlock: i think this is one for openstack-infra@lists.o.o | 00:01 |
| kevinbenton | someone had a cool animation of the speculative merge process in the gate, does anyone have a link to that? | 00:02 |
| clarkb | kevinbenton: yes one sec | 00:02 |
| jeblair | kevinbenton: http://docs.openstack.org/infra/publications/zuul/#%2818%29 | 00:02 |
| clarkb | jeblair wins | 00:03 |
| kevinbenton | clarkb, jeblair: thanks | 00:03 |
| jeblair | kevinbenton: http://docs.openstack.org/infra/publications/zuul/ is the start of the talk | 00:03 |
| jeblair | kevinbenton: and http://docs.openstack.org/infra/publications/ is a bunch of talks | 00:03 |
| *** NikitaKonovalov has quit IRC | 00:04 | |
| *** NikitaKonovalov has joined #openstack-infra | 00:04 | |
| kevinbenton | jeblair: thanks | 00:04 |
| *** jhesketh__ has joined #openstack-infra | 00:05 | |
| *** SumitNaiksatam has quit IRC | 00:07 | |
| krotscheck | ok, clarkb, I thihnk I've addressed all your concerns. | 00:07 |
| clarkb | GheRivero: for 1287975 how are you installing the pypy interpreter? | 00:08 |
| *** zns has quit IRC | 00:08 | |
| clarkb | GheRivero: because vanilla precise doesn't do pypy iirc | 00:09 |
| GheRivero | from the ppa that I found in the infra manifests | 00:09 |
| clarkb | ah ok, so pypy is there from the ppa thanks | 00:09 |
| GheRivero | ppa:pypy/ppa | 00:10 |
| clarkb | Alex_Gaynor: https://bugs.launchpad.net/oslo/+bug/1287975 is that something you are familiar with? | 00:10 |
| Alex_Gaynor | clarkb: nope, never seen that error before in my life | 00:11 |
| *** sarob_ has quit IRC | 00:11 | |
| *** sarob_ has joined #openstack-infra | 00:11 | |
| clarkb | Alex_Gaynor: GheRivero: it is an import error | 00:12 |
| *** sabari has joined #openstack-infra | 00:12 | |
| clarkb | Alex_Gaynor: does lxml work with pypy? | 00:13 |
| clarkb | that is a C binding right? | 00:13 |
| Alex_Gaynor | clarkb: I don't believe so, no, there's a fork that does htough. | 00:13 |
| *** krotscheck has quit IRC | 00:14 | |
| GheRivero | it works. I couldn;t reproduce that bug in a vanilla precise, so i guess it works | 00:14 |
| *** krotscheck has joined #openstack-infra | 00:16 | |
| clarkb | jeblair: jhesketh_ https://review.openstack.org/#/c/67858/ is my comment there sane? | 00:17 |
| clarkb | wenlock: https://review.openstack.org/#/c/72496/ for the escaped $, does puppet escpae that in a single quoted variable? | 00:20 |
| jhesketh__ | clarkb: yes, that sounds good to me and I share the same opinion. I would go so far to say I don't like the name of the option "git_avoid_http". (I had made similar comments on previous patchsets) | 00:20 |
| *** oubiwan__ has quit IRC | 00:21 | |
| jeblair | clarkb, jhesketh__: i think that all may be in service of the code we discussed possibly getting rid of this morning... | 00:21 |
| wenlock | clarkb, trying to recall that one... 1 sec | 00:21 |
| clarkb | jeblair: oh right, good point | 00:22 |
| *** oubiwan__ has joined #openstack-infra | 00:22 | |
| *** krotscheck has quit IRC | 00:22 | |
| jeblair | jhesketh__: i think that code is all about making sure that gerrit replication has happened before considering a change merged | 00:22 |
| jhesketh__ | jeblair: I missed that discussion, which code are you proposing we remove | 00:22 |
| jeblair | jhesketh__: we're starting to time out on that sometimes, and i think it's actually probably irrelevant for us now... | 00:22 |
| *** sweston has joined #openstack-infra | 00:23 | |
| jeblair | jhesketh__: because we expect builders to get time-sensitive repo updates from zuul mergers, and mergers get them directly from gerrit | 00:23 |
| jeblair | jhesketh__: so replication shouldn't be a factor for us | 00:23 |
| jhesketh__ | jeblair: I'm not sure I follow what that has to do with using ssh instead of https? | 00:23 |
| *** changbl has quit IRC | 00:24 | |
| jeblair | jhesketh__: i believe _getInfoRefs is only used in that check | 00:24 |
| *** krotscheck has joined #openstack-infra | 00:24 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/77187 | 00:24 |
| jhesketh__ | ah, so you're saying this patch doesn't matter because that's going away? | 00:25 |
| *** jcoufal has quit IRC | 00:25 | |
| jeblair | jhesketh__: that's what i'm thinking; i haven't confirmed all that or proposed the change yet, but i'm pretty sure that's all correct and we can safely drop the whole thing. | 00:26 |
| wenlock | clarkb, i recall this one now, i had no way to test it, but the intent was to try to get geppeto syntax check on that line to not throw complaints about the $ being miss interpreted as a value for a variable... Adding an escape fixes it, and still results on the same string output from puppet when testing with puppet -e | 00:26 |
| jeblair | jhesketh__: it will have a really good commit message when i propose it. :) | 00:26 |
| jhesketh__ | heh, okay | 00:26 |
| jhesketh__ | sounds good to me | 00:26 |
| clarkb | wenlock: yeah it does | 00:26 |
| jhesketh__ | makes sense for the mergers to be the one caring about the repos | 00:26 |
| clarkb | wenlock: which is really weird to me because $ in '' is meaningless | 00:26 |
| jhesketh__ | jeblair: but what about the mergers fetching the change over ssh? | 00:26 |
| clarkb | wenlock: lgtm | 00:26 |
| jeblair | jhesketh__: did this change actually alter that? | 00:27 |
| clarkb | it didn't | 00:27 |
| clarkb | my change addresses that | 00:27 |
| clarkb | whcih I will write a test for as soon as I am caught up on other things | 00:27 |
| * jhesketh__ might be confusing the patches | 00:28 | |
| *** cadenzajon_ has joined #openstack-infra | 00:28 | |
| *** mayu has joined #openstack-infra | 00:30 | |
| *** nati_ueno has quit IRC | 00:30 | |
| mayu | jaypipes: http://paste.openstack.org/show/72379/ | 00:31 |
| *** cadenzajon has quit IRC | 00:32 | |
| clarkb | mayu: you should look at the logs devstack logs | 00:32 |
| *** sarob_ has quit IRC | 00:32 | |
| *** sarob_ has joined #openstack-infra | 00:33 | |
| mayu | clarkb:ok | 00:34 |
| *** Ryan_Lane has quit IRC | 00:34 | |
| *** Alexandra has joined #openstack-infra | 00:35 | |
| *** sarob_ has quit IRC | 00:37 | |
| *** markmc has quit IRC | 00:37 | |
| *** atiwari has quit IRC | 00:37 | |
| *** mayu has quit IRC | 00:38 | |
| *** arborism is now known as amcrn | 00:40 | |
| pleia2 | derekh: https://review.openstack.org/#/c/60375/3 | 00:40 |
| *** nati_ueno has joined #openstack-infra | 00:41 | |
| *** mgagne has quit IRC | 00:44 | |
| *** yamahata__ has quit IRC | 00:44 | |
| *** hogepodge has quit IRC | 00:46 | |
| *** CaptTofu has joined #openstack-infra | 00:46 | |
| clarkb | jeblair: https://review.openstack.org/#/c/71648/3 will that make changes unmergable? it will right? we need a gate pipeline with at least one job in it | 00:48 |
| *** dstanek has joined #openstack-infra | 00:49 | |
| jeblair | clarkb: i think so | 00:49 |
| *** unicell has joined #openstack-infra | 00:50 | |
| *** CaptTofu has quit IRC | 00:50 | |
| *** unicell has quit IRC | 00:52 | |
| fungi | seeing what i've missed... | 00:52 |
| *** unicell has joined #openstack-infra | 00:52 | |
| clarkb | it feels good to sit down and do a proper afternoon of review | 00:52 |
| *** zehicle_at_dell has quit IRC | 00:53 | |
| *** Ryan_Lane has joined #openstack-infra | 00:54 | |
| anteaya | clarkb: strange to do it on ff day though | 00:55 |
| anteaya | pleasantly so | 00:55 |
| clarkb | anteaya: yes very strang | 00:56 |
| *** Hefeweizen has quit IRC | 00:57 | |
| *** Hefeweizen has joined #openstack-infra | 00:58 | |
| * clarkb walks home to pick up packages before the office closes | 00:59 | |
| anteaya | enjoy your walk | 01:00 |
| fungi | clarkb: jeblair: wenlock: yes to meeting agenda | 01:01 |
| *** yongli has joined #openstack-infra | 01:01 | |
| clarkb | woot success if I adjust the backlight brightness then the screen comes back on after switching bcak to a different display | 01:04 |
| *** blamar has quit IRC | 01:05 | |
| *** lcheng_ has quit IRC | 01:05 | |
| *** bhuvan__ has joined #openstack-infra | 01:06 | |
| *** bhuvan___ has joined #openstack-infra | 01:06 | |
| *** prad has joined #openstack-infra | 01:08 | |
| *** bhuvan_ has quit IRC | 01:10 | |
| *** bhuvan has quit IRC | 01:10 | |
| *** dprince has quit IRC | 01:10 | |
| *** wchrisj has quit IRC | 01:11 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add configurable footer-message reports https://review.openstack.org/77740 | 01:12 |
| geekinutah | so just randomly tracking down a bug I ran into this http://logs.openstack.org/65/77065/3/gate/gate-tempest-dsvm-neutron/eacf1d6/console.html#_2014-03-05_00_50_43_338 | 01:15 |
| geekinutah | the errors that this ignored seem related to the bug I am tracking, thought it was interesting that it causes checks to fail in nova but the gate will let it through for tempest | 01:15 |
| fungi | geekinutah: we have a script which is checking the various service logs for errors on tempest runs, but there are currently so many false negatives from it that we aren't enforcing that yet | 01:18 |
| fungi | we were briefly, but then something broke the enforcement just long enough for tons of new uncategorized (presumably benign) error messages to creep into the logs | 01:19 |
| *** jcoufal has joined #openstack-infra | 01:19 | |
| jeblair | fungi: i think in the project meeting today sdague said he planned on tackling that again after i3 | 01:19 |
| geekinutah | I see, there does seem to be a ton of variance for errors related to trying to set tags on non-existent tap devices | 01:19 |
| geekinutah | I think I have tracked down what appear to be at least 3 seperate services pulling the rug out from under the q-agt | 01:20 |
| fungi | i hope so--that was a huge potential benefit for making the logs useful not only for diagnosing test failures but even moreso in production | 01:20 |
| openstackgerrit | A change was merged to openstack-infra/config: increase conference extensions range https://review.openstack.org/73264 | 01:22 |
| fungi | if that nova change at the head of the gate doesn't take too long merging, there are 9 more changes already with successful job completion ready to merge right behind it | 01:23 |
| fungi | here it goes... | 01:24 |
| *** derekh has quit IRC | 01:24 | |
| clarkb | jogo is this ml thread on scheduler testing be done with fake libvirt like largeops | 01:25 |
| anteaya | reset | 01:25 |
| fungi | anteaya: yep. 72368,8 (nova) seems to have done the same as we've been witnessing earlier, with 77215,4 (python-keystoneclient) and all the other 9 changes whose testing had completed behind it are being retested now | 01:26 |
| anteaya | is that our old friend apache check timeout? | 01:26 |
| anteaya | :( | 01:26 |
| fungi | anteaya: looks like it | 01:26 |
| anteaya | including 3 neutron | 01:26 |
| anteaya | is it just after a nova patch? | 01:26 |
| fungi | the last remaining job on 72368,8 completed successfully and it seems to have merged | 01:26 |
| fungi | anteaya: seems like nova is currently the only project with a large enough git repository to cross the timeout threshold | 01:27 |
| jogo | clarkb: I must have missed that thread | 01:27 |
| jogo | I think that was just for local testing | 01:27 |
| jogo | not sure | 01:27 |
| jogo | whats the thread title | 01:27 |
| clarkb | jogo: let me find it for you | 01:29 |
| *** rwsu has quit IRC | 01:29 | |
| clarkb | jogo: http://lists.openstack.org/pipermail/openstack-dev/2014-March/028751.html | 01:29 |
| clarkb | jogo: It seems to be right up what you did's alley | 01:29 |
| *** gokrokve_ has quit IRC | 01:30 | |
| *** harlowja has quit IRC | 01:31 | |
| *** tomhe has quit IRC | 01:31 | |
| *** markmcclain has quit IRC | 01:32 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 01:33 |
| jeblair | clarkb, fungi, pleia2: ^ that's about half the work needed to manage irc channel access in the config repo | 01:34 |
| jeblair | maybe 80% | 01:34 |
| *** jcoufal has quit IRC | 01:35 | |
| jogo | clarkb: so I assumed that thread wasn't about testing in gate | 01:36 |
| jogo | but locally | 01:36 |
| *** vkozhukalov has joined #openstack-infra | 01:36 | |
| jogo | so ignored it | 01:36 |
| clarkb | jogo: it may be, but they are floudering and suggesting silly things like btrfs and containers | 01:36 |
| clarkb | when none of that is necessary | 01:36 |
| *** wchrisj has joined #openstack-infra | 01:37 | |
| jogo | there are only so many threads I can be involved in ... so since I didn't think it was about testing in gate I just ignored it | 01:37 |
| *** mrodden has joined #openstack-infra | 01:38 | |
| geekinutah | clarkb: I am interested in what the un-silly way to go about this is | 01:38 |
| clarkb | geekinutah: do what jogo did | 01:39 |
| clarkb | and have a fake driver behind there that says yup I scheduled a thing | 01:39 |
| clarkb | then you don't have to touch anything IO bound | 01:39 |
| clarkb | or use unstable filesystems | 01:39 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 01:39 |
| *** yamahata__ has joined #openstack-infra | 01:39 | |
| geekinutah | so yeah, that's what I did, used the fake driver | 01:39 |
| jogo | clarkb: so that mostly works | 01:39 |
| anteaya | jeblair: commented | 01:39 |
| geekinutah | in tons of greenthreads | 01:39 |
| anteaya | jeblair: two typos | 01:39 |
| geekinutah | and it mostly works... :-) | 01:40 |
| jogo | so if that is too heavy, just test the algorithm via unit tests | 01:40 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 01:40 |
| jeblair | anteaya: thx | 01:40 |
| jogo | geekinutah: get a bigger box? | 01:40 |
| clarkb | geekinutah: what doesn't work? | 01:40 |
| anteaya | np | 01:40 |
| clarkb | if fake driver is too heavy containers and btrfs will only make it worse | 01:41 |
| geekinutah | it's not too heavy I don't think | 01:41 |
| geekinutah | I was simulating thousands of compute nodes | 01:41 |
| geekinutah | the problem was in greenthreads really and the points of concurrency | 01:41 |
| *** vkozhukalov has quit IRC | 01:42 | |
| jogo | geekinutah: ahh so its an eventlet realated issue | 01:42 |
| geekinutah | which is why if I had to do it again I would spin up a few containers to run 500 or so threads each instead of10k greenthreads | 01:42 |
| geekinutah | jogo: yeah essentially, also just a normal context switching problem I think | 01:43 |
| geekinutah | kernel can only handle so many of those, and 10k greenthreads was encroaching on the ridiculous zone | 01:43 |
| clarkb | geekinutah: I guess I am still missing how containers help | 01:43 |
| geekinutah | clarkb: I haven't proved this at all, but my thinking is that we will be able to avoid the limitations of eventlet (not that you need a container to do that) | 01:44 |
| geekinutah | containers are pretty light, so it seems like less work to me | 01:44 |
| clarkb | compared to running multiple processes? | 01:45 |
| geekinutah | I suppose just spinning up more python procs achieves the same thing, I just haven't thought through all the implications | 01:45 |
| clarkb | I feel like we are taking a simple problem and making it hard | 01:45 |
| clarkb | davidlenwell: ^^ speaking of, imo the refstack thingy shouldn't be containerized either | 01:45 |
| jeblair | Stop! Think! There must be a harder way! | 01:45 |
| clarkb | tempest installs in a virtualenv, just pip install tempest and run it | 01:46 |
| davidlenwell | clarkb: I don't dissagree with you in general .. | 01:46 |
| geekinutah | ie. I'm not sure if there are synchronization points that all sub-processes would share | 01:46 |
| geekinutah | which is why I thought containers | 01:46 |
| clarkb | davidlenwell: I mean feel free to use containers to run it when you run it | 01:46 |
| clarkb | but making docker a dependency of refstack is a bit much | 01:47 |
| davidlenwell | clarkb: agreed .. thats why the script runs inside or outside of docker | 01:47 |
| *** wenlock has quit IRC | 01:47 | |
| davidlenwell | my plan for the official running copy of refstack is to use gearman jobs.. the docker container is meant as an easy way for operators to deploy tests in an envoirnment we have a little control over .. but behind their firewall | 01:48 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/77187 | 01:48 |
| jogo | backing up fora second, what did you do | 01:48 |
| jogo | devstack with fake virt and 500 nova compute instances? | 01:48 |
| clarkb | davidlenwell: why not pip install tempest? | 01:48 |
| davidlenwell | clarkb: I've also screamed up and down that I don't wanna use docker at all because its too immature | 01:48 |
| anteaya | to whom? | 01:48 |
| davidlenwell | anteaya: the defcore and refstack team .. we had a f2f on monday | 01:49 |
| jogo | davidlenwell: wait WAT what do they need docker for? | 01:49 |
| clarkb | for running refstack aiui | 01:49 |
| jogo | geekinutah: ^^ | 01:49 |
| jogo | just use cloud instances? | 01:50 |
| jogo | for refstck | 01:50 |
| * clarkb feels bad for ditching the conversation now but is going to do that to play video games | 01:50 | |
| geekinutah | jogo: think devstack with fake virt and 500 nova compute greenthreads | 01:50 |
| clarkb | I will watch the zuul status page | 01:50 |
| davidlenwell | thanks clarkb | 01:50 |
| anteaya | clarkb: happy video games | 01:50 |
| clarkb | davidlenwell: stick of truth is running now | 01:50 |
| geekinutah | jogo: each greenthread sees itself as a nova-compute service | 01:50 |
| jogo | geekinutah: 500 in one process? | 01:50 |
| geekinutah | yeah | 01:50 |
| jogo | ahh | 01:50 |
| clarkb | davidlenwell: its south park humor + obsidian should be great | 01:50 |
| geekinutah | they don't do much since it's the fake backend | 01:50 |
| geekinutah | but they excercise the DB and the message bus | 01:50 |
| jogo | why not do x in greenthread and x processes | 01:50 |
| davidlenwell | clarkb: .. sounds like fun | 01:51 |
| geekinutah | jogo: did that also, works better | 01:51 |
| jogo | because otherwise your aren't taking advanage of the many CPUs you have | 01:51 |
| anteaya | from my seat in the peanut gallery it looks to me like refstack and defcore are doing their own thing, and separating away from openstack, rather than positioning themselves to define it | 01:51 |
| geekinutah | right | 01:51 |
| davidlenwell | so to be clear .. refstack is not dependant on docker .. | 01:51 |
| jogo | geekinutah: so what was the issue? | 01:51 |
| *** tianst20 has joined #openstack-infra | 01:51 | |
| clarkb | davidlenwell: ok, good to know I must've misread that that was the official way to run it | 01:52 |
| davidlenwell | we've made a docker thing that makes it easy to deploy our tests and fetch results .. but thats just some python code that the docker file runs .. that code works in or out of a docker container | 01:52 |
| jogo | davidlenwell: sounds very un-openstack | 01:52 |
| *** sparkycollier has joined #openstack-infra | 01:52 | |
| geekinutah | jogo: well, it mostly worked for scheduler testing I think, because it produced the load that we wanted it for (message bus and DB) | 01:52 |
| anteaya | jogo: +1 | 01:52 |
| clarkb | right but what is the documented supported way of doing the thing? | 01:52 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 01:52 |
| *** ctracey has quit IRC | 01:52 | |
| geekinutah | jogo: but the hacky way we did it broke other things, I never got around to making it right | 01:52 |
| *** krotscheck has quit IRC | 01:53 | |
| jogo | geekinutah: ahh, well what you did sounds like the right way (lightest weight and closest to reality) | 01:53 |
| davidlenwell | jogo: who defines what is openstacky | 01:53 |
| jogo | davidlenwell: the APIs openstack provides | 01:53 |
| jogo | refstack should run on openstack | 01:53 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add a job to check IRC channel access https://review.openstack.org/78061 | 01:53 |
| davidlenwell | I'm sorry jogo .. we've not been formally introduced.. | 01:53 |
| geekinutah | jogo, clarkb: now that I think about it I think the only thing containers could buy you is the ability to simulate "nodes" without spinning up another VM? | 01:54 |
| jogo | davidlenwell: I guess not, hi I am Joe Gordon | 01:54 |
| davidlenwell | sounds like you have a lot of opinions on this topic and I'd love for you to join the conversaiton about it | 01:54 |
| geekinutah | maybe just a tad bit lighter than a full VM? | 01:54 |
| jogo | davidlenwell: well I think defcore is insane | 01:54 |
| jogo | but I would be happy to rant | 01:54 |
| davidlenwell | jogo: no dissagreements | 01:54 |
| jogo | if you point me where | 01:54 |
| anteaya | davidlenwell: well we do have a way of doing things | 01:54 |
| davidlenwell | #refstack | 01:54 |
| clarkb | geekinutah: right | 01:54 |
| anteaya | for the openstack programs | 01:54 |
| clarkb | geekinutah: I don't think it makes parallelization of resources any easier | 01:54 |
| clarkb | geekinutah: but would simulate cross host communication of processes | 01:55 |
| davidlenwell | anteaya: we do have a way of doing things .. but there is no mandate that new projects have to follow those guidlines | 01:55 |
| anteaya | and this doesn't seem yet to embrace what is currently in existance in the openstack programs | 01:55 |
| anteaya | that doesn't sound like embracing what is currently going on, to me | 01:55 |
| davidlenwell | anteaya: please don't rush to judgment .. its still very new | 01:55 |
| geekinutah | clarkb: beyond the scheduler test that this thread mentions, there are peer to peer cases that we wanted to test (think zmq) | 01:56 |
| davidlenwell | anteaya: and still very incomplete | 01:56 |
| geekinutah | it's just been so long since I last thought about this seriously.....sigh | 01:56 |
| anteaya | I'm not judging, I'm stating what I see | 01:56 |
| anteaya | a ship pulling away from the harbour may not be at its destination | 01:56 |
| *** nosnos has joined #openstack-infra | 01:56 | |
| anteaya | but you can tell from whether it is facing east or west where it might end up | 01:56 |
| davidlenwell | So the mechinism for triggereing tests will look a lot like what infra does with stuff | 01:56 |
| *** sparkycollier has quit IRC | 01:56 | |
| clarkb | davidlenwell: anyways +1 to not requiring docker. and fwiw the problems with docker are they don't actually make it easier for folk in many cases. You need a very new kernel and until recently needed fs drivers that werent in tree | 01:57 |
| anteaya | what do you see infra doing with stuff | 01:57 |
| clarkb | this is getting better with docker which makes me happy but folks that are likely to have firewall trouble are also more likely to be running rhel5 | 01:57 |
| *** thuc has quit IRC | 01:57 | |
| jogo | geekinutah: what do you wnat to test in zmq case? | 01:58 |
| *** thuc has joined #openstack-infra | 01:58 | |
| geekinutah | jogo: I was trying to simulate the gearman style scheduler that soren and I talked about in HK | 01:58 |
| *** cody-somerville has joined #openstack-infra | 01:59 | |
| *** cody-somerville has quit IRC | 01:59 | |
| *** cody-somerville has joined #openstack-infra | 01:59 | |
| geekinutah | jogo: I was able to validate parts of that story, but needed better isolation of nova-computes to really answer some of my curiosities | 01:59 |
| *** mgagne has joined #openstack-infra | 02:00 | |
| jogo | geekinutah: you have a big cloud right? | 02:00 |
| jogo | geekinutah: spin up a bunch if really tiny VMs on your cloud | 02:00 |
| davidlenwell | anteaya: So as I have stated .. the docker portion is just a bonus.. the final , official tester will be triggered with gearman workers and work very much the way that you guys trigger tempest runs and digs through the subunit output .. in fact I've borrowed code from infra stuff to do that | 02:00 |
| geekinutah | jogo: I have a big cloud that has most of its resources claimed | 02:00 |
| jogo | geekinutah: use really tiny vms | 02:01 |
| jeblair | davidlenwell: i'm assuming this will eventually be run in infra, yeah? | 02:01 |
| davidlenwell | jeblair: that is my hope .. | 02:01 |
| jeblair | great | 02:01 |
| davidlenwell | but it also will alow private cloud operators to run it on there own systems | 02:01 |
| *** thuc has quit IRC | 02:02 | |
| *** rfolco has joined #openstack-infra | 02:02 | |
| davidlenwell | which is why we made the docker thing.. specifially dell and ibm want to start showing off the interop features in their private cloud offerings | 02:03 |
| jogo | geekinutah: so there are some other ways to make the gearmany style schedule testing easier too | 02:03 |
| geekinutah | jogo: I think I could do that maybe..... I really need to look at how much extra capacity we have at any given moment | 02:03 |
| jogo | geekinutah: I assume there are a few parts to this, perf and functinality | 02:03 |
| jeblair | davidlenwell: makes sense | 02:03 |
| davidlenwell | and jogo.. to answer your question about why we don't just pip install tempest .. we need to be able to get very version specific with our tests .. don't wan to run icehouse tempest against a grizzly cloud | 02:03 |
| jogo | davidlenwell: just install the right version then | 02:04 |
| davidlenwell | sometimes the right version is a specific commit to the git repo .. | 02:05 |
| davidlenwell | isn't packaged | 02:05 |
| jogo | so? | 02:05 |
| jogo | who is running tempest you or someone else? | 02:05 |
| davidlenwell | anyone | 02:05 |
| jogo | so we can make releases of tempest so you can pip install | 02:06 |
| davidlenwell | we'll run it .. and anyone else can run it and collect data using refstack | 02:06 |
| jogo | or however we want to | 02:06 |
| davidlenwell | we preffer straight from git | 02:06 |
| jogo | this sounds a lot like how we do tempest | 02:06 |
| *** dkliban has joined #openstack-infra | 02:06 | |
| jogo | davidlenwell: you will be using tempest for the tests though right? | 02:06 |
| *** bhuvan_ has joined #openstack-infra | 02:06 | |
| *** bhuvan has joined #openstack-infra | 02:06 | |
| davidlenwell | sure | 02:06 |
| jogo | good thats what I meant | 02:07 |
| geekinutah | jogo: we should talk about this more sometime, I gotta run | 02:07 |
| jogo | geekinutah: for sure, just ping me anytime | 02:07 |
| jogo | geekinutah: I like the gearmen scheduler idea alot | 02:07 |
| davidlenwell | jogo: I don't see why we'd use something other that tempest to test openstack .. thats what tempest is for | 02:07 |
| davidlenwell | refstack just collects data about lots of clouds and creates interop maps | 02:08 |
| jogo | davidlenwell: just making sure | 02:08 |
| davidlenwell | jogo: here is an early blog post about how it works .. we've deviated from this greatly .. but it gets the general point accross | 02:09 |
| davidlenwell | http://www.pistoncloud.com/2013/12/how-to-become-interoperable-refstack/ | 02:09 |
| *** mgagne1 has joined #openstack-infra | 02:10 | |
| *** bhuvan___ has quit IRC | 02:10 | |
| *** bhuvan__ has quit IRC | 02:10 | |
| *** wchrisj has quit IRC | 02:11 | |
| *** bhuvan_ has quit IRC | 02:11 | |
| *** bhuvan has quit IRC | 02:11 | |
| *** mgagne has quit IRC | 02:11 | |
| *** rfolco has quit IRC | 02:12 | |
| *** dcramer__ has quit IRC | 02:12 | |
| jogo | davidlenwell: cool | 02:12 |
| jogo | anyway done to many code reviews today | 02:12 |
| jogo | time to go AFK | 02:12 |
| *** reed has quit IRC | 02:15 | |
| *** mwagner_lap has joined #openstack-infra | 02:16 | |
| *** gokrokve has joined #openstack-infra | 02:16 | |
| *** prad has quit IRC | 02:17 | |
| *** harlowja has joined #openstack-infra | 02:17 | |
| *** dcramer__ has joined #openstack-infra | 02:24 | |
| *** briancurtin has joined #openstack-infra | 02:27 | |
| *** CaptTofu has joined #openstack-infra | 02:29 | |
| *** wchrisj has joined #openstack-infra | 02:31 | |
| *** amcrn has quit IRC | 02:35 | |
| *** gokrokve has quit IRC | 02:35 | |
| *** jcooley_ has quit IRC | 02:36 | |
| *** Ryan_Lane has quit IRC | 02:37 | |
| *** wchrisj__ has joined #openstack-infra | 02:37 | |
| *** wchrisj has quit IRC | 02:37 | |
| *** prad has joined #openstack-infra | 02:40 | |
| *** prad has quit IRC | 02:40 | |
| *** wchrisj__ has quit IRC | 02:41 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add GerritBot to openstack-keystone https://review.openstack.org/78071 | 02:43 |
| *** nati_ueno has quit IRC | 02:43 | |
| *** khyati has quit IRC | 02:44 | |
| *** khyati has joined #openstack-infra | 02:45 | |
| *** khyati has quit IRC | 02:45 | |
| *** dcramer__ has quit IRC | 02:50 | |
| *** changbl has joined #openstack-infra | 02:50 | |
| *** coolsvap1 has quit IRC | 02:55 | |
| *** ctracey has joined #openstack-infra | 02:57 | |
| *** thomasem has joined #openstack-infra | 02:59 | |
| *** thomasem has quit IRC | 02:59 | |
| *** dcramer__ has joined #openstack-infra | 03:03 | |
| *** gokrokve has joined #openstack-infra | 03:05 | |
| *** SumitNaiksatam has joined #openstack-infra | 03:06 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Add GerritBot to openstack-keystone https://review.openstack.org/78071 | 03:06 |
| *** thuc has joined #openstack-infra | 03:08 | |
| *** sweston has quit IRC | 03:09 | |
| *** VijayTripathi has joined #openstack-infra | 03:09 | |
| *** thuc has quit IRC | 03:13 | |
| *** harlowja is now known as harlowja_away | 03:13 | |
| *** Will has joined #openstack-infra | 03:21 | |
| *** Will is now known as Guest58153 | 03:21 | |
| *** zns has joined #openstack-infra | 03:24 | |
| *** weshay has quit IRC | 03:27 | |
| jogo | pleia2: are you caltraining tomorrow? | 03:28 |
| *** Shrews has quit IRC | 03:32 | |
| *** Shrews has joined #openstack-infra | 03:32 | |
| *** mgagne has joined #openstack-infra | 03:33 | |
| *** mgagne1 has quit IRC | 03:34 | |
| *** pcrews has quit IRC | 03:37 | |
| anteaya | dang that one test on the horizon patch is taking so long all the other tests are finished | 03:39 |
| anteaya | chances are high we are going to reset again | 03:39 |
| anteaya | :( | 03:39 |
| *** chandan_kumar has joined #openstack-infra | 03:42 | |
| *** VijayTripathi has quit IRC | 03:44 | |
| *** chandan_kumar has quit IRC | 03:46 | |
| *** wchrisj has joined #openstack-infra | 03:47 | |
| *** wchrisj has quit IRC | 03:49 | |
| dkehn_ | anteaya, nay ideas on what version of gerrit you all are running? | 03:51 |
| dkehn_ | s/nay/any/ | 03:52 |
| *** CaptTofu has quit IRC | 03:54 | |
| *** Ryan_Lane has joined #openstack-infra | 03:54 | |
| *** CaptTofu has joined #openstack-infra | 03:54 | |
| *** esker has joined #openstack-infra | 03:55 | |
| *** melwitt has joined #openstack-infra | 03:57 | |
| *** david-lyle has joined #openstack-infra | 03:57 | |
| zns | hi - how do I modify the groups in gerrit? Do they sync with launchpad or is there another way to do it? | 03:58 |
| *** wenlock has joined #openstack-infra | 03:58 | |
| anteaya | dkehn_: one second | 03:58 |
| dkehn_ | anteaya, np | 03:58 |
| anteaya | dkehn_: 2.4 | 03:58 |
| anteaya | a forked version of 2.4 | 03:58 |
| *** CaptTofu has quit IRC | 03:58 | |
| anteaya | and we are working on an upgrade | 03:59 |
| dkehn_ | anteaya, thx, just curious, twiodling around with setting one up | 03:59 |
| anteaya | http://tarballs.openstack.org/ci/gerrit-2.4.4-14-gab7f4c1.war | 03:59 |
| anteaya | if you want it | 03:59 |
| dkehn_ | anteaya, are you going to 2.8 or somewhere else | 03:59 |
| anteaya | dkehn_: https://etherpad.openstack.org/p/mock-manage-projects | 03:59 |
| anteaya | dkehn_: 2.8 I do believe | 03:59 |
| anteaya | those are my notes on setting up gerrit with the intention of running manage-projects | 04:00 |
| *** sweston has joined #openstack-infra | 04:00 | |
| anteaya | the setting up gerrit I can do | 04:00 |
| dkehn_ | anteaya, so trying to educate myself | 04:00 |
| anteaya | the run manage projects, I can't yet | 04:00 |
| anteaya | dkehn_: makes sense to me | 04:00 |
| anteaya | feel free to add if you want, I might have missed somthing | 04:01 |
| anteaya | the top part are the steps I have confidence in the bottom part is my scribbles | 04:01 |
| *** khyati has joined #openstack-infra | 04:01 | |
| dkehn_ | will do | 04:01 |
| anteaya | thanks | 04:02 |
| *** gyee has quit IRC | 04:03 | |
| dkehn_ | anteaya, off to bed we will probably be talking about it, seems I volunteered for it | 04:03 |
| *** esker has quit IRC | 04:03 | |
| zns | Anyone know how I can modify the groups for stackforge/satori in gerrit? | 04:04 |
| *** esker has joined #openstack-infra | 04:04 | |
| zns | Trying to add members... | 04:04 |
| *** krotscheck has joined #openstack-infra | 04:05 | |
| anteaya | dkehn_: good sleep to you | 04:05 |
| anteaya | zns are you the ptl for satori? | 04:06 |
| zns | anteaya: yes | 04:06 |
| anteaya | the ptl has to be added in gerrit by gerrit admins | 04:06 |
| anteaya | what is your gerrit username? | 04:06 |
| zns | We have satori-ptl and satori-core groups (in gerrit and launchpad). | 04:07 |
| anteaya | right | 04:07 |
| anteaya | what is your gerrit username? | 04:07 |
| zns | ziad-sawalha | 04:07 |
| anteaya | great, when one of the gerrit admins show up, they will read the backscroll and add you | 04:08 |
| anteaya | then they will ping you if you are still in channel | 04:08 |
| anteaya | after that you can add your core members to core | 04:08 |
| zns | anteaya: great! Thank you :-) | 04:08 |
| anteaya | welcome | 04:08 |
| anteaya | it will probably be in about 10 hours from now, unless one of them drops in before that | 04:09 |
| *** Alexandra is now known as alex-nomz | 04:10 | |
| zns | ok. I can work around that (prep my patches and approve tomorrow). Thanks again. | 04:11 |
| anteaya | np | 04:11 |
| *** cody-somerville has quit IRC | 04:15 | |
| *** harlowja_away has quit IRC | 04:17 | |
| pleia2 | jogo: I ended up getting a room down here for a couple nights | 04:22 |
| pleia2 | jeblair: love the irc channel reg check | 04:23 |
| *** cody-somerville has joined #openstack-infra | 04:25 | |
| anteaya | okay I'm off | 04:26 |
| anteaya | night | 04:26 |
| *** zns has quit IRC | 04:32 | |
| *** wchrisj has joined #openstack-infra | 04:32 | |
| *** zns has joined #openstack-infra | 04:33 | |
| *** wchrisj has quit IRC | 04:37 | |
| *** thuc has joined #openstack-infra | 04:37 | |
| *** thuc_ has joined #openstack-infra | 04:38 | |
| *** oubiwan__ has quit IRC | 04:41 | |
| *** thuc has quit IRC | 04:41 | |
| *** jp_at_hp has quit IRC | 04:43 | |
| *** alex-nomz is now known as Alexandra | 04:49 | |
| *** lcheng_ has joined #openstack-infra | 04:52 | |
| jogo | pleia2: ahh smart doesn't caltrain stink | 04:53 |
| *** amcrn has joined #openstack-infra | 04:55 | |
| * jogo looks for a hotel for tomorrow night | 04:55 | |
| *** thuc_ has quit IRC | 04:56 | |
| *** thuc has joined #openstack-infra | 04:57 | |
| *** thuc_ has joined #openstack-infra | 04:59 | |
| *** ctracey has quit IRC | 04:59 | |
| *** amcrn_ has joined #openstack-infra | 05:00 | |
| *** thuc has quit IRC | 05:01 | |
| *** amcrn has quit IRC | 05:01 | |
| *** odyi has joined #openstack-infra | 05:04 | |
| *** lcheng_ has quit IRC | 05:09 | |
| *** wchrisj has joined #openstack-infra | 05:11 | |
| *** reed has joined #openstack-infra | 05:14 | |
| *** resker has joined #openstack-infra | 05:17 | |
| *** esker has quit IRC | 05:20 | |
| *** thuc has joined #openstack-infra | 05:26 | |
| *** resker has quit IRC | 05:28 | |
| *** chandankumar_ has quit IRC | 05:28 | |
| *** esker has joined #openstack-infra | 05:29 | |
| *** amcrn_ is now known as amcrn | 05:30 | |
| *** thuc_ has quit IRC | 05:30 | |
| *** thuc has quit IRC | 05:31 | |
| *** resker has joined #openstack-infra | 05:31 | |
| *** esker has quit IRC | 05:33 | |
| StevenK | Blah, why did we just restart the job at the head of the gate, did the patch before last job fail? | 05:33 |
| *** amcrn is now known as DrKatz | 05:34 | |
| *** jcooley_ has joined #openstack-infra | 05:36 | |
| *** lcheng_ has joined #openstack-infra | 05:36 | |
| *** markmcclain has joined #openstack-infra | 05:37 | |
| *** resker has quit IRC | 05:37 | |
| *** chandan_kumar has joined #openstack-infra | 05:38 | |
| *** markmcclain1 has joined #openstack-infra | 05:39 | |
| *** jcooley_ has quit IRC | 05:40 | |
| *** talluri has joined #openstack-infra | 05:41 | |
| *** markmcclain has quit IRC | 05:41 | |
| *** talluri has quit IRC | 05:41 | |
| *** talluri has joined #openstack-infra | 05:42 | |
| *** jcooley_ has joined #openstack-infra | 05:42 | |
| *** DrKatz is now known as amcrn | 05:42 | |
| *** amotoki has joined #openstack-infra | 05:46 | |
| *** VijayTripathi has joined #openstack-infra | 05:46 | |
| *** VijayTripathi has quit IRC | 05:46 | |
| *** lcheng_ has quit IRC | 05:48 | |
| *** briancurtin has quit IRC | 05:48 | |
| *** alexpilotti_ has joined #openstack-infra | 05:49 | |
| alexpilotti_ | jeblair: hi | 05:50 |
| *** unicell has quit IRC | 05:53 | |
| *** CaptTofu has joined #openstack-infra | 05:55 | |
| *** nicedice has quit IRC | 05:55 | |
| *** unicell has joined #openstack-infra | 05:58 | |
| *** esker has joined #openstack-infra | 05:59 | |
| *** CaptTofu has quit IRC | 06:00 | |
| *** resker has joined #openstack-infra | 06:01 | |
| *** alexpilotti_ has quit IRC | 06:02 | |
| *** esker has quit IRC | 06:04 | |
| *** resker has quit IRC | 06:06 | |
| *** esker has joined #openstack-infra | 06:09 | |
| *** resker has joined #openstack-infra | 06:11 | |
| *** esker has quit IRC | 06:13 | |
| *** krotscheck has quit IRC | 06:14 | |
| *** resker has quit IRC | 06:16 | |
| *** jcooley_ has quit IRC | 06:22 | |
| *** reed has quit IRC | 06:28 | |
| *** wchrisj has quit IRC | 06:29 | |
| *** ctracey has joined #openstack-infra | 06:32 | |
| *** denis_makogon has joined #openstack-infra | 06:34 | |
| *** basha has joined #openstack-infra | 06:35 | |
| *** thuc has joined #openstack-infra | 06:37 | |
| *** gokrokve has quit IRC | 06:37 | |
| *** jcooley_ has joined #openstack-infra | 06:38 | |
| *** alexpilotti has quit IRC | 06:39 | |
| *** thuc has quit IRC | 06:41 | |
| *** pblaho has joined #openstack-infra | 06:42 | |
| *** vkozhukalov has joined #openstack-infra | 06:47 | |
| *** ctracey has quit IRC | 06:48 | |
| *** mrda is now known as mrda_away | 06:50 | |
| *** jcooley_ has quit IRC | 06:50 | |
| *** jcooley_ has joined #openstack-infra | 06:51 | |
| ttx | clarkb: still around ? | 06:51 |
| SpamapS | man.. hopefully hardware issues squashed on tripleo-check cloud :-/ | 06:53 |
| *** jcooley_ has quit IRC | 06:56 | |
| * ttx likes when he goes to bed and finds the same reviews at the top of gate he left 10 hours ago | 06:56 | |
| * ttx tries to find the answer in backlog | 06:57 | |
| *** vkozhukalov has quit IRC | 07:00 | |
| *** esker has joined #openstack-infra | 07:02 | |
| *** saju_m has joined #openstack-infra | 07:02 | |
| *** jcooley_ has joined #openstack-infra | 07:03 | |
| *** openstackstatus has quit IRC | 07:04 | |
| *** esker has quit IRC | 07:07 | |
| *** yolanda_ has joined #openstack-infra | 07:08 | |
| *** afazekas has joined #openstack-infra | 07:09 | |
| SergeyLukjanov | ttx, AFAIR there were no issues last night | 07:11 |
| ttx | SergeyLukjanov: weird that the top of gate was enqueued > 10 hours ago though | 07:12 |
| ttx | when I left the queue was not THAT deep | 07:12 |
| ttx | must have been a lot of resets | 07:12 |
| SergeyLukjanov | ttx, yup, see it, I think that it was ok when I've fall asleep... | 07:12 |
| SergeyLukjanov | ttx, there were free slaves (due to the slaves graph) a few hours ago | 07:13 |
| SergeyLukjanov | ttx, oh, probably it was resets due to the nova pep8 fail (check_autoupdate), can't remember anything more | 07:15 |
| *** flaper87|afk is now known as flaper87 | 07:15 | |
| *** jcooley_ has quit IRC | 07:16 | |
| *** jcooley_ has joined #openstack-infra | 07:16 | |
| *** jcooley_ has quit IRC | 07:21 | |
| *** ildikov_ has joined #openstack-infra | 07:21 | |
| *** ildikov_ has quit IRC | 07:21 | |
| *** ildikov_ has joined #openstack-infra | 07:22 | |
| *** afazekas is now known as afazekas|food | 07:23 | |
| *** mkoderer has joined #openstack-infra | 07:28 | |
| *** afazekas|food is now known as afazekas | 07:28 | |
| *** amcrn has quit IRC | 07:36 | |
| *** wenlock has quit IRC | 07:37 | |
| *** zns has quit IRC | 07:37 | |
| *** melwitt has quit IRC | 07:42 | |
| *** Alexandra is now known as alexandra | 07:45 | |
| *** alexandra is now known as Alexandra | 07:45 | |
| *** nati_ueno has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:47 | |
| *** khyati has quit IRC | 07:50 | |
| *** Alexandra is now known as alex-out | 07:53 | |
| *** jcooley_ has joined #openstack-infra | 07:53 | |
| *** CaptTofu has joined #openstack-infra | 07:56 | |
| *** basha has quit IRC | 07:56 | |
| *** jcooley_ has quit IRC | 07:57 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 08:02 | |
| *** mgagne1 has joined #openstack-infra | 08:07 | |
| *** e0ne has quit IRC | 08:11 | |
| *** talluri has quit IRC | 08:11 | |
| *** sweston has quit IRC | 08:11 | |
| *** david-lyle has quit IRC | 08:11 | |
| *** mgagne has quit IRC | 08:11 | |
| *** SumitNaiksatam has quit IRC | 08:11 | |
| *** dcramer__ has quit IRC | 08:11 | |
| *** mwagner_lap has quit IRC | 08:11 | |
| *** nosnos has quit IRC | 08:11 | |
| *** luis_ has quit IRC | 08:11 | |
| *** homeless has quit IRC | 08:11 | |
| *** gmoro has quit IRC | 08:11 | |
| *** openstackgerrit has quit IRC | 08:11 | |
| *** antonym has quit IRC | 08:11 | |
| *** maurosr has quit IRC | 08:11 | |
| *** alaski has quit IRC | 08:11 | |
| *** soren has quit IRC | 08:11 | |
| *** ilyashakhat has quit IRC | 08:11 | |
| *** asadoughi has quit IRC | 08:11 | |
| *** mattoliverau has quit IRC | 08:11 | |
| *** che-arne has quit IRC | 08:11 | |
| *** afazekas has quit IRC | 08:11 | |
| *** yolanda_ has quit IRC | 08:11 | |
| *** saju_m has quit IRC | 08:11 | |
| *** denis_makogon has quit IRC | 08:11 | |
| *** unicell has quit IRC | 08:11 | |
| *** Ryan_Lane has quit IRC | 08:11 | |
| *** Shrews has quit IRC | 08:11 | |
| *** cadenzajon_ has quit IRC | 08:11 | |
| *** mfink has quit IRC | 08:11 | |
| *** eharney has quit IRC | 08:11 | |
| *** ociuhandu has quit IRC | 08:11 | |
| *** sahumada has quit IRC | 08:11 | |
| *** clarkb has quit IRC | 08:11 | |
| *** jpeeler has quit IRC | 08:11 | |
| *** kevinbenton has quit IRC | 08:11 | |
| *** yamahata_ has quit IRC | 08:11 | |
| *** HenryG has quit IRC | 08:11 | |
| *** marun has quit IRC | 08:11 | |
| *** KurtMartin has quit IRC | 08:11 | |
| *** mestery has quit IRC | 08:11 | |
| *** chandan_kumar has quit IRC | 08:11 | |
| *** dstanek has quit IRC | 08:11 | |
| *** bogdando has quit IRC | 08:11 | |
| *** koolhead17 has quit IRC | 08:11 | |
| *** viktors has quit IRC | 08:11 | |
| *** beagles has quit IRC | 08:11 | |
| *** dstufft has quit IRC | 08:11 | |
| *** flaper87 has quit IRC | 08:11 | |
| *** shortstop has quit IRC | 08:11 | |
| *** mattymo has quit IRC | 08:11 | |
| *** SpamapS has quit IRC | 08:11 | |
| *** cp16net has quit IRC | 08:11 | |
| *** locke105 has quit IRC | 08:11 | |
| *** dhellmann_ has quit IRC | 08:11 | |
| *** dkehn has quit IRC | 08:11 | |
| *** kiall has quit IRC | 08:11 | |
| *** sdake has quit IRC | 08:11 | |
| *** moted has quit IRC | 08:11 | |
| *** CaptTofu has quit IRC | 08:11 | |
| *** nati_ueno has quit IRC | 08:11 | |
| *** cody-somerville has quit IRC | 08:11 | |
| *** dkliban has quit IRC | 08:11 | |
| *** tianst20 has quit IRC | 08:11 | |
| *** yamahata__ has quit IRC | 08:11 | |
| *** mrodden has quit IRC | 08:11 | |
| *** yongli has quit IRC | 08:11 | |
| *** sabari has quit IRC | 08:11 | |
| *** bada has quit IRC | 08:11 | |
| *** bradm has quit IRC | 08:11 | |
| *** proffalken has quit IRC | 08:11 | |
| *** krtaylor has quit IRC | 08:11 | |
| *** dripton has quit IRC | 08:11 | |
| *** esmute has quit IRC | 08:11 | |
| *** bknudson has quit IRC | 08:11 | |
| *** juice has quit IRC | 08:11 | |
| *** sileht has quit IRC | 08:11 | |
| *** dolphm has quit IRC | 08:11 | |
| *** persia has quit IRC | 08:11 | |
| *** wchrisj_ has quit IRC | 08:11 | |
| *** akscram has quit IRC | 08:11 | |
| *** jamielennox has quit IRC | 08:11 | |
| *** sbadia has quit IRC | 08:11 | |
| *** echohead_ has quit IRC | 08:11 | |
| *** cyeoh has quit IRC | 08:11 | |
| *** mkerrin has quit IRC | 08:11 | |
| *** dachary has quit IRC | 08:11 | |
| *** simonmcc has quit IRC | 08:11 | |
| *** Alex_Gaynor has quit IRC | 08:11 | |
| *** AaronGr has quit IRC | 08:11 | |
| *** dklyle has quit IRC | 08:11 | |
| *** oubiwann has quit IRC | 08:11 | |
| *** spredzy has quit IRC | 08:11 | |
| *** mtreinish has quit IRC | 08:11 | |
| *** gaelL has quit IRC | 08:11 | |
| *** sdague has quit IRC | 08:11 | |
| *** markmcclain1 has quit IRC | 08:11 | |
| *** jhesketh__ has quit IRC | 08:11 | |
| *** lcestari has quit IRC | 08:11 | |
| *** jraim has quit IRC | 08:11 | |
| *** jgrimm has quit IRC | 08:11 | |
| *** miqui has quit IRC | 08:11 | |
| *** saschpe has quit IRC | 08:11 | |
| *** annegentle has quit IRC | 08:11 | |
| *** dragondm has quit IRC | 08:11 | |
| *** tristanC has quit IRC | 08:11 | |
| *** bookwar has quit IRC | 08:11 | |
| *** skraynev has quit IRC | 08:11 | |
| *** rainya has quit IRC | 08:11 | |
| *** zhiyan has quit IRC | 08:11 | |
| *** wayneeseguin has quit IRC | 08:11 | |
| *** mrda_away has quit IRC | 08:11 | |
| *** vipul has quit IRC | 08:11 | |
| *** dansmith has quit IRC | 08:11 | |
| *** davidlenwell has quit IRC | 08:11 | |
| *** enikanorov has quit IRC | 08:11 | |
| *** EntropyWorks has quit IRC | 08:11 | |
| *** loquacities has quit IRC | 08:11 | |
| *** jeblair has quit IRC | 08:11 | |
| *** Mithrandir has quit IRC | 08:11 | |
| *** jd__ has quit IRC | 08:11 | |
| *** amrith has quit IRC | 08:11 | |
| *** jgriffith has quit IRC | 08:11 | |
| *** katyafervent has quit IRC | 08:11 | |
| *** nijaba has quit IRC | 08:11 | |
| *** pabelanger has quit IRC | 08:11 | |
| *** greghaynes has quit IRC | 08:11 | |
| *** mancdaz has quit IRC | 08:11 | |
| *** amotoki has quit IRC | 08:11 | |
| *** Hefeweizen has quit IRC | 08:11 | |
| *** NikitaKonovalov has quit IRC | 08:11 | |
| *** jaypipes has quit IRC | 08:11 | |
| *** salv-orlando has quit IRC | 08:11 | |
| *** michchap has quit IRC | 08:11 | |
| *** alex-out has quit IRC | 08:11 | |
| *** westmaas has quit IRC | 08:11 | |
| *** kashyap has quit IRC | 08:11 | |
| *** medieval1 has quit IRC | 08:11 | |
| *** gema has quit IRC | 08:11 | |
| *** DuncanT- has quit IRC | 08:11 | |
| *** dangers_away has quit IRC | 08:11 | |
| *** SergeyLukjanov has quit IRC | 08:11 | |
| *** ruhe has quit IRC | 08:11 | |
| *** DinaBelova has quit IRC | 08:11 | |
| *** sdake_ has quit IRC | 08:11 | |
| *** ogelbukh has quit IRC | 08:11 | |
| *** jasond` has quit IRC | 08:11 | |
| *** gothicmindfood has quit IRC | 08:11 | |
| *** notmyname has quit IRC | 08:11 | |
| *** pleia2 has quit IRC | 08:11 | |
| *** comstud has quit IRC | 08:11 | |
| *** Guest6432 has quit IRC | 08:11 | |
| *** niska has quit IRC | 08:11 | |
| *** Ng has quit IRC | 08:11 | |
| *** lyxus has quit IRC | 08:11 | |
| *** branen has quit IRC | 08:11 | |
| *** dkehn_ has quit IRC | 08:11 | |
| *** samalba has quit IRC | 08:11 | |
| *** geekinutah has quit IRC | 08:11 | |
| *** rahmu has quit IRC | 08:11 | |
| *** nibalizer has quit IRC | 08:11 | |
| *** plomakin_ has quit IRC | 08:11 | |
| *** rdwrer has quit IRC | 08:11 | |
| *** dosaboy has quit IRC | 08:11 | |
| *** vladan has quit IRC | 08:11 | |
| *** spiffxp_ has quit IRC | 08:11 | |
| *** jcooley has quit IRC | 08:11 | |
| *** belliott has quit IRC | 08:11 | |
| *** adarazs_ has quit IRC | 08:11 | |
| *** jogo has quit IRC | 08:11 | |
| *** guitarzan has quit IRC | 08:11 | |
| *** mburned has quit IRC | 08:11 | |
| *** aude has quit IRC | 08:11 | |
| *** Steap has quit IRC | 08:11 | |
| *** jeremyb has quit IRC | 08:11 | |
| *** mugsie has quit IRC | 08:11 | |
| *** gilliard has quit IRC | 08:11 | |
| *** lifeless has quit IRC | 08:11 | |
| *** mika has quit IRC | 08:11 | |
| *** jesusaurus has quit IRC | 08:11 | |
| *** primeministerp has quit IRC | 08:11 | |
| *** gpocentek has quit IRC | 08:11 | |
| *** GheRivero has quit IRC | 08:11 | |
| *** BobBall has quit IRC | 08:11 | |
| *** mordred has quit IRC | 08:11 | |
| *** ihrachys|wfh has quit IRC | 08:11 | |
| *** dtroyer has quit IRC | 08:11 | |
| *** Daviey has quit IRC | 08:11 | |
| *** mkoderer has quit IRC | 08:11 | |
| *** tteggel has quit IRC | 08:11 | |
| *** boris-42 has quit IRC | 08:11 | |
| *** Adri2000 has quit IRC | 08:11 | |
| *** stevebaker has quit IRC | 08:11 | |
| *** StevenK has quit IRC | 08:11 | |
| *** EmilienM has quit IRC | 08:11 | |
| *** chmouel has quit IRC | 08:11 | |
| *** fungi has quit IRC | 08:11 | |
| *** zigo has quit IRC | 08:11 | |
| *** pvo has quit IRC | 08:11 | |
| *** Guest34377 has quit IRC | 08:11 | |
| *** wendar has quit IRC | 08:11 | |
| *** jbryce has quit IRC | 08:11 | |
| *** russellb has quit IRC | 08:11 | |
| *** jlk has quit IRC | 08:11 | |
| *** adam_g has quit IRC | 08:11 | |
| *** ttx has quit IRC | 08:11 | |
| *** gondoi has quit IRC | 08:11 | |
| *** msuriar_ has quit IRC | 08:11 | |
| *** mikal has quit IRC | 08:11 | |
| *** rockstar has quit IRC | 08:11 | |
| *** jeckersb_gone has quit IRC | 08:11 | |
| *** hughsaunders has quit IRC | 08:11 | |
| *** isviridov has quit IRC | 08:11 | |
| *** fbo_away has quit IRC | 08:11 | |
| *** ChanServ has quit IRC | 08:11 | |
| *** morganfainberg has quit IRC | 08:11 | |
| *** ildikov_ has quit IRC | 08:11 | |
| *** Guest58153 has quit IRC | 08:11 | |
| *** jerryz has quit IRC | 08:11 | |
| *** ekarlso has quit IRC | 08:11 | |
| *** Hunner has quit IRC | 08:11 | |
| *** bodepd has quit IRC | 08:11 | |
| *** zaro has quit IRC | 08:11 | |
| *** psedlak has quit IRC | 08:11 | |
| *** bnemec has quit IRC | 08:11 | |
| *** therve has quit IRC | 08:11 | |
| *** tian has quit IRC | 08:11 | |
| *** anteaya has quit IRC | 08:11 | |
| *** tru_tru has quit IRC | 08:11 | |
| *** tchaypo has quit IRC | 08:11 | |
| *** portante has quit IRC | 08:11 | |
| *** hdd has quit IRC | 08:11 | |
| *** hub_cap has quit IRC | 08:11 | |
| *** phschwartz has quit IRC | 08:11 | |
| *** med_ has quit IRC | 08:11 | |
| *** jamespage has quit IRC | 08:11 | |
| *** shardy has quit IRC | 08:11 | |
| *** NobodyCam has quit IRC | 08:11 | |
| *** devananda has quit IRC | 08:11 | |
| *** SlickNik has quit IRC | 08:11 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 08:11 | |
| *** sweston has joined #openstack-infra | 08:29 | |
| *** david-lyle has joined #openstack-infra | 08:29 | |
| *** mgagne has joined #openstack-infra | 08:29 | |
| *** dcramer__ has joined #openstack-infra | 08:29 | |
| *** mwagner_lap has joined #openstack-infra | 08:29 | |
| *** nosnos has joined #openstack-infra | 08:29 | |
| *** homeless has joined #openstack-infra | 08:29 | |
| *** soren has joined #openstack-infra | 08:29 | |
| *** asadoughi has joined #openstack-infra | 08:29 | |
| *** mattoliverau has joined #openstack-infra | 08:29 | |
| *** talluri has joined #openstack-infra | 08:29 | |
| *** sweston has quit IRC | 08:29 | |
| *** david-lyle has quit IRC | 08:29 | |
| *** mgagne has quit IRC | 08:29 | |
| *** dcramer__ has quit IRC | 08:29 | |
| *** mwagner_lap has quit IRC | 08:29 | |
| *** nosnos has quit IRC | 08:29 | |
| *** homeless has quit IRC | 08:29 | |
| *** soren has quit IRC | 08:29 | |
| *** jlibosva has joined #openstack-infra | 08:30 | |
| *** talluri has quit IRC | 08:49 | |
| *** talluri has joined #openstack-infra | 08:50 | |
| *** markmc has joined #openstack-infra | 09:02 | |
| *** jp_at_hp has joined #openstack-infra | 09:02 | |
| *** dizquierdo has joined #openstack-infra | 09:02 | |
| *** jgallard has joined #openstack-infra | 09:02 | |
| *** basha has joined #openstack-infra | 09:02 | |
| *** openstackgerrit has joined #openstack-infra | 09:02 | |
| *** gmoro has joined #openstack-infra | 09:02 | |
| *** antonym has joined #openstack-infra | 09:02 | |
| *** mwagner_lap has joined #openstack-infra | 09:02 | |
| *** dcramer__ has joined #openstack-infra | 09:02 | |
| *** mkoderer has joined #openstack-infra | 09:02 | |
| *** e0ne_ has joined #openstack-infra | 09:02 | |
| *** maurosr has joined #openstack-infra | 09:02 | |
| *** nosnos_ has joined #openstack-infra | 09:02 | |
| *** sweston has joined #openstack-infra | 09:02 | |
| *** alaski_ has joined #openstack-infra | 09:02 | |
| *** luisg has joined #openstack-infra | 09:02 | |
| *** soren has joined #openstack-infra | 09:02 | |
| *** ilyashakhat_ has joined #openstack-infra | 09:02 | |
| *** rlandy has joined #openstack-infra | 09:02 | |
| *** homeless_ has joined #openstack-infra | 09:02 | |
| *** david_lyle_ has joined #openstack-infra | 09:02 | |
| *** nati_ueno has joined #openstack-infra | 09:02 | |
| *** ildikov_ has joined #openstack-infra | 09:02 | |
| *** afazekas has joined #openstack-infra | 09:02 | |
| *** yolanda_ has joined #openstack-infra | 09:02 | |
| *** saju_m has joined #openstack-infra | 09:02 | |
| *** denis_makogon has joined #openstack-infra | 09:02 | |
| *** unicell has joined #openstack-infra | 09:02 | |
| *** amotoki has joined #openstack-infra | 09:02 | |
| *** markmcclain1 has joined #openstack-infra | 09:02 | |
| *** chandan_kumar has joined #openstack-infra | 09:02 | |
| *** cody-somerville has joined #openstack-infra | 09:02 | |
| *** Ryan_Lane has joined #openstack-infra | 09:02 | |
| *** Shrews has joined #openstack-infra | 09:02 | |
| *** Guest58153 has joined #openstack-infra | 09:02 | |
| *** dkliban has joined #openstack-infra | 09:02 | |
| *** tianst20 has joined #openstack-infra | 09:02 | |
| *** yamahata__ has joined #openstack-infra | 09:02 | |
| *** mrodden has joined #openstack-infra | 09:02 | |
| *** yongli has joined #openstack-infra | 09:02 | |
| *** Hefeweizen has joined #openstack-infra | 09:02 | |
| *** dstanek has joined #openstack-infra | 09:02 | |
| *** alex-out has joined #openstack-infra | 09:02 | |
| *** cadenzajon_ has joined #openstack-infra | 09:02 | |
| *** jhesketh__ has joined #openstack-infra | 09:02 | |
| *** NikitaKonovalov has joined #openstack-infra | 09:02 | |
| *** mfink has joined #openstack-infra | 09:02 | |
| *** jaypipes has joined #openstack-infra | 09:02 | |
| *** lcestari has joined #openstack-infra | 09:02 | |
| *** echohead_ has joined #openstack-infra | 09:02 | |
| *** eharney has joined #openstack-infra | 09:02 | |
| *** gondoi has joined #openstack-infra | 09:02 | |
| *** bada has joined #openstack-infra | 09:02 | |
| *** ociuhandu has joined #openstack-infra | 09:02 | |
| *** che-arne has joined #openstack-infra | 09:02 | |
| *** bradm has joined #openstack-infra | 09:02 | |
| *** jraim has joined #openstack-infra | 09:02 | |
| *** jgrimm has joined #openstack-infra | 09:02 | |
| *** annegentle has joined #openstack-infra | 09:02 | |
| *** proffalken has joined #openstack-infra | 09:02 | |
| *** krtaylor has joined #openstack-infra | 09:02 | |
| *** ekarlso has joined #openstack-infra | 09:02 | |
| *** miqui has joined #openstack-infra | 09:02 | |
| *** zhiyan has joined #openstack-infra | 09:02 | |
| *** saschpe has joined #openstack-infra | 09:02 | |
| *** salv-orlando has joined #openstack-infra | 09:02 | |
| *** jd__ has joined #openstack-infra | 09:02 | |
| *** gilliard has joined #openstack-infra | 09:02 | |
| *** bogdando has joined #openstack-infra | 09:02 | |
| *** michchap has joined #openstack-infra | 09:02 | |
| *** sahumada has joined #openstack-infra | 09:02 | |
| *** dripton has joined #openstack-infra | 09:02 | |
| *** esmute has joined #openstack-infra | 09:02 | |
| *** beagles has joined #openstack-infra | 09:02 | |
| *** rainya has joined #openstack-infra | 09:02 | |
| *** jerryz has joined #openstack-infra | 09:02 | |
| *** bknudson has joined #openstack-infra | 09:02 | |
| *** clarkb has joined #openstack-infra | 09:02 | |
| *** jpeeler has joined #openstack-infra | 09:02 | |
| *** koolhead17 has joined #openstack-infra | 09:02 | |
| *** tteggel has joined #openstack-infra | 09:02 | |
| *** dragondm has joined #openstack-infra | 09:02 | |
| *** westmaas has joined #openstack-infra | 09:02 | |
| *** kevinbenton has joined #openstack-infra | 09:02 | |
| *** juice has joined #openstack-infra | 09:02 | |
| *** tristanC has joined #openstack-infra | 09:02 | |
| *** bookwar has joined #openstack-infra | 09:02 | |
| *** kashyap has joined #openstack-infra | 09:02 | |
| *** skraynev has joined #openstack-infra | 09:02 | |
| *** yamahata_ has joined #openstack-infra | 09:02 | |
| *** HenryG has joined #openstack-infra | 09:02 | |
| *** medieval1 has joined #openstack-infra | 09:02 | |
| *** sileht has joined #openstack-infra | 09:02 | |
| *** gema has joined #openstack-infra | 09:02 | |
| *** marun has joined #openstack-infra | 09:02 | |
| *** DuncanT- has joined #openstack-infra | 09:02 | |
| *** dangers_away has joined #openstack-infra | 09:02 | |
| *** KurtMartin has joined #openstack-infra | 09:02 | |
| *** boris-42 has joined #openstack-infra | 09:02 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:02 | |
| *** ruhe has joined #openstack-infra | 09:02 | |
| *** DinaBelova has joined #openstack-infra | 09:02 | |
| *** mestery has joined #openstack-infra | 09:02 | |
| *** sdake_ has joined #openstack-infra | 09:02 | |
| *** Adri2000 has joined #openstack-infra | 09:02 | |
| *** Hunner has joined #openstack-infra | 09:02 | |
| *** dolphm has joined #openstack-infra | 09:02 | |
| *** ogelbukh has joined #openstack-infra | 09:02 | |
| *** bodepd has joined #openstack-infra | 09:02 | |
| *** zaro has joined #openstack-infra | 09:02 | |
| *** jasond` has joined #openstack-infra | 09:02 | |
| *** gothicmindfood has joined #openstack-infra | 09:02 | |
| *** notmyname has joined #openstack-infra | 09:02 | |
| *** pleia2 has joined #openstack-infra | 09:02 | |
| *** psedlak has joined #openstack-infra | 09:02 | |
| *** comstud has joined #openstack-infra | 09:02 | |
| *** Guest6432 has joined #openstack-infra | 09:02 | |
| *** niska has joined #openstack-infra | 09:02 | |
| *** Ng has joined #openstack-infra | 09:02 | |
| *** lyxus has joined #openstack-infra | 09:02 | |
| *** branen has joined #openstack-infra | 09:02 | |
| *** dkehn_ has joined #openstack-infra | 09:02 | |
| *** samalba has joined #openstack-infra | 09:02 | |
| *** geekinutah has joined #openstack-infra | 09:02 | |
| *** bnemec has joined #openstack-infra | 09:02 | |
| *** wayneeseguin has joined #openstack-infra | 09:02 | |
| *** mrda_away has joined #openstack-infra | 09:02 | |
| *** persia has joined #openstack-infra | 09:02 | |
| *** vipul has joined #openstack-infra | 09:02 | |
| *** dansmith has joined #openstack-infra | 09:02 | |
| *** davidlenwell has joined #openstack-infra | 09:02 | |
| *** therve has joined #openstack-infra | 09:02 | |
| *** tian has joined #openstack-infra | 09:02 | |
| *** dstufft has joined #openstack-infra | 09:02 | |
| *** flaper87 has joined #openstack-infra | 09:02 | |
| *** loquacities has joined #openstack-infra | 09:02 | |
| *** wchrisj_ has joined #openstack-infra | 09:02 | |
| *** shortstop has joined #openstack-infra | 09:02 | |
| *** jeblair has joined #openstack-infra | 09:02 | |
| *** mattymo has joined #openstack-infra | 09:02 | |
| *** rdwrer has joined #openstack-infra | 09:02 | |
| *** SpamapS has joined #openstack-infra | 09:02 | |
| *** cp16net has joined #openstack-infra | 09:02 | |
| *** phschwartz has joined #openstack-infra | 09:02 | |
| *** locke105 has joined #openstack-infra | 09:02 | |
| *** akscram has joined #openstack-infra | 09:02 | |
| *** jamielennox|away has joined #openstack-infra | 09:02 | |
| *** dhellmann_ has joined #openstack-infra | 09:02 | |
| *** Mithrandir has joined #openstack-infra | 09:02 | |
| *** sbadia has joined #openstack-infra | 09:02 | |
| *** amrith has joined #openstack-infra | 09:02 | |
| *** dkehn has joined #openstack-infra | 09:02 | |
| *** kiall has joined #openstack-infra | 09:02 | |
| *** enikanorov has joined #openstack-infra | 09:02 | |
| *** sdake has joined #openstack-infra | 09:02 | |
| *** moted has joined #openstack-infra | 09:02 | |
| *** stevebaker has joined #openstack-infra | 09:02 | |
| *** jgriffith has joined #openstack-infra | 09:02 | |
| *** katyafervent has joined #openstack-infra | 09:02 | |
| *** nijaba has joined #openstack-infra | 09:02 | |
| *** sdague has joined #openstack-infra | 09:02 | |
| *** mtreinish has joined #openstack-infra | 09:02 | |
| *** gaelL has joined #openstack-infra | 09:02 | |
| *** spredzy has joined #openstack-infra | 09:02 | |
| *** oubiwann has joined #openstack-infra | 09:02 | |
| *** dklyle has joined #openstack-infra | 09:02 | |
| *** AaronGr has joined #openstack-infra | 09:02 | |
| *** Alex_Gaynor has joined #openstack-infra | 09:02 | |
| *** simonmcc has joined #openstack-infra | 09:02 | |
| *** dachary has joined #openstack-infra | 09:02 | |
| *** mkerrin has joined #openstack-infra | 09:02 | |
| *** cyeoh has joined #openstack-infra | 09:02 | |
| *** EntropyWorks has joined #openstack-infra | 09:02 | |
| *** Steap has joined #openstack-infra | 09:02 | |
| *** mancdaz has joined #openstack-infra | 09:02 | |
| *** greghaynes has joined #openstack-infra | 09:02 | |
| *** pabelanger has joined #openstack-infra | 09:02 | |
| *** StevenK has joined #openstack-infra | 09:02 | |
| *** rahmu has joined #openstack-infra | 09:02 | |
| *** nibalizer has joined #openstack-infra | 09:02 | |
| *** plomakin_ has joined #openstack-infra | 09:02 | |
| *** dosaboy has joined #openstack-infra | 09:02 | |
| *** tchaypo has joined #openstack-infra | 09:02 | |
| *** EmilienM has joined #openstack-infra | 09:02 | |
| *** vladan has joined #openstack-infra | 09:02 | |
| *** spiffxp_ has joined #openstack-infra | 09:02 | |
| *** belliott has joined #openstack-infra | 09:02 | |
| *** jcooley has joined #openstack-infra | 09:02 | |
| *** adarazs_ has joined #openstack-infra | 09:02 | |
| *** jogo has joined #openstack-infra | 09:02 | |
| *** guitarzan has joined #openstack-infra | 09:02 | |
| *** mburned has joined #openstack-infra | 09:02 | |
| *** anteaya has joined #openstack-infra | 09:02 | |
| *** tru_tru has joined #openstack-infra | 09:02 | |
| *** aude has joined #openstack-infra | 09:02 | |
| *** portante has joined #openstack-infra | 09:02 | |
| *** jeremyb has joined #openstack-infra | 09:02 | |
| *** mugsie has joined #openstack-infra | 09:02 | |
| *** lifeless has joined #openstack-infra | 09:02 | |
| *** mika has joined #openstack-infra | 09:02 | |
| *** jesusaurus has joined #openstack-infra | 09:02 | |
| *** primeministerp has joined #openstack-infra | 09:02 | |
| *** gpocentek has joined #openstack-infra | 09:02 | |
| *** GheRivero has joined #openstack-infra | 09:02 | |
| *** BobBall has joined #openstack-infra | 09:02 | |
| *** mordred has joined #openstack-infra | 09:02 | |
| *** ihrachys|wfh has joined #openstack-infra | 09:02 | |
| *** Daviey has joined #openstack-infra | 09:02 | |
| *** dtroyer has joined #openstack-infra | 09:02 | |
| *** hdd has joined #openstack-infra | 09:02 | |
| *** hub_cap has joined #openstack-infra | 09:02 | |
| *** chmouel has joined #openstack-infra | 09:02 | |
| *** fungi has joined #openstack-infra | 09:02 | |
| *** zigo has joined #openstack-infra | 09:02 | |
| *** pvo has joined #openstack-infra | 09:02 | |
| *** Guest34377 has joined #openstack-infra | 09:02 | |
| *** wendar has joined #openstack-infra | 09:02 | |
| *** jbryce has joined #openstack-infra | 09:02 | |
| *** russellb has joined #openstack-infra | 09:02 | |
| *** jlk has joined #openstack-infra | 09:02 | |
| *** adam_g has joined #openstack-infra | 09:02 | |
| *** ttx has joined #openstack-infra | 09:02 | |
| *** med_ has joined #openstack-infra | 09:02 | |
| *** jamespage has joined #openstack-infra | 09:02 | |
| *** msuriar_ has joined #openstack-infra | 09:02 | |
| *** mikal has joined #openstack-infra | 09:02 | |
| *** rockstar has joined #openstack-infra | 09:02 | |
| *** jeckersb_gone has joined #openstack-infra | 09:02 | |
| *** hughsaunders has joined #openstack-infra | 09:02 | |
| *** morganfainberg has joined #openstack-infra | 09:02 | |
| *** isviridov has joined #openstack-infra | 09:02 | |
| *** fbo_away has joined #openstack-infra | 09:02 | |
| *** ChanServ has joined #openstack-infra | 09:02 | |
| *** shardy has joined #openstack-infra | 09:02 | |
| *** NobodyCam has joined #openstack-infra | 09:02 | |
| *** devananda has joined #openstack-infra | 09:02 | |
| *** SlickNik has joined #openstack-infra | 09:02 | |
| *** dickson.freenode.net sets mode: +o ChanServ | 09:02 | |
| *** soren has quit IRC | 09:02 | |
| *** soren has joined #openstack-infra | 09:02 | |
| *** SumitNaiksatam has quit IRC | 09:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 09:02 | |
| *** jlibosva has quit IRC | 09:02 | |
| *** jlibosva has joined #openstack-infra | 09:02 | |
| *** zhiwei has joined #openstack-infra | 09:02 | |
| *** mgagne1 has quit IRC | 09:02 | |
| *** mgagne1 has joined #openstack-infra | 09:02 | |
| zhiwei | clarkb: hi | 09:02 |
| zhiwei | fungi: hi | 09:02 |
| *** fbo_away is now known as fbo | 09:05 | |
| *** jpich has joined #openstack-infra | 09:05 | |
| *** talluri has quit IRC | 09:12 | |
| *** talluri has joined #openstack-infra | 09:12 | |
| *** 16WAAZIB1 has joined #openstack-infra | 09:14 | |
| *** rossella_s has joined #openstack-infra | 09:14 | |
| kashyap | zhiwei, Just post your message, they'll pick it up when they're awake. They maybe in a different timezone. | 09:14 |
| *** rlandy_ has joined #openstack-infra | 09:14 | |
| zhiwei | oh, thanks. | 09:14 |
| *** cody-somerville has quit IRC | 09:14 | |
| *** rlandy has quit IRC | 09:14 | |
| *** 16WAAZIB1 has quit IRC | 09:14 | |
| *** rossella_s has quit IRC | 09:15 | |
| *** rossella_s has joined #openstack-infra | 09:15 | |
| *** yassine has joined #openstack-infra | 09:16 | |
| *** rossella_s has quit IRC | 09:18 | |
| *** markmc has quit IRC | 09:19 | |
| *** saju_m has quit IRC | 09:19 | |
| *** rossella-s has joined #openstack-infra | 09:19 | |
| *** andreaf has joined #openstack-infra | 09:20 | |
| *** Guest58153 has quit IRC | 09:28 | |
| *** dizquierdo has quit IRC | 09:32 | |
| openstackgerrit | Akihiro Motoki proposed a change to openstack-infra/config: Job to push Horizon translation to Transifex https://review.openstack.org/68042 | 09:34 |
| *** saju_m has joined #openstack-infra | 09:34 | |
| *** johnthetubaguy has joined #openstack-infra | 09:37 | |
| *** ociuhandu has quit IRC | 09:39 | |
| *** alexpilotti has joined #openstack-infra | 09:41 | |
| *** markmcclain1 has quit IRC | 09:43 | |
| *** jooools has joined #openstack-infra | 09:44 | |
| *** alexpilotti has quit IRC | 09:45 | |
| *** Aleksey has joined #openstack-infra | 09:47 | |
| *** denis_makogon has quit IRC | 09:48 | |
| *** rossella-s has quit IRC | 09:48 | |
| ttx | hrrm, looks like we have routine resets at top of gate, every time I look we added another hour to the top of gate process time | 09:50 |
| *** oubiwan__ has joined #openstack-infra | 09:50 | |
| *** rossella-s has joined #openstack-infra | 09:51 | |
| *** hashar has joined #openstack-infra | 09:51 | |
| *** zhiwei has quit IRC | 09:52 | |
| *** oubiwan__ has quit IRC | 09:54 | |
| *** andre__ has joined #openstack-infra | 09:57 | |
| *** rlandy_ has quit IRC | 10:05 | |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| *** dstanek has quit IRC | 10:08 | |
| *** rossella_s has joined #openstack-infra | 10:09 | |
| *** rossella_s has quit IRC | 10:10 | |
| *** ihrachys|wfh is now known as ihrachys | 10:15 | |
| *** tnurlygayanov has joined #openstack-infra | 10:16 | |
| tnurlygayanov | Hi there :) | 10:16 |
| tnurlygayanov | i have a small question about the global-requirements gates for all projects | 10:17 |
| *** rlandy_ has joined #openstack-infra | 10:17 | |
| tnurlygayanov | We have some problems with global requirements now and we plan to fix it, but now we can see that this gate failed for any commits (this gate is 'no voiting') | 10:18 |
| *** rossella-s has quit IRC | 10:19 | |
| tnurlygayanov | and we want to have the ability to ignore some known issues with requirements and check new changes in requirements.txt for our repositories | 10:21 |
| *** rossella-s has joined #openstack-infra | 10:22 | |
| *** bada has quit IRC | 10:22 | |
| tnurlygayanov | we have custom scripts for this, but, probably we can change infra-script for gates jobs to make this script omre intellectual | 10:22 |
| *** alexpilotti has joined #openstack-infra | 10:24 | |
| tnurlygayanov | s/omre/more | 10:24 |
| *** rlandy_ is now known as rlandy | 10:29 | |
| *** jhesketh_ has quit IRC | 10:31 | |
| *** jhesketh__ has quit IRC | 10:31 | |
| *** mattymo has quit IRC | 10:32 | |
| *** mattymo has joined #openstack-infra | 10:32 | |
| *** katyafervent is now known as katyafervent_awa | 10:33 | |
| *** rlandy has quit IRC | 10:33 | |
| *** rlandy has joined #openstack-infra | 10:37 | |
| *** sarob_ has joined #openstack-infra | 10:40 | |
| sdague | tnurlygayanov: what are your problems with global requirements? | 10:42 |
| sdague | like what are these known issues | 10:42 |
| sdague | ttx: zuul has a bug | 10:43 |
| sdague | where it requires the git mirroring to complete before a job is considered merged | 10:43 |
| sdague | which can time out under load | 10:43 |
| ttx | sdague: ah, was suspecting something weird was going on but couldn't find explanation in backlog | 10:43 |
| sdague | I'd been seeing it for a while, fungi finally figured out what it was yesterday | 10:44 |
| sdague | or monday | 10:44 |
| sdague | jeblair: said it's caused 4 - 7 resets a day over the last couple of days | 10:44 |
| ttx | sdague: i think that stat just went up | 10:44 |
| *** sarob_ has quit IRC | 10:44 | |
| sdague | but they didn't want to risk the fix until after the milestone | 10:44 |
| sdague | yeh, definitely could be | 10:44 |
| ttx | 13h49 at top of gate right now | 10:45 |
| sdague | there are some real fails in the resets as well | 10:45 |
| ttx | let's see if the pile of successes at top of gate right now makes it | 10:45 |
| sdague | but yeh, I think it might be the more overwhelming reason for delay at the moment | 10:46 |
| sdague | honestly, I only caught the behavior by staring at zuul | 10:46 |
| ttx | yes, been staring all morning | 10:46 |
| ttx | trying to get WTF was going on | 10:47 |
| sdague | yeh, there is some scrollback here about it | 10:47 |
| sdague | from a couple of days ago | 10:47 |
| ttx | sdague: btw to facilitate your task next week, been collecting FFEs and expectations on an etherpad | 10:47 |
| sdague | great | 10:47 |
| ttx | https://etherpad.openstack.org/p/jpVIMKKxqb | 10:48 |
| ttx | sdague: since anyone can add to rc1 milestone it lets us keep track of..; suspicious additions | 10:48 |
| sdague | I see yuo went with the incredibly easy to remember url | 10:49 |
| ttx | it has VIM in it : | 10:49 |
| ttx | ! | 10:49 |
| ttx | ok ok, will dump it somewhere else | 10:50 |
| ttx | https://etherpad.openstack.org/p/icehouse-FFEs | 10:50 |
| *** ociuhandu has quit IRC | 10:51 | |
| ttx | sdague: only Heat was discussed with PTL at this point | 10:51 |
| sdague | crap, we just reset again | 10:51 |
| sdague | on this issue | 10:51 |
| sdague | nova changes seem to be more suseptible to it | 10:51 |
| ttx | yay, 14:25 at top of gate now | 10:52 |
| *** gokrokve has joined #openstack-infra | 10:52 | |
| *** rlandy_ has joined #openstack-infra | 10:53 | |
| ttx | sdague: well, nova git repo is the largest | 10:53 |
| sdague | yep | 10:53 |
| sdague | also our git servers are pretty loaded | 10:53 |
| sdague | every time we fix one scaling issue we move to a new one :) | 10:54 |
| *** gokrokve_ has joined #openstack-infra | 10:54 | |
| ttx | sdague: what happens to those fails ? Do they get reenqueued ? | 10:54 |
| ttx | that would explain why the queue remains deep | 10:54 |
| sdague | so actually, the thing the "fails" is the one that merged | 10:54 |
| sdague | but a bad return is given | 10:54 |
| sdague | so zuul thinks it failed | 10:54 |
| sdague | so restarts the queue | 10:55 |
| sdague | it didn't, it landed | 10:55 |
| ttx | hah. That explains what I've been seeing | 10:55 |
| sdague | yep | 10:55 |
| ttx | it's like, a perfectly working thing gets suddenly delayed | 10:55 |
| sdague | there is a fix waiting | 10:55 |
| sdague | yep | 10:55 |
| ttx | generally *after* a nova merge | 10:55 |
| *** rlandy has quit IRC | 10:55 | |
| sdague | and the whole stack is at a pretty high level of reliability right now | 10:56 |
| tnurlygayanov | sdague: now our pproject has some external requirements, which are not presented in global requirements | 10:56 |
| ttx | OK, looks like we'll have to wait a bit before I3 after all | 10:56 |
| openstackgerrit | Timur Nurlygayanov proposed a change to openstack/requirements: Added several pacakges to global requirements https://review.openstack.org/78158 | 10:56 |
| sdague | ttx: yeh. Honestly, it might be worth figuring out how disruptive the zuul restart would be when fungi gets up. | 10:56 |
| ttx | sdague: At least it's just slowing things down, not creating reverify despair | 10:57 |
| *** gokrokve has quit IRC | 10:57 | |
| sdague | ttx: no, definitely, in the class of fails, this is actually not bad | 10:57 |
| ttx | we are getting better at fails | 10:57 |
| *** katyafervent_awa is now known as katyafervent | 10:58 | |
| ttx | it's actually not failing at all. Just way slower than it should be | 10:58 |
| *** gokrokve_ has quit IRC | 10:59 | |
| openstackgerrit | Thomas Leaman proposed a change to openstack/requirements: Move to newer version of python-swiftclient https://review.openstack.org/76170 | 11:04 |
| sdague | ttx: yeh, well we got a ton of eyes on real bugs in January | 11:04 |
| sdague | after the melt down | 11:04 |
| sdague | and we've been actually not terrible since then | 11:04 |
| *** zhiwei has joined #openstack-infra | 11:07 | |
| sdague | we even dealt with an upstream library breaking use twice yesterday, and recovered :) | 11:07 |
| sdague | even though that upstream library was us | 11:07 |
| kashyap | I read on a thread where there's a plan to move form launchpad -> StoryBoard | 11:08 |
| kashyap | Is that only for Blueprints? Or for bugs too? | 11:08 |
| ttx | kashyap: for bugs too | 11:09 |
| ttx | but first it needs to exist. | 11:09 |
| kashyap | ttx, Noted. Does it have a CLI client? (Please say yes :-) ) | 11:09 |
| ttx | kashyap: it has a REST API, a webclient and will have a CLI/lib | 11:10 |
| kashyap | Thank you. (And presumably supports some modern Markup languages?) | 11:10 |
| kashyap | Coming by bugzilla land, one of the things I dearly miss with launchpad is bug query/manipulation from CLI | 11:11 |
| kashyap | Although inter-webs throw some references to certain CLI tools, etc. | 11:11 |
| *** unicell has quit IRC | 11:11 | |
| ttx | kashyap: well, there is a Cli and decent lib for LP | 11:11 |
| alexpilotti | hi guys, I'm trying to figure out the status of a patchset still the verify queue | 11:11 |
| *** jp_at_hp1 has joined #openstack-infra | 11:11 | |
| alexpilotti | but looking at http://status.openstack.org/zuul/ I get "Loading..." | 11:12 |
| zigo | sdague: I'm lost in the serries, and can't fix the "Depends on commit <sha256> which has no change associated with it." thing, what's the way? | 11:12 |
| *** jp_at_hp has quit IRC | 11:12 | |
| ttx | alexpilotti: the console support in horizon ? | 11:12 |
| kashyap | ttx, Is that what you're referring to? -- https://launchpad.net/lp-cli | 11:12 |
| alexpilotti | ttx yep | 11:12 |
| sdague | alexpilotti: it's loading fine here. | 11:12 |
| ttx | alexpilotti: #4 | 11:12 |
| ttx | exepcted ETA 33min | 11:12 |
| sdague | alexpilotti: what browser? | 11:12 |
| alexpilotti | sdague: chrome | 11:13 |
| sdague | I'd just try a reload | 11:13 |
| sdague | it might take a little bit to load the data | 11:13 |
| sdague | it's all js async | 11:13 |
| alexpilotti | sdague: safari came up ok | 11:13 |
| ttx | alexpilotti: waiting on it before cutting horizon i3 | 11:13 |
| alexpilotti | ttx: thanks! | 11:13 |
| sdague | zigo: this the migrate series? | 11:13 |
| ttx | alexpilotti: so unless it gets dumped at bottom of queue for some reason, it should be in | 11:14 |
| zigo | sdague: Yeah. | 11:14 |
| *** unicell has joined #openstack-infra | 11:14 | |
| sdague | zigo: on sec, a patch underneath maybe got rebased | 11:14 |
| zigo | sdague: Which is what I'm not able to fix, yes. :/ | 11:14 |
| alexpilotti | sdague: firefox is also ok, only chrome has this issue | 11:15 |
| sdague | ttx: barbican... so close - https://review.openstack.org/#/c/77647/ - but now they seem to have just accidentally disabled their tests | 11:15 |
| *** ociuhandu has joined #openstack-infra | 11:15 | |
| alexpilotti | sdague: let me try some housecleaning | 11:15 |
| zigo | sdague: Let me know what I'm doing wrong please! :) | 11:15 |
| ttx | kashyap: https://launchpad.net/lptools has the lp-shell command | 11:16 |
| alexpilotti | sdague: a cache cleaning fixed it on chrome as well, sorry for the false alarm | 11:17 |
| sdague | zigo: so in situations like this I tend to just grab at the top of the patch series and do the rebase from there. let me try to clean this up | 11:17 |
| kashyap | ttx, Thanks for looking up. (/me is on a slow USB internet.) | 11:18 |
| zigo | sdague: That's what I tried. :/ | 11:18 |
| *** unicell has quit IRC | 11:19 | |
| zigo | sdague: git-review insist into committing again "UniqueConstraint named and escaped twice" which is already in master, then fails on me. | 11:19 |
| sdague | yeh, so it looks like there was a base rebase issue, then the patches ended up reordered at some point. | 11:21 |
| sdague | zigo: so why did you do this - https://review.openstack.org/#/c/77480/2/migrate/changeset/ansisql.py | 11:21 |
| sdague | instead of a hasattr? | 11:21 |
| zigo | sdague: hasattr? | 11:21 |
| *** dizquierdo has joined #openstack-infra | 11:21 | |
| sdague | let me redo that patch, because I'm concerned on that one | 11:22 |
| zigo | sdague: Well, that's not the last version. | 11:22 |
| zigo | sdague: https://review.openstack.org/gitweb?p=stackforge%2Fsqlalchemy-migrate.git;a=commitdiff;h=d8b5dc7b9cf6087962239cdd244a24256872b852 | 11:22 |
| zigo | sdague: It's there in another patch as well. | 11:22 |
| zigo | sdague: Also, I'm concerned that there's other instances of "<obj>.quote" that aren't covered by the scope of the unit tests. | 11:23 |
| zigo | sdague: Is there a way to use python-coverage? | 11:23 |
| sdague | zigo: what's the last version? | 11:24 |
| jpich | ttx: There are a couple more horizon patches that were approved yesterday that haven't merged yet (e.g. https://review.openstack.org/#/c/72553/ , https://review.openstack.org/#/c/73433/ , https://review.openstack.org/#/c/71061/ ) | 11:24 |
| sdague | ok, let me push the fixed rebase I've got | 11:24 |
| sdague | then we can talk about that patch | 11:24 |
| zigo | Ok. | 11:24 |
| ttx | jpich: yes, waiting for those | 11:25 |
| *** chandan_kumar has quit IRC | 11:25 | |
| jpich | ttx: Ah, ok! Cool, thank you | 11:25 |
| *** jgallard has quit IRC | 11:28 | |
| sdague | zigo: so I still haven't managed to fully connect the trees, but I did push a fixed version of this the ansisql.py | 11:30 |
| sdague | I also think we *really* should not merge this code until we get an SQLA expert to look at it. We should poke jaypipes today to get his view on this. As I'm concerned just making the tests work could cause some security exposures here. | 11:31 |
| sdague | zigo: https://review.openstack.org/#/c/77480/5 | 11:32 |
| *** unicell has joined #openstack-infra | 11:33 | |
| *** CaptTofu has joined #openstack-infra | 11:35 | |
| sdague | also, it's bad form to +2 your own patches :P | 11:38 |
| sdague | I just fixed the queue so it's linear again | 11:38 |
| *** rossella-s has quit IRC | 11:39 | |
| *** unicell has quit IRC | 11:39 | |
| *** rossella-s has joined #openstack-infra | 11:40 | |
| *** rossella-s has quit IRC | 11:41 | |
| *** rossella-s has joined #openstack-infra | 11:42 | |
| anteaya | ttx sdague yes you figured it out, the rest is due to a repo confirmation which we don't need anymore and are just going to remove | 11:44 |
| *** fbo is now known as fbo_away | 11:44 | |
| anteaya | since the addition of zuul mergers make that repo check obsolete | 11:45 |
| anteaya | and it only happens on a nova patch, the nova patch will failed to be acknowledged as merged even though the tests pass and the gate resets | 11:45 |
| anteaya | and usually only happens when we have a long line of patches that can be merged in quick succession, | 11:46 |
| anteaya | at least those are the observed conditions | 11:46 |
| anteaya | s/rest/reset | 11:46 |
| *** dripton_ has joined #openstack-infra | 11:47 | |
| anteaya | the reason it is nova is due to the nova's size, at least that is the supposition | 11:47 |
| *** persia__ has joined #openstack-infra | 11:47 | |
| *** mtreinish_ has joined #openstack-infra | 11:48 | |
| *** gaelL_ has joined #openstack-infra | 11:48 | |
| *** dkliban_ has joined #openstack-infra | 11:50 | |
| *** simonmcc_ has joined #openstack-infra | 11:50 | |
| anteaya | yeah, openstackgerrit only acknowledged 3 merged patches to neutron while I slept | 11:52 |
| *** bada has joined #openstack-infra | 11:52 | |
| *** wchrisj__ has joined #openstack-infra | 11:52 | |
| *** dachary_ has joined #openstack-infra | 11:52 | |
| *** gokrokve has joined #openstack-infra | 11:52 | |
| zigo | sdague: I just tried to +2 my own patch because it didn't work, and wanted to test. :) | 11:52 |
| zigo | Thanks for fixing, though I'd really like to know what I should have done. | 11:52 |
| anteaya | zigo: what do you need to test with a +2 on a patch? | 11:53 |
| zigo | anteaya: Test if it would merge or not, and I was strongly guessing it wouldn't. | 11:53 |
| *** mkoderer_ has joined #openstack-infra | 11:54 | |
| zigo | anteaya: Do you plan on helping fixing -migrate for SQLA 0.9??? | 11:54 |
| *** alexpilotti_ has joined #openstack-infra | 11:54 | |
| *** Alex_Gaynor_ has joined #openstack-infra | 11:55 | |
| anteaya | do I plan on fixing -migrate? | 11:55 |
| *** gaelL has quit IRC | 11:55 | |
| *** mtreinish has quit IRC | 11:55 | |
| *** dripton has quit IRC | 11:55 | |
| *** persia has quit IRC | 11:55 | |
| *** akscram has quit IRC | 11:56 | |
| *** simonmcc has quit IRC | 11:56 | |
| *** dkliban has quit IRC | 11:56 | |
| *** wchrisj_ has quit IRC | 11:56 | |
| anteaya | it hadn't crossed my mind, no, mostly because I have never looked at the migrate code before | 11:56 |
| *** wchrisj__ is now known as wchrisj_ | 11:56 | |
| *** simonmcc_ is now known as simonmcc | 11:56 | |
| *** mtreinish_ is now known as mtreinish | 11:56 | |
| *** nati_ueno has quit IRC | 11:56 | |
| *** Alex_Gaynor has quit IRC | 11:56 | |
| *** gokrokve has quit IRC | 11:56 | |
| *** dachary has quit IRC | 11:56 | |
| *** antonym has quit IRC | 11:56 | |
| *** mkoderer has quit IRC | 11:56 | |
| *** sbadia has quit IRC | 11:56 | |
| *** dachary_ is now known as dachary | 11:56 | |
| *** alexpilotti has quit IRC | 11:56 | |
| *** esmute has quit IRC | 11:56 | |
| *** juice has quit IRC | 11:56 | |
| *** bradm has quit IRC | 11:57 | |
| *** spredzy has quit IRC | 11:57 | |
| *** juice has joined #openstack-infra | 11:57 | |
| *** alexpilotti_ is now known as alexpilotti | 11:57 | |
| *** mkoderer_ is now known as mkoderer | 11:57 | |
| *** Alex_Gaynor_ is now known as Alex_Gaynor | 11:57 | |
| anteaya | I think if you were to ask someone familiar with the code base, I would guess they would like working code, but I am also guessing you already know that | 11:57 |
| *** bradm1 has joined #openstack-infra | 11:57 | |
| *** sbadia has joined #openstack-infra | 11:57 | |
| *** esmute has joined #openstack-infra | 11:57 | |
| zigo | sdague: Arg, I think I destroyed it again, trying to fix #77396 | 11:58 |
| zigo | Please tell me the correct workflow!!! | 11:58 |
| *** antonym has joined #openstack-infra | 11:58 | |
| sdague | zigo: yeh, I'm reworking the series right now, including dealing the the hard tab | 11:58 |
| *** spredzy has joined #openstack-infra | 11:58 | |
| zigo | Ooops! :( | 11:58 |
| zigo | sdague: I'm not touching anything then. | 11:58 |
| sdague | no worries, I'll just push over top | 11:58 |
| *** rossella-s has quit IRC | 11:59 | |
| zigo | anteaya: FYI, nobody is really familliar with that code, OpenStack had to become the new upstream, because upstream stopped working on it. | 11:59 |
| zigo | anteaya: And we need to move forward with SQLA 0.9 ... | 11:59 |
| *** Ryan_Lane has quit IRC | 12:00 | |
| anteaya | zigo: sounds like you know more about it than myself | 12:01 |
| *** rossella-s has joined #openstack-infra | 12:01 | |
| anteaya | and thanks for sharing that information, which I did not know | 12:01 |
| *** Ryan_Lane has joined #openstack-infra | 12:02 | |
| *** rossella-s has quit IRC | 12:03 | |
| *** rossella-s has joined #openstack-infra | 12:03 | |
| sdague | zigo: yeh, we do need to make sure we do this right though and not rush it. Because a lot of people will be impacted if we don't. | 12:04 |
| sdague | so I colapsed the 2 quote changes into one | 12:05 |
| *** aysyanne has joined #openstack-infra | 12:06 | |
| sdague | zigo: so I think - https://review.openstack.org/#/c/77480/ is currently obsoleted by me merging those. So you should abandon that patch, it will make looking at the series clearer | 12:06 |
| *** ArxCruz has joined #openstack-infra | 12:08 | |
| *** CaptTofu has quit IRC | 12:09 | |
| *** CaptTofu has joined #openstack-infra | 12:09 | |
| *** basha has quit IRC | 12:10 | |
| *** chandan_kumar has joined #openstack-infra | 12:11 | |
| *** yamahata__ has quit IRC | 12:14 | |
| *** CaptTofu has quit IRC | 12:14 | |
| *** CaptTofu has joined #openstack-infra | 12:16 | |
| *** rossella_s has joined #openstack-infra | 12:21 | |
| *** rossella_s has quit IRC | 12:21 | |
| ruhe | sdague: re https://review.openstack.org/#/c/78158/ (new dependencies in global-requirements). if a package is not present in ubuntu archives, should we contribute it to ubuntu-cloud-archive or to the main package repository? | 12:23 |
| sdague | ruhe: both ubuntu and red hat really need to be willing to support the package (if it's not already in there) | 12:25 |
| sdague | so the big question is asking folks from there what their feeling is on the proposed package | 12:26 |
| ruhe | sdague: ok. thank you | 12:26 |
| sdague | but even then, if the package isn't being regularly maintained, it's off the list. Because we can't add new dependencies that aren't python3 compatible | 12:27 |
| sdague | so like I said, that throws out 3 of 4 in that list | 12:27 |
| zigo | sdague: I agree about the "not rushing", though I have 20 Debian packages waiting for -migrate to get fixed. :/ | 12:28 |
| sdague | zigo: that's fine, but we're fundamentally touching sql quoting here | 12:28 |
| sdague | which is *exactly* how most sql exploits happen | 12:29 |
| sdague | so care is really warrented | 12:29 |
| zigo | sdague: That's not really runtime though, is it? I mean, migrations would happen only at setup time... | 12:29 |
| sdague | some of the migrations end up as data migrations, and actually manage internal data | 12:30 |
| sdague | but you are right, the surface is less dangerous | 12:30 |
| anteaya | yay 3 merged | 12:30 |
| sdague | especially as it will impact upgrade | 12:30 |
| *** salv-orlando has quit IRC | 12:31 | |
| ruhe | sdague: we're going to get rid of all 3 of them. but we have our own library used internally. in case if we make it well-documented, python3-compatible, available on pypi, do you think we'll be able to submit DEB and RPM specs to Ubuntu and Fedore respectively? Would it be enough to add this dependency to global-requirements? | 12:31 |
| sdague | ruhe: I think those projects would need to decide | 12:31 |
| sdague | I think starting with publishing to pypi is a good direction though | 12:32 |
| sdague | especially if it's reusable | 12:32 |
| sdague | however, you will get push back if the functionality could be handled with existing requirements. So make sure you've ruled those out first | 12:33 |
| ruhe | sdague: i've got a direction, thank you very much :) | 12:33 |
| * zigo just understood what "hasattr" does reading the patch | 12:34 | |
| *** sarob_ has joined #openstack-infra | 12:41 | |
| *** rossella-s has quit IRC | 12:41 | |
| *** pcm_ has joined #openstack-infra | 12:43 | |
| *** mbacchi has joined #openstack-infra | 12:43 | |
| ttx | sdague: new reset | 12:43 |
| ttx | after an horizon merge this time | 12:44 |
| *** rfolco has joined #openstack-infra | 12:44 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Make token storage configurable https://review.openstack.org/78188 | 12:45 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth Token Middleware https://review.openstack.org/74735 | 12:45 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 12:45 |
| *** lcostantino has joined #openstack-infra | 12:45 | |
| sdague | zigo: oh, it looks like I did the logic backwards | 12:46 |
| *** sarob_ has quit IRC | 12:46 | |
| *** hashar has quit IRC | 12:46 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Load superusers from a yaml file https://review.openstack.org/77884 | 12:46 |
| anteaya | ttx I think a horizon patch just failed, was it one of the ones you were waiting on? | 12:49 |
| anteaya | ttx did that horizon patch merge? | 12:49 |
| ttx | anteaya: I suspect it did. It's the reset-after-merge bug | 12:49 |
| anteaya | never seen it after horizon | 12:50 |
| anteaya | and #openstack-merges didn't pick up a horizon merge | 12:50 |
| anteaya | last one to merge was a cinder patch | 12:50 |
| *** CaptTofu has quit IRC | 12:50 | |
| *** CaptTofu has joined #openstack-infra | 12:51 | |
| ttx | anteaya: hmm, checking | 12:51 |
| *** wchrisj has joined #openstack-infra | 12:51 | |
| anteaya | ttx http://git.openstack.org/cgit/openstack/openstack/commit/?id=47456f0a12212d124902e530e96e3b71127c0dde | 12:51 |
| anteaya | that is the most recent entry for openstack/openstack | 12:51 |
| *** gokrokve has joined #openstack-infra | 12:52 | |
| ttx | anteaya: was watching a pile of all-green successes and it turned into a bunch of "queued" | 12:53 |
| anteaya | yeah | 12:53 |
| anteaya | I figured the horizon, neutron patches at the top were fine | 12:53 |
| ttx | can't find trace of a failed test | 12:53 |
| anteaya | I was concerned about the nova patch beneath | 12:53 |
| anteaya | and -qa doesn't have the fail bot reporting anymore | 12:54 |
| zigo | sdague: yup | 12:54 |
| ttx | anteaya: maybe that washttps://review.openstack.org/#/c/76471/ | 12:55 |
| *** CaptTofu has quit IRC | 12:55 | |
| anteaya | no it was a horizon patch | 12:55 |
| anteaya | I don't have the number, I should have got it | 12:55 |
| ttx | anteaya: no recent test -1 at https://review.openstack.org/#/q/status:open+project:openstack/horizon,n,z | 12:56 |
| sdague | zigo: ok, redone with a function | 12:56 |
| ttx | anteaya: https://review.openstack.org/#/c/73433/ | 12:56 |
| *** gokrokve has quit IRC | 12:56 | |
| ttx | anteaya: "Submitted, Merge Pending" | 12:56 |
| anteaya | that must be it | 12:57 |
| anteaya | I saw submitted, merge pending for the first time last night | 12:57 |
| ttx | sdague: ever seen one that before ? | 12:57 |
| ttx | that one | 12:57 |
| anteaya | which changed to merged after I refreshed the page | 12:57 |
| sdague | ttx: nope | 12:58 |
| sdague | that might be a new failure mode here | 12:58 |
| ttx | anteaya: that's fine. What's not fine is that it's been 16min | 12:58 |
| anteaya | so yeah, it merged or should and also reset the gate, you are correct | 12:58 |
| anteaya | so not limited to nova after all | 12:58 |
| anteaya | yes | 12:58 |
| ttx | sdague: sounds more nefarious though | 12:59 |
| sdague | well, there is not much to be done until fungi is awake | 12:59 |
| ttx | I fear that this half-done state might break the integrity of the gate | 12:59 |
| sdague | yeh, don't know | 13:00 |
| *** rossella-s has joined #openstack-infra | 13:00 | |
| ttx | sdague: At least it's the only one stuck there: https://review.openstack.org/#/q/status:submitted,n,z | 13:00 |
| anteaya | 13 patches in the queue before the next horizon patch | 13:01 |
| anteaya | hopefully we can address that patches status before then | 13:01 |
| ttx | at the current rhythm, gives us about 4 hours | 13:02 |
| anteaya | fungi should be around in an hour and a half | 13:02 |
| anteaya | and yeah, slow rhythm | 13:02 |
| *** smarcet has joined #openstack-infra | 13:05 | |
| *** jgallard has joined #openstack-infra | 13:06 | |
| sdague | gah, I screwed up this migrate patch set again.... grrrr. ok finally setting up local unit testing | 13:06 |
| fungi | i'll see whether i can dig up that change, if it exists yet... but i also think i realized another way that can break us if we really do stop checking it | 13:08 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Load superusers from a yaml file https://review.openstack.org/77884 | 13:08 |
| anteaya | morning fungi | 13:08 |
| fungi | has anyone witnessed any changes besides nova changes triggering this pattern? | 13:09 |
| anteaya | yes | 13:09 |
| anteaya | https://review.openstack.org/#/c/73433/ | 13:09 |
| anteaya | this triggered the latest reset | 13:09 |
| anteaya | ttx found it | 13:09 |
| anteaya | https://review.openstack.org/#/q/status:submitted,n,z | 13:09 |
| fungi | that looks like a separate gerrit bug | 13:09 |
| anteaya | k | 13:09 |
| fungi | we've seen that from time to time when gerrit gets confused | 13:10 |
| anteaya | I saw submitted, merge pending last night, which changed to merged when I refreshed the page | 13:10 |
| ttx | fungi: we got a lot of top-of-gate resets, which added up to that 15hours queue length now | 13:10 |
| anteaya | last night I saw a reset and it was due to a nova patch | 13:10 |
| ttx | fungi: most were the reset-after-merge issue, although the lastet was https://review.openstack.org/#/c/73433/ | 13:10 |
| fungi | the change itself successfully merges, but for some reason gerrit doesn't update its database to reflect that, so the ui will list that change as "submitted, merge pending" until we restart gerrit | 13:11 |
| ttx | fungi: the queue is currently sluggish, about 4 per hour all morning | 13:11 |
| *** mfink has quit IRC | 13:11 | |
| fungi | was every 4th change a nova change? ;) | 13:11 |
| ttx | fungi: mayyybe. | 13:11 |
| fungi | though i guess if you average in legitimate resets for actual job failures, that seems likely | 13:11 |
| ttx | fungi: not sure it's just a DB issue. Change doesn't appear merged | 13:12 |
| *** fbo_away is now known as fbo | 13:13 | |
| anteaya | horizon doesn't show it: http://git.openstack.org/cgit/openstack/horizon/log/ | 13:13 |
| ttx | fungi: not mirrored to github | 13:13 |
| *** eharney has quit IRC | 13:13 | |
| sdague | fungi: so is there a hot fix on the merge timeout issue? like a way to renice some process to make it less likely to fail | 13:13 |
| ttx | nor on cgit | 13:13 |
| ttx | fungi: so it's not just a status issue. It's actually not merged. | 13:14 |
| fungi | ttx: agreed... checking gerrit's queues now to see if it's wedged | 13:14 |
| sdague | ttx: is it not merged in gerrit? | 13:14 |
| sdague | remember gerrit is a git server as well | 13:14 |
| ttx | sdague: how can I check that ? | 13:14 |
| sdague | that's a good question... and I forget how | 13:14 |
| fungi | pull from your gerrit remote | 13:14 |
| fungi | (is the easiest way) | 13:14 |
| fungi | 'git pull gerrit master' in your clone of horizon | 13:15 |
| jaypipes | sdague: can I help with something? | 13:15 |
| jpich | There's a comment on that "submitted" horizon review reading "Change cannot be merged due to unsatisfiable dependencies", not sure if that's been mentioned | 13:16 |
| sdague | jaypipes: yes, I'd like your reviews on the sqlalchemy-migrate patches for 0.9 compatibility, to make sure we didn't entirely f this up | 13:16 |
| *** dstanek has joined #openstack-infra | 13:16 | |
| ttx | sdague: gerrit has 3e4f269591feefc6335b51e17de7797fc8da8a8d on head | 13:16 |
| sdague | jpich: yep, scrollback :) | 13:16 |
| fungi | i do see this in progress in gerrit's queue... a7689930 push git@github.com:openstack/horizon.git | 13:16 |
| jaypipes | sdague: link for me pls? | 13:16 |
| sdague | jaypipes: yep, let me fix this unit test first :) | 13:16 |
| ttx | so apparently not merged in gerrit either | 13:16 |
| jaypipes | no worries | 13:16 |
| jpich | sdague: Ok, thank you! I'm not good at effectively skimming | 13:16 |
| fungi | oho, jpich good eye! | 13:17 |
| sdague | jpich: no worries | 13:17 |
| anteaya | it isn't on gerrit | 13:17 |
| ttx | jpich has the laser eyes | 13:17 |
| anteaya | git clone ssh://anteaya@review.openstack.org:29418/openstack/horizon | 13:17 |
| sdague | oh, damn, I missed the comment context | 13:17 |
| fungi | so i wonder why we have that change depending on commits which aren't present... investigating | 13:17 |
| sdague | fungi: because of the break? | 13:17 |
| ttx | fwiw that caused a reset alright | 13:18 |
| sdague | or an earlier merge race issue | 13:18 |
| fungi | two of the commits it mentions aren't even approved in gerrit yet | 13:18 |
| fungi | and the third doesn't seem to exist at all | 13:18 |
| anteaya | jpich: I'm glad you mentioned it, I must have missed it the first time | 13:18 |
| sdague | jaypipes: https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate+branch:master+topic:sqla_fixes,n,z | 13:18 |
| fungi | i wonder whether gerrit somehow missed associating dependent changes | 13:18 |
| fungi | i'll clone that change and see what i get | 13:18 |
| sdague | it's 5 patches, the final one is just a tox change to fix things | 13:18 |
| sdague | s/fix/test/ | 13:19 |
| jpich | anteaya: Cool, it wasn't immediately obvious to me either | 13:19 |
| sdague | fungi: so there was definitely some total weirdness with the migrate patches this morning in the same way | 13:19 |
| *** esker has joined #openstack-infra | 13:19 | |
| sdague | which I ignored | 13:19 |
| *** david_lyle_ has quit IRC | 13:20 | |
| *** talluri has quit IRC | 13:20 | |
| ttx | argh, new reset. Real fail this time | 13:20 |
| ttx | looks like I won't be able to cut I3 today :/ | 13:21 |
| fungi | it doesn't look like we're pushing the gerrit server *too* hard... load average is sometimes spiking up high enough that it reads 10 over a 5-minute sample, but mostly seems to hang around 2 | 13:21 |
| *** talluri has joined #openstack-infra | 13:21 | |
| anteaya | that patch passed earlier | 13:21 |
| sdague | jaypipes: missed a file | 13:21 |
| sdague | so jaypipes: https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate+branch:master+topic:sqla_fixes,n,z is repushed | 13:21 |
| jaypipes | kk | 13:21 |
| anteaya | both neutron test failures on az2 | 13:23 |
| *** dcramer__ has quit IRC | 13:24 | |
| sdague | anteaya: is az2 the one that has wonky half working ipv6? | 13:24 |
| *** talluri has quit IRC | 13:25 | |
| fungi | i don't know whether we ever narrowed down where we were occasionally seeing that ipv6 issue in hpcloud | 13:25 |
| anteaya | sdague: I know it can't build images very well | 13:26 |
| anteaya | sdague: I don't know its ipv6 status | 13:26 |
| fungi | it seemed to subside after i added some extra debugging output to d-g to help us pinpoint it | 13:26 |
| anteaya | it is a networking error | 13:27 |
| anteaya | checking logs | 13:27 |
| fungi | and yeah, the nodepool images in az2 are stale by a couple weeks. there's a case open with hpcloud support to look into why we get an eof on the ssh connection to them when trying to build a template server | 13:27 |
| anteaya | neutron-rootwrap timeout error: http://logs.openstack.org/88/77788/2/gate/gate-tempest-dsvm-neutron-pg/c7a1cfe/logs/screen-q-agt.txt.gz?level=ERROR | 13:28 |
| sdague | anteaya: ok, this is the ovs timeouts issue | 13:28 |
| sdague | yeh, so that's just normal fails | 13:29 |
| sdague | I don't think related to infrastructure really | 13:29 |
| ttx | yep, it still fails the old way from time to time | 13:30 |
| *** rlandy__ has joined #openstack-infra | 13:31 | |
| sdague | yeh, there are still races in there | 13:32 |
| *** hashar has joined #openstack-infra | 13:32 | |
| *** CaptTofu has joined #openstack-infra | 13:32 | |
| sdague | we have not yet fixed all the openstack bugs | 13:32 |
| fungi | okay, i think whoever put together the stack of changes which triggered the 73433 merge problem must have done something really, really crazy | 13:32 |
| anteaya | same error in both: http://logs.openstack.org/88/77788/2/gate/gate-tempest-dsvm-neutron-pg/c7a1cfe/logs/screen-q-agt.txt.gz?level=ERROR | 13:32 |
| fungi | checkout the 73433 change from gerrit and you can see the commits it mentions are in fact the ones which aren't approved yet | 13:33 |
| fungi | the first of which doesn't even exist in gerrit, but looks like a never-pushed version of https://review.openstack.org/73003 | 13:34 |
| ttx | jpich: ^ | 13:34 |
| *** rlandy_ has quit IRC | 13:34 | |
| * jpich looking | 13:34 | |
| sdague | fungi: so how did that get gate enqueued? | 13:35 |
| anteaya | fungi: where are these dependencies coming from, since that patch lists no dependencies | 13:35 |
| jpich | I pinged the author but he's not online yet - I know it was a series that had to be rebased a few times, dependent patches can get a bit hairy after a while | 13:35 |
| fungi | so i think somehow gerrit allowed someone to push a dependent series with one of the commits not actually existing in gerrit, and that allowed gerrit to consider 73433 disassociated from all those other changes | 13:35 |
| sdague | fungi: interesting | 13:35 |
| fungi | even though when it came time to merge, gerrit saw that it depended on them and refused | 13:35 |
| sdague | well, at least it did the right thing | 13:36 |
| sdague | in the end | 13:36 |
| fungi | so i strongly suspect this is a corner case in gerrit's handling of dependency identification | 13:36 |
| *** rlandy__ is now known as rlandy | 13:36 | |
| sdague | yeh | 13:36 |
| anteaya | so how does this answer why the gate was reset after this patch left the gate? | 13:37 |
| anteaya | because it couldn't be merged? | 13:37 |
| fungi | anteaya: because that change can't actually be merged, yes | 13:37 |
| sdague | anteaya: right, exactly | 13:38 |
| anteaya | k | 13:38 |
| sdague | that's actually working as designed | 13:38 |
| anteaya | yay gerrit | 13:38 |
| sdague | fungi: so on the phantom merge resets | 13:38 |
| anteaya | so how to unwind 73433? | 13:38 |
| jpich | To resolve it, should I suggest to the author to resubmit the whole series, double-checking the ChangeId are present/correct etc? | 13:38 |
| sdague | is there something we can do to either provide priority to the mirror job (nice / ionice) or increase the timeouts | 13:39 |
| fungi | jpich: right, they could probably cherry-pick each change onto the next, starting from the tip of master, and then push the resulting topic branch | 13:39 |
| fungi | sdague: maybe... how do you do that to specific java processes within a common jvm? | 13:39 |
| sdague | because that seems to represent an actual majority of resets right now | 13:39 |
| sdague | fungi: it's all inside one process? | 13:39 |
| *** rossella-s has quit IRC | 13:40 | |
| sdague | yeh, bummer | 13:40 |
| fungi | well, it depends on which server you're talking about | 13:40 |
| fungi | within the ubuntu virtual machine it's all one process | 13:40 |
| jpich | fungi: Thanks, I'll pass on the suggestion | 13:40 |
| fungi | within the java virtual machine there are many processes | 13:40 |
| sdague | fungi: yeh, os level process :) | 13:40 |
| sdague | basically something nice would work on | 13:40 |
| fungi | sdague: right. someone thought it would be amusing to write a large code review system in java | 13:40 |
| fungi | joke's on us apparently | 13:41 |
| jaypipes | sdague: can't find any issues with any of those patches. you and zigo did nice work. :) | 13:41 |
| sdague | jaypipes: thanks much | 13:41 |
| sdague | zigo / dripton_: so with jaypipes +1, I'm cool with all that being pushed in | 13:41 |
| *** pdmars has joined #openstack-infra | 13:41 | |
| anteaya | * cue Ryan_Lane and phabricator | 13:41 |
| zigo | jaypipes: I was fearing that we miss some of the table.quote things not covered by the unit tests. | 13:41 |
| jaypipes | zigo: not that I could see. | 13:42 |
| zigo | And yeah, thanks. | 13:42 |
| sdague | anteaya: yeh, then we just trade known issues for unknown issues | 13:42 |
| zigo | Let's wait until Jenkins tests it ... | 13:43 |
| sdague | zigo: yep | 13:43 |
| fungi | sdague: so we have a couple options to patch zuul for the "phantom resets". one is to just increase the timeout, which kicks the can down the road (but maybe far enough?) and increases the time it takes zuul to decide whether a change really merged to gerrit. the other is to stop checking whether the change replicated to gerrit's local mirror, but after pondering that some more i think there's a new | 13:43 |
| fungi | race condition that increases our chances of hitting | 13:43 |
| anteaya | yeah, java for php | 13:43 |
| sdague | fungi: so I honestly think increasing the timeout is fine | 13:44 |
| sdague | what's the current timeout? | 13:44 |
| fungi | 60 seconds | 13:44 |
| sdague | so given that git review often takes me 35 seconds | 13:44 |
| *** rossella-s has joined #openstack-infra | 13:44 | |
| sdague | I'm not entirely surprised that is hitting that from time to time | 13:45 |
| anteaya | if we make a change is it a gerrit restart? | 13:45 |
| sdague | my instinct would be to bump that to 300 seconds | 13:45 |
| sdague | and call it a day | 13:45 |
| *** nosnos_ has quit IRC | 13:45 | |
| fungi | apparently from the time zuul virtually "clicks" the submit button in gerrit to the time gerrit actually replicates that merged ref to its local mirror in /var/lib/git can take more than 60 seconds for nova commits | 13:45 |
| sdague | because loosing a minute here or there is much better than a reset | 13:45 |
| *** krtaylor has quit IRC | 13:46 | |
| sdague | so could we bump that and restart gerrit in a relatively non disruptive way? | 13:46 |
| sdague | or is that going to just cause chaos | 13:46 |
| anteaya | if we bump to 300 seconds can we time how long it takes, so that after the next day or so, we can fine tune? | 13:46 |
| sdague | also, when was the last gc on the nova tree there? | 13:47 |
| dstanek | what is the litmus test for whether a check belongs in hacking or upstream in pep8? | 13:48 |
| *** akscram has joined #openstack-infra | 13:48 | |
| *** nosnos has joined #openstack-infra | 13:48 | |
| sdague | dstanek: if you can get it into upstream pep8 :) | 13:48 |
| dstanek | sdague: so basically asking Johann if he'll accept? | 13:49 |
| *** nosnos has quit IRC | 13:49 | |
| sdague | dstanek: what's the change in question | 13:49 |
| *** thuc has joined #openstack-infra | 13:49 | |
| *** nosnos has joined #openstack-infra | 13:50 | |
| dstanek | this one triggered my question - https://review.openstack.org/#/c/77950/ | 13:50 |
| *** thuc_ has joined #openstack-infra | 13:50 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/zuul: Increase replication timeout to 5 minutes https://review.openstack.org/78222 | 13:50 |
| fungi | sdague: so that's what a change to the hard-coded replication timeout looks like ^ | 13:51 |
| dstanek | but i'm creating a bunch more - https://review.openstack.org/#/c/78119/ | 13:51 |
| fungi | sdague: ideally if we go that route, we should make it configurable | 13:51 |
| sdague | dstanek: so if it's in pep8 the document | 13:51 |
| sdague | it should be in pep8 the code | 13:51 |
| dstanek | sdague: i think we talking about adding an additional 10 - 15 checks info keystone, but i'd like to have a path to transition those things to pep8 or hacking | 13:52 |
| *** sarob_ has joined #openstack-infra | 13:52 | |
| *** dkranz has joined #openstack-infra | 13:52 | |
| *** freyes has joined #openstack-infra | 13:52 | |
| fungi | sdague: what we discussed yesterday was dropping the replication check entirely, but i'm starting to wonder if that would increase the chance that jobs updating (non-zuul, branch) refs from remote git mirrors test on something older than actual master | 13:52 |
| sdague | fungi: oh, so this requires a zuul restart? | 13:52 |
| *** gokrokve has joined #openstack-infra | 13:52 | |
| fungi | sdague: absolutely | 13:52 |
| fungi | sdague: which is why we were putting it off until after i-3 | 13:52 |
| sdague | yeh | 13:52 |
| *** julim has joined #openstack-infra | 13:52 | |
| sdague | I wasn't sure if this was zuul timing out, or gerrit timing out | 13:52 |
| fungi | all zuul all the time | 13:53 |
| fungi | ;) | 13:53 |
| sdague | does zuul have a config file that is continuously reparsed? | 13:53 |
| sdague | so we could land a timeout adjustment without a restart | 13:53 |
| *** zehicle_at_dell has joined #openstack-infra | 13:53 | |
| fungi | sdague: we could land a change to make this value configurable, restart zuul, then adjust it from configuration in the future | 13:54 |
| sdague | right | 13:54 |
| *** nosnos has quit IRC | 13:54 | |
| sdague | but can we adjust without a restart in the future | 13:54 |
| *** thuc has quit IRC | 13:54 | |
| *** alaski_ is now known as alaski | 13:54 | |
| sdague | because if it's always going to be a zuul restart, then it's cleaner, but not more useful | 13:54 |
| fungi | if we stick with doing this check, then yes i think we should add a knob for it and make sure it's adjusted on reconfig | 13:54 |
| dstanek | sdague: if some of the changes are not accepted into pep8 should i try hacking? or just keep them in keystone? | 13:55 |
| fungi | sdague: 78222 is just a straw man mostly intended to point out where this value is buried | 13:55 |
| sdague | fungi: yep | 13:55 |
| sdague | dstanek: honestly, if you can't get upstream pep8 folks to accept pep8 things, there may be a reason | 13:56 |
| *** sarob_ has quit IRC | 13:56 | |
| sdague | so, anyway, I'd say start there | 13:56 |
| sdague | then reevaluate | 13:56 |
| *** gokrokve has quit IRC | 13:57 | |
| *** mfer has joined #openstack-infra | 13:59 | |
| dstanek | sdague: thanks for the info | 14:00 |
| *** eharney has joined #openstack-infra | 14:00 | |
| *** mfink has joined #openstack-infra | 14:00 | |
| *** salv-orlando has joined #openstack-infra | 14:03 | |
| *** weshay has joined #openstack-infra | 14:04 | |
| *** mfink has quit IRC | 14:05 | |
| *** mfink has joined #openstack-infra | 14:05 | |
| sdague | fungi: so unless you are checking on something else on fire - https://review.openstack.org/#/c/77728/ would be good to get in. So I don't break hooks again by accident | 14:07 |
| *** jeckersb_gone is now known as jeckersb | 14:08 | |
| sdague | also, any thoughts on if the gc would help? | 14:09 |
| *** amotoki has quit IRC | 14:09 | |
| *** lahoucine_ has joined #openstack-infra | 14:11 | |
| *** briancurtin has joined #openstack-infra | 14:11 | |
| *** lahoucine_ has quit IRC | 14:12 | |
| sdague | ok, under / over, does the nova change cause a reset? | 14:14 |
| *** dhellmann_ is now known as dhellmann | 14:14 | |
| *** saju_m has quit IRC | 14:15 | |
| * ttx watches too | 14:15 | |
| *** zns has joined #openstack-infra | 14:15 | |
| sdague | because I think if we are really reseting on every nova change | 14:15 |
| sdague | we need to take a zuul restart | 14:15 |
| SergeyLukjanov | sdague, re https://review.openstack.org/#/c/77728/ | 14:15 |
| ttx | sdague: the next nova chnage is pretty far down the queue though | 14:15 |
| anteaya | nope it went it | 14:15 |
| sdague | it seems to have made it | 14:15 |
| SergeyLukjanov | sdague, will devstack start anyway? | 14:16 |
| sdague | SergeyLukjanov: yes | 14:16 |
| sdague | SergeyLukjanov: no, we replaced gate_hook | 14:16 |
| zns | Hi - any gerrit admins around? Could you add ziad-sawalha to satori-core? | 14:16 |
| anteaya | hi zns | 14:17 |
| anteaya | we have a few very important items on the go right now | 14:17 |
| anteaya | but when there is a moment and we have some time to breath, someone will read the backscroll and add you as satori-ptl | 14:17 |
| SergeyLukjanov | sdague, looking on devstack-gate now, the line I'm worry about is https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L241 and then just "else" and stack.sh exec under else | 14:18 |
| sdague | I'm just excited my devstack changes that might fix sqlalchemy tempest run is in there | 14:18 |
| *** bknudson has quit IRC | 14:18 | |
| *** mriedem has joined #openstack-infra | 14:18 | |
| sdague | SergeyLukjanov: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L250 | 14:18 |
| zns | anteaya: ok, thanks. | 14:19 |
| ttx | nice pile of greens at top of gate right now, if those two tempest changes pass | 14:19 |
| sdague | we'll never run devstack-vm-gate.sh | 14:19 |
| SergeyLukjanov | sdague, got it, thx for explanation | 14:19 |
| fungi | sorry, was busy annotating the replication timeout change | 14:19 |
| *** esker has quit IRC | 14:19 | |
| SergeyLukjanov | sdague, => your change lgtm | 14:20 |
| fungi | sdague: running check-dg-hooks-dsvm voting but only in the check pipeline? | 14:22 |
| ttx | back under 15h, phew | 14:22 |
| *** mgagne1 has quit IRC | 14:22 | |
| sdague | fungi: that should be enough to block issues | 14:22 |
| anteaya | fungi: for later when lahoucine returns to the channel he is after some gerrit db clean up on his account: http://lists.openstack.org/pipermail/openstack-infra/2014-March/000958.html | 14:22 |
| fungi | sdague: gc where... for gerrit's repos and local mirrors? we already repack them weekly | 14:22 |
| sdague | though I'm not sure if that was intentional or not :) | 14:22 |
| sdague | fungi: right, but it's been a busy week | 14:23 |
| sdague | and I wonder if doing that on nova in /var/lib/git might speed things up | 14:23 |
| sdague | as a work around | 14:23 |
| sdague | or hot fix | 14:23 |
| sdague | just to reduce the chance of phantom reset | 14:23 |
| ttx | next nova change in #8 slot | 14:25 |
| *** gokrokve has joined #openstack-infra | 14:25 | |
| fungi | sdague: well, i'll do a catch-up repack on /var/lib/git and pack-refs too (we do the latter on the remote mirrors but not on the local mirror) | 14:25 |
| sdague | fungi: cool | 14:26 |
| *** dkliban_ is now known as dkliban | 14:27 | |
| *** homeless_ has quit IRC | 14:28 | |
| *** gokrokve has quit IRC | 14:28 | |
| *** homeless has joined #openstack-infra | 14:28 | |
| *** pblaho has quit IRC | 14:28 | |
| anteaya | I was planning on being here all morning though I had to leave this afternoon for an appointment, that is until I broke a filling on a tooth last night | 14:29 |
| anteaya | good news is I got an appointment this morning to fix it | 14:29 |
| anteaya | bad news is I got an appointment this morning to fix it | 14:29 |
| *** esker has joined #openstack-infra | 14:29 | |
| *** thuc_ has quit IRC | 14:29 | |
| ttx | fungi: the post pipeline seems to be struggling for resources | 14:29 |
| fungi | anteaya: teeth are nothing to trifle with. the sooner the better | 14:30 |
| ttx | fungi: is that expected ? | 14:30 |
| *** thuc has joined #openstack-infra | 14:30 | |
| fungi | ttx: the post pipeline has a lower priority than the gate pipeline | 14:30 |
| ttx | my scripts timeout after &5 minutes waiting for a tarball build | 14:30 |
| ttx | fungi: ok | 14:30 |
| ttx | 15* | 14:30 |
| fungi | ttx: post and check get to fight it out for resources, gate trumps them | 14:30 |
| anteaya | so I am disappearing and wont be back all day | 14:31 |
| anteaya | sorry about that | 14:31 |
| anteaya | fungi: thanks for understanding | 14:31 |
| fungi | anteaya: good luck! | 14:31 |
| *** pblaho has joined #openstack-infra | 14:32 | |
| *** gokrokve has joined #openstack-infra | 14:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: add job for testing that hooks work https://review.openstack.org/77728 | 14:34 |
| *** thuc has quit IRC | 14:34 | |
| openstackgerrit | sahid proposed a change to openstack-infra/config: Configure check and gate to stackforge/warm https://review.openstack.org/78231 | 14:34 |
| *** jomara has joined #openstack-infra | 14:35 | |
| jomara | hi, i am the guy that broke gerrit with a 5-deep nested horizon patch set | 14:38 |
| *** ildikov_ has quit IRC | 14:38 | |
| fungi | jomara: it looks like one of the commits didn't make it onto gerrit. were you using git-review? any special command-line options (maybe -R)? | 14:39 |
| jomara | fungi: just basic git review | 14:39 |
| jomara | fungi: no special options that i remember | 14:39 |
| jomara | fungi: i had to fix some things in patch 1 & 2 (where 1 is the first patch), so i got a fresh master, cherry picked them one at a time (fixing them along the way, where necessary), and git-reviewed each time | 14:40 |
| *** krtaylor has joined #openstack-infra | 14:40 | |
| *** dizquierdo has quit IRC | 14:40 | |
| jomara | so master -> cherry pick a.1 -> modify to a.2 -> git review -> cherry pick b.1 -> modify to b.2 -> git review, etc | 14:40 |
| *** Aleksey has quit IRC | 14:42 | |
| *** dizquierdo has joined #openstack-infra | 14:42 | |
| jomara | fungi: jpich recommended that i do that process again, which i am happy to do | 14:43 |
| *** sarob_ has joined #openstack-infra | 14:43 | |
| *** bknudson has joined #openstack-infra | 14:43 | |
| jpich | jomara: when you cherry-picked b.1, were you still on top of a.2? | 14:44 |
| jomara | yes | 14:44 |
| fungi | jomara: here's what i see in the git log when i checkout the one which got approved but failed to merge http://paste.openstack.org/show/72615 | 14:44 |
| mordred | morning all | 14:45 |
| fungi | jomara: that second commit (the parent) somehow didn't make it onto gerrit | 14:45 |
| fungi | jomara: the commit sha (530ecf882dd89a8058a9f78fb01cf2fe953658d8) doesn't correspond to any of the patchsets for change id I87ebd7feaeddcfe7fe76c92b075c53de7016846b | 14:46 |
| jomara | that is...odd | 14:47 |
| jomara | let me walk through the history | 14:47 |
| SergeyLukjanov | mordred, o/ | 14:47 |
| fungi | jomara: that suggests that you somehow managed to modify your commit for I87ebd7feaeddcfe7fe76c92b075c53de7016846b after you ran git review on it | 14:47 |
| *** sarob_ has quit IRC | 14:47 | |
| *** rlandy_ has joined #openstack-infra | 14:48 | |
| fungi | so probably either a commit --amend or rebase or something along those lines | 14:48 |
| mordred | SergeyLukjanov: how're things/ | 14:48 |
| SergeyLukjanov | mordred, mostly fine, trying to make a checklist for renaming ;) | 14:48 |
| jomara | fungi: i was rebasing and amending quite a bit to fix some stuff, so its very possible that i made a mistake | 14:49 |
| SergeyLukjanov | mordred, but gate is quite slow | 14:49 |
| jpich | jomara: FWIW when I have to push dependent patches, I use git review only once at the end. It first confirms with you that you're about to upload several patches. Perhaps it would help in a similar situation in the future | 14:49 |
| jomara | jpich: the problem is that i worked on these over a several week period | 14:49 |
| fungi | jomara: i always do the same thing jpich is suggesting, so i agree | 14:49 |
| jomara | jpich: they just didnt have eyes on them until this week | 14:49 |
| *** jnoller has joined #openstack-infra | 14:50 | |
| *** rlandy has quit IRC | 14:50 | |
| *** mgagne has joined #openstack-infra | 14:51 | |
| fungi | zns: you're now the initial member of https://review.openstack.org/#/admin/groups/283,members and https://review.openstack.org/#/admin/groups/284,members | 14:51 |
| jpich | jomara: Yes, of course. When you start updating one patch in the middle though, the others usually need a freshening up too or their dependency tree shows as "Outdated" - so most will need to be uploaded too | 14:51 |
| *** jgrimm has quit IRC | 14:52 | |
| *** thomasem has joined #openstack-infra | 14:52 | |
| jomara | jpich: in this case, i started from the first patch though | 14:53 |
| jomara | what if i : | 14:53 |
| *** dcramer__ has joined #openstack-infra | 14:53 | |
| jomara | checkout master again | 14:53 |
| jomara | apply 1-5 fixed patches | 14:53 |
| jomara | git review *once* | 14:53 |
| jomara | do you think that will fix the situation? | 14:53 |
| mordred | SergeyLukjanov: woot! I love it when gate is quite slow at FF | 14:53 |
| jpich | jomara: Then the dependencies should be handled correctly by gerrit - IMO. Make sure the ChangeId all remain the same :) | 14:54 |
| *** ildikov_ has joined #openstack-infra | 14:54 | |
| jomara | jpich: the changeid should be in the commit message and not get changed when i cherry pick it right? how would it get changed>? | 14:54 |
| SergeyLukjanov | mordred, 13h is probably too slow :) | 14:54 |
| jpich | jomara: Magic! But you're right there's not reason it should change. | 14:55 |
| fungi | jomara: you're right, they shouldn't change unless you do something to change them by editing the commit message | 14:55 |
| fungi | jomara: so if you do modify commit messages on them, just make sure not to alter/delete the change-id line and make sure it stays in the final paragraph of the commit message | 14:56 |
| *** rlandy__ has joined #openstack-infra | 14:56 | |
| mordred | SergeyLukjanov: oh. nevermind. wrong kind of slow :) | 14:56 |
| SergeyLukjanov | mordred :) | 14:56 |
| *** zns has quit IRC | 14:57 | |
| *** zns has joined #openstack-infra | 14:58 | |
| *** rlandy_ has quit IRC | 14:58 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Load storyboard projects from projects.yaml https://review.openstack.org/78249 | 14:58 |
| *** homeless has quit IRC | 14:58 | |
| *** esker has quit IRC | 14:58 | |
| *** eharney has quit IRC | 14:58 | |
| *** homeless has joined #openstack-infra | 14:58 | |
| mordred | NikitaKonovalov, SergeyLukjanov: ^^ that look right to you? | 15:00 |
| jomara | ok, im going to walk through this right now | 15:00 |
| SergeyLukjanov | mordred, looking now | 15:00 |
| jomara | fungi: would you mind giving me a sanity check once i have it ready | 15:00 |
| *** jeckersb is now known as jeckersb_gone | 15:00 | |
| *** resker has joined #openstack-infra | 15:00 | |
| zns | fungi: thank you! | 15:00 |
| SergeyLukjanov | mordred, should we add "depends on" ```exec { 'load-projects-yaml':``` and something else... like storyboard starting? | 15:02 |
| fungi | jomara: i can, though gerrit will provide you with one as well... just look at the "dependency" section in the webui and follow the links backwards and forwards to make sure the relationships of your changes are reflected appropriately | 15:02 |
| jomara | kk | 15:02 |
| jomara | waaaait | 15:03 |
| jomara | fungi: ok so | 15:03 |
| jomara | my 1st change is already merged | 15:03 |
| jomara | should i this with just 2-4? | 15:03 |
| fungi | jomara: it didn't actually merge did it? which change? | 15:03 |
| mordred | SergeyLukjanov: don't think we need it started ... but I do think we should probably run migrations first if we need to run them | 15:03 |
| *** CaptTofu has quit IRC | 15:04 | |
| jomara | fungi: i just pulled from ... git@github.com:openstack/horizon.git - my commit is the latest | 15:04 |
| *** eharney has joined #openstack-infra | 15:04 | |
| jomara | 2014-01-14 10:25 Jordan OMara o │ [h1] [gerrit/master] [jsomara/heat-stack-update] Heat Stack update view/fo~ | 15:04 |
| fungi | jomara: oh, okay, not the one which failed to merge then... a different one | 15:04 |
| jomara | fungi: it must have been the 2nd or 3rd one | 15:04 |
| jomara | fungi: ill just star twith 2, and do 2-5 | 15:04 |
| jomara | same process though | 15:04 |
| fungi | jomara: so yes, i suggest just cherry-picking the others which aren't in master yet | 15:04 |
| SergeyLukjanov | mordred, yup, looks like migrations should be run first to not | 15:05 |
| *** denis_makogon has joined #openstack-infra | 15:07 | |
| denis_makogon | Guys, goodday, i need someone who could help me with issue explanation | 15:07 |
| jomara | fungi: http://fpaste.org/82610/03208313/ | 15:08 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Load storyboard projects from projects.yaml https://review.openstack.org/78249 | 15:08 |
| *** mwagner_lap has quit IRC | 15:08 | |
| jeblair | fungi: good morning. there's a lot of scrollback; anything i should know? | 15:08 |
| denis_makogon | http://logs.openstack.org/51/77551/3/check/gate-trove-python27/47beba1/console.html |||| http://logs.openstack.org/51/77551/3/check/gate-trove-python26/17f6ee4/console.html - i'm seing this issue | 15:08 |
| *** mgagne1 has joined #openstack-infra | 15:09 | |
| mordred | morning jeblair | 15:10 |
| sdague | denis_makogon: interesting | 15:10 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcImplbmtpbnMgOiAzIGluY29ycmVjdCBwYXNzd29yZCBhdHRlbXB0c1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzk0MDMyMTg5NzQxfQ== | 15:10 |
| fungi | jeblair: https://review.openstack.org/78222 is a straw man change for increasing the replication timeout value in zuul, along with some musings on why removing that check could be detrimental after all | 15:11 |
| sdague | fungi: check out that logstash | 15:11 |
| denis_makogon | sdague, how this even possible ? | 15:12 |
| sdague | that's a good question | 15:12 |
| *** mgagne has quit IRC | 15:12 | |
| openstackgerrit | A change was merged to openstack-infra/gerrit: add a new .gitreview file https://review.openstack.org/73126 | 15:12 |
| SergeyLukjanov | mordred, #2 lgtm | 15:13 |
| fungi | sdague: what's the concern there? | 15:13 |
| SergeyLukjanov | mordred, # considering my poor puppet knowledge | 15:14 |
| *** basha has joined #openstack-infra | 15:14 | |
| *** sandywalsh has joined #openstack-infra | 15:14 | |
| sdague | fungi: so what is otherwise a successful run - http://logs.openstack.org/51/77551/3/check/gate-trove-python27/47beba1/console.html | 15:15 |
| sdague | dies on a sudo issue at the end in the jenkins cleanup | 15:15 |
| jeblair | sdague: works as intended then | 15:15 |
| jomara | fungi: if you ACK me i will git-review that | 15:15 |
| *** jeckersb_gone is now known as jeckersb | 15:15 | |
| jeblair | sdague: it's never okay for a unit test suite to "sudo". | 15:15 |
| fungi | jomara: i'm hoping i have a moment to take a look once i'm done dealing with broken things | 15:15 |
| jomara | fungi: np | 15:15 |
| sdague | jeblair: is that what's it's catching? | 15:16 |
| jeblair | sdague: yep. | 15:16 |
| sdague | ok, interesting | 15:16 |
| sdague | I guess it's hard to understand that from the output | 15:16 |
| *** CaptTofu has joined #openstack-infra | 15:16 | |
| jeblair | sdague: people try to do it so often (especially neutron) that we have a special test for it in the tox wrapper | 15:16 |
| *** atiwari has joined #openstack-infra | 15:17 | |
| fungi | sdague: the only thing i find mildly concerning is that there were three hits for it in the gate pipeline for the past week, in gate-trove-python27 running "sudo innobackupex --stream=xbstream --incremental --incremental-lsn=54321 /var/lib/mysql" | 15:17 |
| sdague | jeblair: do you think the unit test actually did that? | 15:17 |
| jeblair | sdague: it's supposed to output a message if it fails | 15:17 |
| sdague | yeh, it's just dumping a line without an explanation | 15:18 |
| *** krotscheck has joined #openstack-infra | 15:18 | |
| fungi | sdague: given that this is trove tests and sudoing commands having to do with mysql server management, i'm pretty certain the unit tests are trying to do that | 15:18 |
| jeblair | sdague: i think it might be because of a "set -e" further up in the script | 15:19 |
| denis_makogon | so, it means that unit test trying to perform manipulations over real server ? | 15:19 |
| fungi | denis_makogon: yeah, it looks like something one or more of the trove unit tests do ends up calling sudo | 15:20 |
| jeblair | sdague: so it aborts before printing the error | 15:20 |
| openstackgerrit | sahid proposed a change to openstack-infra/config: Configure check and gate to stackforge/warm https://review.openstack.org/78231 | 15:20 |
| denis_makogon | fungi, and since jenkins is a java base application, "/usr/bin/killall java" breaks it | 15:21 |
| *** pblaho has quit IRC | 15:23 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: be more explicit about why we are failing on sudo check https://review.openstack.org/78257 | 15:23 |
| sdague | so how about that | 15:23 |
| *** pblaho has joined #openstack-infra | 15:23 | |
| sdague | just to be more explicit | 15:23 |
| sdague | denis_makogon: no, it means the jenkins user can't sudo | 15:23 |
| fungi | denis_makogon: yeah, that (if it were allowed, which it isn't because the jenkins user doesn't have sudo access on the machines where that ran) would in theory kill the jenkins slave agent running there, disrupt communication with the jenkins master and cause weird nasty jenkins errors instead of getting you a real error from the slave, possibly | 15:24 |
| sdague | and we fail tests that try to | 15:24 |
| denis_makogon | sdague, you're awesome =) | 15:24 |
| jeblair | sdague: no you don't understand... | 15:24 |
| jeblair | sdague: there already is an explanation | 15:24 |
| *** resker has quit IRC | 15:24 | |
| jeblair | sdague: someone broke it by adding a "set -e" higher up in the script | 15:24 |
| jeblair | sdague: in the "run-unittests.sh" script | 15:24 |
| sdague | jeblair: oh | 15:24 |
| jeblair | sdague: that's what needs to be fixed | 15:24 |
| ociuhandu | hi all | 15:24 |
| ociuhandu | In the last couple of days we started seeing in the hyper-v CI a high occurence of the following error in nova-compute.log: "RequiredOptError: value required for option: lock_path" (on tempest runs) | 15:24 |
| fungi | sdague: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/jenkins-sudo-grep.sh | 15:25 |
| ociuhandu | the only thing we found related to this error is https://bugs.launchpad.net/nova/+bug/1264058 | 15:25 |
| jeblair | modules/jenkins/files/slave_scripts/run-unittests.sh | 15:25 |
| fungi | sdague: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/run-unittests.sh#n70 | 15:25 |
| fungi | oh, jeblair beat me to it | 15:26 |
| ociuhandu | can anyone help us on this? | 15:26 |
| *** resker has joined #openstack-infra | 15:26 | |
| *** yamahata has joined #openstack-infra | 15:27 | |
| *** krotscheck has quit IRC | 15:27 | |
| SpamapS | fungi: hi. So we think we finally chased down our hardware issues to a combination of the bad motherboard _AND_ a buggy NIC driver.... | 15:27 |
| *** coolsvap has joined #openstack-infra | 15:27 | |
| fungi | ociuhandu: you might try asking in #openstack-nova | 15:28 |
| sdague | fungi: yep | 15:28 |
| ociuhandu | fungi: ok, thanks | 15:28 |
| sdague | so did clarkb add that to catch a testr fail? | 15:28 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: remove inline set -e that is preventing explanations https://review.openstack.org/78262 | 15:29 |
| *** talluri has joined #openstack-infra | 15:29 | |
| denis_makogon | sdague, so, this issue related to project or to the jenkins ? | 15:29 |
| sdague | denis_makogon: your project | 15:29 |
| sdague | it needs to not call sudo in unit tests | 15:30 |
| denis_makogon | hm, ok, thanks | 15:30 |
| fungi | jomara: that looks fine, i suppose. the real test will be checking what gerrit does with it once you run git review | 15:30 |
| jomara | fungi: ok, lets do this! | 15:32 |
| jomara | ok, this looks correct | 15:32 |
| jomara | 4 patches | 15:32 |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Ruby metrics plugin support added https://review.openstack.org/73528 | 15:34 |
| *** mdenny has joined #openstack-infra | 15:35 | |
| *** mgagne1 is now known as mgagne | 15:35 | |
| *** rcleere has joined #openstack-infra | 15:36 | |
| *** krotscheck has joined #openstack-infra | 15:37 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Auth controller https://review.openstack.org/68642 | 15:37 |
| jeblair | fungi: responded to the zuul patch (i still think it's safe) | 15:37 |
| jeblair | to remove | 15:37 |
| jomara | fungi: this is weird, i think the dpeendency chain si correct, but it added back some -1 reviews to one of the patch that have been resolved | 15:38 |
| sdague | fungi: so on https://review.openstack.org/#/c/78262/ I think if set -e has other side effects, those need to be caught. Because it won't do what is really intended | 15:38 |
| fungi | jeblair: oh, i should review devstack-gate. for some reason i thought we pulled zuul refs from the zuul mergers and other refs from the git mirrors | 15:38 |
| jeblair | fungi: we do -- but in what circumstance is that wrong? | 15:39 |
| *** wenlock has joined #openstack-infra | 15:39 | |
| fungi | jeblair: i tried to explain the race there in my comment, but maybe i wasn't clear (or maybe i'm just wrong) | 15:40 |
| jeblair | fungi: that's what i meant by "if it's participating in the gate stack" we'll pull it from zuul... anything that isn't participating in the gate stack we won't, but it doesn't matter because that means it's not about to change in a racy way | 15:40 |
| jeblair | fungi: oh, it's about the window | 15:40 |
| fungi | jeblair: more about how we might start testing a change which depends on a change in another project which just merged, but there are no other changes ahead for the same project so there's no zuul ref which incorporates that now-merged change | 15:41 |
| fungi | which might not yet be on the mirrors when we pull the branch from them | 15:41 |
| *** krotscheck has quit IRC | 15:42 | |
| fungi | and the most common example of "depends on a change in another project" is when we have an integration-test-impacting bug fix which we've promoted in the gate | 15:42 |
| *** ryanpetrello has joined #openstack-infra | 15:43 | |
| fungi | we might see changes behind it which haven't started testing yet (either because they're beyond the window or because they coincidentally got approved at the same moment) fail on the bug for which we just merged a fix | 15:43 |
| jeblair | fungi: understood | 15:44 |
| *** david-lyle has joined #openstack-infra | 15:44 | |
| fungi | i'm not saying that's necessarily likely, nor that it can't happen already now under the right circumstances, just pointing out that by removing the check we *in theory* increase our chances of losing that race | 15:45 |
| *** pcrews has joined #openstack-infra | 15:45 | |
| fungi | or winning, for some definitions of winning | 15:45 |
| *** basha has quit IRC | 15:45 | |
| jeblair | fungi: yeah, since we're checking 1 out of 7 mirrors we replicate to, basically we're increasing the chances because we're removing a "sleep" statement. | 15:46 |
| jeblair | because the check itself is ineffective | 15:46 |
| jeblair | it's just the time the check takes that may reduce the chances of what you describe | 15:46 |
| *** resker has quit IRC | 15:47 | |
| *** esker has joined #openstack-infra | 15:48 | |
| jeblair | fungi: i do believe the window affects this because it synchronizes the start of a new change with the merging of an old one; so it actually _increases_ the chance of this | 15:48 |
| dtroyer | clarkb: here yet? https://review.openstack.org/#/c/75852/ really is wedged on the no-newlines in the existing repo. /project-requirements-change.py looks like it checks the exiting file and fails (the current situation) rather than just using what is in the repo and checking the update. | 15:49 |
| dtroyer | to refresh: the original requirements.txt and test-requirements.txt files didn't end with newlines. any attempt to change them fails in /project-requirements-change.py with a 'file doesn't end in newline' error | 15:50 |
| jeblair | fungi: one solution might be to keep the merged change around for a little bit and continue to use it to compose changes | 15:50 |
| *** Sukhdev has joined #openstack-infra | 15:51 | |
| dtroyer | can we push in that fix manually or shall I do disable that job temporarily? | 15:51 |
| *** chandan_kumar has quit IRC | 15:51 | |
| fungi | jeblair: agreed on the window making this more likely. as for it being purely a sleep statement, not really. we at least know we waited long enough for gerrit to process the submit and start replicating, so it's a sort of dynamic/adaptive sleep | 15:51 |
| fungi | since as we've seen it varies depending on the complexity of the project to which we merged the commit | 15:52 |
| jeblair | fungi: we always wait for it to process the submit -- the check only causes us to wait until 1 out of the 7 replicas completes updating | 15:52 |
| jeblair | fungi: it might be the first or the last of the 7 | 15:52 |
| fungi | jeblair: oh, i completely agree | 15:52 |
| *** esker has quit IRC | 15:52 | |
| fungi | jeblair: and yes, keeping merged changes around for a little while and making zuul refs out of them would likely mitigate this to a great extent while improving gating performance, though it sounds non-trivial to implement | 15:53 |
| *** zhiyan is now known as zhiyan_ | 15:53 | |
| jeblair | fungi: yeah, a bit. just brainstorming. | 15:53 |
| fungi | anyway, we know what we are doing now is suboptimal. we're weighing unknowns on deawbacks for the solutions, so i suppose we can just try something and see if it's better/worse than what we're doing now | 15:55 |
| *** jgrimm has joined #openstack-infra | 15:55 | |
| fungi | s/deawbacks/drawbacks/ | 15:55 |
| *** thedodd has joined #openstack-infra | 15:55 | |
| *** esker has joined #openstack-infra | 15:55 | |
| jeblair | i'm certainly okay increasing the timeout and restarting immediately if we think this is impacting i3 | 15:55 |
| *** krotscheck has joined #openstack-infra | 15:56 | |
| fungi | that's a question for sdague/ttx... ^ | 15:56 |
| jeblair | (also, i think that the exact problem you describe could happen on a false gate reset, because again, it's synchronizing job starts with a merge-related replication race) | 15:56 |
| ttx | jeblair: maybe we could do that just after the next reset | 15:56 |
| fungi | jeblair: yep, no argument there. it definitely could | 15:57 |
| ttx | jeblair: I have a number of changes coming to the top of the gate that would trigger new I3 cuts when they complete | 15:57 |
| *** chandan_kumar has joined #openstack-infra | 15:57 | |
| ttx | so i'd rather not delay them just now | 15:57 |
| *** rlandy__ is now known as rlandy | 15:58 | |
| *** amotoki has joined #openstack-infra | 15:58 | |
| *** skraynev is now known as skraynev_afk | 15:58 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/storyboard: Remove Branch and Milestone legacy tables https://review.openstack.org/77187 | 15:59 |
| ttx | jeblair: looks like we just had a reset though | 16:01 |
| ttx | jeblair: so i guess restarting now would have limited impact ? | 16:01 |
| jeblair | ttx: we haven't approved the change yet, so we're still about 30 minutes from being able to restart it with it. was mostly looking for whether you wanted to do it at all. | 16:02 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/zuul: Stop waiting for Gerrit commit replication https://review.openstack.org/78277 | 16:02 |
| fungi | so there's ^ the alternate approach | 16:02 |
| ttx | jeblair: i blame the reset-after-merge for having built that gigantic backlog | 16:03 |
| ttx | jeblair: would definitely like to see it gone, and pushing it in just after a reset should limit the delays | 16:03 |
| *** jaypipes has quit IRC | 16:03 | |
| *** jergerber has joined #openstack-infra | 16:04 | |
| jeblair | fungi: i think if we're in a hurry, then we should approve the 300sec change to minimize functional changes right now | 16:04 |
| fungi | jeblair: i could go either way, but if you have a more immediate preference for the timeout increase then let's move forward with it for now | 16:05 |
| jeblair | fungi: yeah, that will change less of zuul's behavior and give us time to think about whether there's a better way to solve your concern | 16:06 |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Auth support https://review.openstack.org/73219 | 16:06 |
| *** krotscheck has quit IRC | 16:06 | |
| fungi | wfm | 16:06 |
| *** krotscheck has joined #openstack-infra | 16:08 | |
| *** jcoufal has joined #openstack-infra | 16:10 | |
| jeblair | dtroyer: should probably make that job nonvoting for now | 16:14 |
| dtroyer | jeblair: roger | 16:14 |
| *** freyes has quit IRC | 16:16 | |
| *** mwagner_lap has joined #openstack-infra | 16:17 | |
| *** sandywalsh has quit IRC | 16:17 | |
| *** zehicle_at_dell has quit IRC | 16:21 | |
| *** krotscheck has quit IRC | 16:21 | |
| *** vkozhukalov has joined #openstack-infra | 16:21 | |
| *** rossella-s has quit IRC | 16:21 | |
| *** esker has quit IRC | 16:23 | |
| *** esker has joined #openstack-infra | 16:24 | |
| *** rfolco has quit IRC | 16:24 | |
| jraim | dhellmann: ping | 16:24 |
| *** rfolco has joined #openstack-infra | 16:25 | |
| *** rossella-s has joined #openstack-infra | 16:25 | |
| *** andreaf has quit IRC | 16:25 | |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/config: Temporarily no-vote the requirements check for openstacksdk https://review.openstack.org/78288 | 16:28 |
| *** resker has joined #openstack-infra | 16:28 | |
| *** resker has quit IRC | 16:28 | |
| dtroyer | jeblair: is theis the correct way to set that job non-voting for only openstacksdk? https://review.openstack.org/78288 | 16:28 |
| *** esker has quit IRC | 16:28 | |
| jeblair | dtroyer: no i don't think that'll work. maybe just remove the check-requrements template completely for now. | 16:29 |
| dtroyer | ok, will do | 16:29 |
| jeblair | dtroyer: then merge the fix then add it back | 16:29 |
| openstackgerrit | Dean Troyer proposed a change to openstack-infra/config: Temporarily no-vote the requirements check for openstacksdk https://review.openstack.org/78288 | 16:30 |
| *** melwitt has joined #openstack-infra | 16:30 | |
| *** melwitt1 has joined #openstack-infra | 16:31 | |
| *** jcoufal has quit IRC | 16:31 | |
| *** sandywalsh has joined #openstack-infra | 16:32 | |
| *** krotscheck has joined #openstack-infra | 16:33 | |
| *** melwitt has quit IRC | 16:34 | |
| *** melwitt1 is now known as melwitt | 16:34 | |
| *** vogxn has joined #openstack-infra | 16:36 | |
| jraim | sdague: ping | 16:36 |
| *** esker has joined #openstack-infra | 16:38 | |
| *** sandywalsh has quit IRC | 16:38 | |
| *** markmcclain has joined #openstack-infra | 16:41 | |
| sdague | jraim: pong | 16:41 |
| *** vkozhukalov has quit IRC | 16:42 | |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add pypy to satori checks (it's already in gates) https://review.openstack.org/78296 | 16:42 |
| jraim | sdague: we fixed our gate issue. Turns out is was caused by a merge of the newer oslo.logging code that broke something in a way our tests didn't catch | 16:42 |
| jraim | sdague: here is the fix - https://review.openstack.org/#/c/78057/ | 16:42 |
| sdague | jraim: so I posted back about that already. It didn't run any tests | 16:43 |
| sdague | http://logs.openstack.org/57/78057/1/check/gate-barbican-devstack-dsvm/92409cb/logs/devstack-gate-post-test-hook.txt.gz | 16:43 |
| sdague | "2014-03-05 02:02:35.969 | Ran 0 tests in 0.000s" | 16:44 |
| jraim | our tests are all CloudCafe stuff right now, which doesn't run in the gate. Our plan is to start moving things over, but my understanding is that was for graduation | 16:44 |
| *** sarob_ has joined #openstack-infra | 16:44 | |
| jraim | since it is a non-voting job right now | 16:44 |
| sdague | jraim: but that doesn't actually demonstrate the service actually responding | 16:44 |
| jraim | sdague: or are you just looking for 1 or 2 sanity tests? | 16:44 |
| sdague | jraim: right, the point of this is to know the service can come up do something | 16:45 |
| sdague | and right now there is actually no validation of that | 16:45 |
| jraim | we included some checks in our devstack exercises and such, just nothing in tempest | 16:45 |
| sdague | that gate job doesn't actually tell you if barbican is running | 16:45 |
| sdague | jraim: this doesn't need to be tempest | 16:46 |
| *** thuc has joined #openstack-infra | 16:46 | |
| sdague | the post_test_hook is in your tree | 16:46 |
| jraim | oh okay, just something as part of the jon | 16:46 |
| jraim | *job | 16:46 |
| sdague | right, so me where in that job barbican is sanity checked to be functioning | 16:46 |
| jraim | we do have some checks in our exercises, we'll port those over so they run as part of the gate job | 16:46 |
| *** krotscheck has quit IRC | 16:46 | |
| sdague | even in a trivial way | 16:46 |
| jraim | okay, that seems reasonable | 16:46 |
| sdague | great | 16:46 |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily no-vote the requirements check for openstacksdk https://review.openstack.org/78288 | 16:46 |
| jraim | cool, I'll ping you when we get it moved over | 16:47 |
| *** gyee has joined #openstack-infra | 16:47 | |
| sdague | jraim: the reason for this whole exercise is that we have integrated projects that are barely demonstrating functioning with the rest of OpenStack, and have taken *many* cycles to get into the gate | 16:47 |
| *** esker has quit IRC | 16:47 | |
| sdague | so we want to make sure new services, while young, starting in a functioning way with the rest of the stack | 16:47 |
| sdague | so this isn't a giant surprise that they actually need to work in this environment | 16:48 |
| jraim | sdague: that makes sense to me | 16:48 |
| sdague | you guys are the first test of the process, so we'll figure out how to make it more clear in the future on expectations | 16:48 |
| sdague | so appreciate your patience in working through this | 16:48 |
| *** openstackstatus has joined #openstack-infra | 16:48 | |
| *** openstackstatus has quit IRC | 16:48 | |
| *** openstackstatus has joined #openstack-infra | 16:49 | |
| jraim | no problem, you guys have had a lot of patience with our changes - we appreciate it :) | 16:49 |
| *** sarob_ has quit IRC | 16:49 | |
| jeblair | https://launchpad.net/heat/+milestone/icehouse-rc1 (for reference� | 16:49 |
| *** mrodden has quit IRC | 16:49 | |
| *** sandywalsh has joined #openstack-infra | 16:50 | |
| *** jlibosva has quit IRC | 16:51 | |
| jeblair | https://launchpad.net/heat/+milestone/icehouse-rc1 (f | 16:52 |
| jeblair | https://launchpad.net/heat/+milestone/icehouse-rc1 (for referenceà | 16:52 |
| *** openstackstatus has quit IRC | 16:52 | |
| jeblair | haha! | 16:52 |
| jeblair | that was it | 16:52 |
| jeblair | ttx: killed statusbot earlier with a unicode error. :) | 16:52 |
| ttx | nice | 16:53 |
| jraim | sdague: so poking at this a bit more, we are doing an 'it's running test' here https://github.com/stackforge/barbican/blob/master/functionaltests/run_tests.sh | 16:54 |
| jraim | sdague: it's being called from the post_test_hook | 16:54 |
| jraim | but it doesn't seem to output anything | 16:54 |
| *** dklyle has quit IRC | 16:54 | |
| jraim | we copied this from Solum...is there a way to make this type of test show up? | 16:54 |
| sdague | jraim: yeh, so outputing something would be good, so we have some idea if it worked or not | 16:54 |
| fungi | jeblair: yeah, i've got the tracebacks for a couple of ways in which encoding issues have been killing it in the open bug | 16:54 |
| jraim | sdague: okay, I'll poke at it | 16:55 |
| sdague | jraim: and honestly, can we do at least minimal payload check | 16:55 |
| fungi | jeblair: probably just filtering the input for potential encoding translation issues would solve that? | 16:55 |
| sdague | like our horizon exercise in devstack scrapes the page and looks for "Login" | 16:55 |
| sdague | to make sure the return wasn't a stack trace | 16:55 |
| jraim | seems like we could patch that in pretty easily | 16:56 |
| sdague | great | 16:57 |
| ttx | jeblair: hold steady, potential nova reset | 17:00 |
| *** reaper has joined #openstack-infra | 17:00 | |
| *** jgrimm has quit IRC | 17:00 | |
| jeblair | ttx: i don't think the zuul change has merged | 17:00 |
| *** jgrimm has joined #openstack-infra | 17:00 | |
| *** jgrimm has quit IRC | 17:00 | |
| ttx | sacrificing a chicken to zuul | 17:00 |
| ttx | although damage is limited, there was a reset two down | 17:01 |
| fungi | ttx: if the repack and pack-refs i did on review's /var/lib/git mirror a little while ago helps, then maybe it will replicate quickly enough | 17:01 |
| ttx | I think it passed | 17:01 |
| *** mriedem1 has joined #openstack-infra | 17:03 | |
| *** dkliban has quit IRC | 17:03 | |
| fungi | in unrelated news, the resize2fs for the logs.openstack.org filesystem just finished, so we have a few more terabytes of breathing room which should carry us through at least a couple more months | 17:03 |
| *** rwsu has joined #openstack-infra | 17:03 | |
| fungi | right now we're holding onto about 8.5tib of job logs from the past 6 months | 17:04 |
| *** markmcclain1 has joined #openstack-infra | 17:04 | |
| fungi | craaaazy | 17:04 |
| StevenK | fungi: We used to have those discussions about the Launchpad Librarian. "We have 1.5T of free space, which will hold us for 1.5 weeks." | 17:04 |
| *** mrodden has joined #openstack-infra | 17:05 | |
| *** mriedem has quit IRC | 17:05 | |
| fungi | StevenK: yeah, i just expanded the logical volume for it to 11tib | 17:05 |
| *** sabari2 has joined #openstack-infra | 17:05 | |
| StevenK | When I left, we had 16TiB for the Librarian | 17:05 |
| ttx | fungi: what does the SKIPPED mean on the cinder change in slot #6 at the gate right now ? | 17:05 |
| fungi | StevenK: we can push to about 12.5 tib before we hit the kernel/xen limit on the number of virtual block devices which can be added to a guest | 17:05 |
| ttx | fungi: is it a sure FAIL at this point ? | 17:06 |
| fungi | ttx: hover over the colored dot next to it | 17:06 |
| *** markmcclain has quit IRC | 17:06 | |
| ttx | fungi: OK, so that one will have to go to the bottom of queue again, right | 17:06 |
| ttx | just trying to keep track of what may land and what i should be waiting on | 17:06 |
| *** amotoki has quit IRC | 17:06 | |
| fungi | ttx: it probably merge-conflicts with another cinder change which merged ahead of it in the gate after the last time it had check jobs run | 17:07 |
| *** SumitNaiksatam has quit IRC | 17:07 | |
| ttx | ok | 17:07 |
| *** harlowja has joined #openstack-infra | 17:07 | |
| ttx | that means i can cut cinder now | 17:07 |
| *** hogepodge has joined #openstack-infra | 17:07 | |
| fungi | ttx: so it will need a rebase, then it will get rechecked, then it will need to be approved all over again | 17:07 |
| *** jcoufal has joined #openstack-infra | 17:07 | |
| *** esker has joined #openstack-infra | 17:07 | |
| ttx | fungi: ok, was just wondering why it did not return the failure yet | 17:07 |
| *** e0ne_ has quit IRC | 17:08 | |
| ttx | since by now it can't be saved | 17:08 |
| fungi | ttx: zuul isn't currently smart enough to pay attention to the fact that it now merge-conflicts with the branch tip, and eject it immediately | 17:08 |
| ttx | okidoki | 17:08 |
| fungi | ttx: i think it's on the to-do list | 17:09 |
| *** zhiwei has quit IRC | 17:09 | |
| fungi | (somewhere) | 17:09 |
| *** esker has quit IRC | 17:09 | |
| *** esker has joined #openstack-infra | 17:09 | |
| *** sabari3 has joined #openstack-infra | 17:09 | |
| *** sabari2 has quit IRC | 17:10 | |
| *** jcooley_ has joined #openstack-infra | 17:12 | |
| zaro | sdague: working on BUILD_TIMEOUT env var -> https://github.com/jenkinsci/build-timeout-plugin/pull/20 | 17:12 |
| *** sabari3 is now known as sabari | 17:13 | |
| *** dangers_away is now known as dangers | 17:13 | |
| *** dizquierdo has quit IRC | 17:13 | |
| *** esker has quit IRC | 17:14 | |
| *** derekh has joined #openstack-infra | 17:14 | |
| *** dprince has joined #openstack-infra | 17:15 | |
| *** amcrn has joined #openstack-infra | 17:15 | |
| *** chandan_kumar has quit IRC | 17:16 | |
| *** browne has joined #openstack-infra | 17:16 | |
| *** amcrn has quit IRC | 17:17 | |
| *** browne has left #openstack-infra | 17:17 | |
| *** cadenzajon_ has quit IRC | 17:17 | |
| *** amcrn has joined #openstack-infra | 17:18 | |
| *** krtaylor has quit IRC | 17:23 | |
| *** reed has joined #openstack-infra | 17:23 | |
| *** KurtMartin is now known as kmartin | 17:24 | |
| *** CaptTofu has quit IRC | 17:25 | |
| *** CaptTofu has joined #openstack-infra | 17:26 | |
| *** whoops has joined #openstack-infra | 17:28 | |
| *** jcooley_ has quit IRC | 17:28 | |
| *** krotscheck has joined #openstack-infra | 17:30 | |
| *** krtaylor has joined #openstack-infra | 17:30 | |
| *** CaptTofu has quit IRC | 17:30 | |
| *** melwitt has quit IRC | 17:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:32 | |
| *** openstackgerrit has quit IRC | 17:32 | |
| *** zns has quit IRC | 17:33 | |
| *** jpich has quit IRC | 17:33 | |
| *** vkozhukalov has joined #openstack-infra | 17:34 | |
| *** dkliban has joined #openstack-infra | 17:34 | |
| *** dims has quit IRC | 17:35 | |
| *** krotscheck has quit IRC | 17:36 | |
| *** esker has joined #openstack-infra | 17:36 | |
| clarkb | morning | 17:36 |
| pleia2 | morning | 17:37 |
| *** dims has joined #openstack-infra | 17:37 | |
| *** blamar has joined #openstack-infra | 17:39 | |
| *** bhuvan_ has joined #openstack-infra | 17:39 | |
| *** bhuvan has joined #openstack-infra | 17:39 | |
| *** beagles is now known as beagles_brb | 17:44 | |
| jogo | who is the pypy expert around these parts again? | 17:45 |
| clarkb | Alex_Gaynor: ^ | 17:45 |
| Alex_Gaynor | What's up | 17:46 |
| jogo | Alex_Gaynor: oh right thanks | 17:46 |
| jogo | Alex_Gaynor: I just commented on [openstack-dev] [neutron][rootwrap] Performance considerations, sudo? | 17:46 |
| Alex_Gaynor | jogo: I'll take a look, I don't read -dev regularly | 17:47 |
| jogo | Alex_Gaynor: want to make sure I didn't get anything wrong in http://lists.openstack.org/pipermail/openstack-dev/2014-March/029057.html | 17:47 |
| Alex_Gaynor | jogo: So we don't generally reccomend RPython for anything besides writing interpreter, it's a slightly prickly language; but rootwrap is relatively constrained, so it actually might make sense | 17:48 |
| *** sabari has quit IRC | 17:49 | |
| *** krotscheck has joined #openstack-infra | 17:49 | |
| jogo | [Btly m | 17:49 |
| *** hashar has quit IRC | 17:50 | |
| *** Sukhdev has quit IRC | 17:50 | |
| jogo | Alex_Gaynor: exactly (ignore my ssh timeout above) | 17:51 |
| fungi | ...NO CARRIER | 17:51 |
| Alex_Gaynor | jogo: I imagine the biggest issue that will creep up is that almost all of the stdlib is not accessible in RPython | 17:51 |
| *** bhuvan_ has quit IRC | 17:52 | |
| *** bhuvan has quit IRC | 17:52 | |
| jogo | Alex_Gaynor: http://git.openstack.org/cgit/openstack/oslo.rootwrap/tree/oslo/rootwrap | 17:53 |
| jogo | if you know a little RPython ( I don't) you can take a look at and see for yourself | 17:53 |
| *** mrodden has quit IRC | 17:53 | |
| Alex_Gaynor | logging is probably the biggest thing that jumps out; rpython has no logging library | 17:54 |
| jogo | that shouldn't be a big usse | 17:56 |
| clarkb | fungi: jeblair: catching up on sb. Is run-unittests still broken? that isn't entirely clear to me (I can hack on a fix if it needs it). And did we make a deicsion on what to do with zuul? | 17:57 |
| *** Sukhdev has joined #openstack-infra | 17:57 | |
| fungi | clarkb: weigh in on https://review.openstack.org/78262 for run-unittests.sh fixing | 17:58 |
| jeblair | clarkb: we decided to increase the timeout in zuul, but the change hasn't merged yet | 17:58 |
| *** esker has quit IRC | 17:59 | |
| fungi | clarkb: decision on zuul is https://review.openstack.org/78222 plus a restart at the soonest opportunity, then either https://review.openstack.org/78277 or a fancier approach not yet fleshed-out | 17:59 |
| *** jgallard has quit IRC | 17:59 | |
| *** CaptTofu has joined #openstack-infra | 18:00 | |
| *** nati_ueno has joined #openstack-infra | 18:02 | |
| *** thuc has quit IRC | 18:02 | |
| clarkb | and it looks like there are window related races because changes behind will pull tip of master which updates post merge when the window shifts? | 18:02 |
| clarkb | that is a fun one | 18:02 |
| *** Ryan_Lane1 has joined #openstack-infra | 18:02 | |
| *** beagles_brb is now known as beagles | 18:02 | |
| fungi | though we basically already have that now, we just haven't seen it in action i don't think | 18:02 |
| *** thuc has joined #openstack-infra | 18:02 | |
| clarkb | roger | 18:03 |
| fungi | at least haven't witnessed it first hand anyway | 18:03 |
| *** khyati has joined #openstack-infra | 18:03 | |
| *** resker has joined #openstack-infra | 18:03 | |
| clarkb | I have no idea why that set -e exists | 18:04 |
| * clarkb looks at git logs | 18:04 | |
| fungi | i should have done a git blame on it | 18:04 |
| clarkb | well logs are screwy there because that file was renamed and lost that check | 18:05 |
| clarkb | then I had to readd it again later iirc | 18:05 |
| jogo | Alex_Gaynor: so your this might work, would be appreacted on the ML | 18:05 |
| clarkb | fungi: there was no set -e in the old run-tox.sh portion that did a similar check | 18:07 |
| mgagne | Could someone give a kick to gerrit bot? | 18:07 |
| *** fbo is now known as fbo_away | 18:07 | |
| fungi | 1732 <-- openstackgerrit (~openstack@review.openstack.org) has quit (Ping timeout: 244 seconds) | 18:08 |
| fungi | fun | 18:08 |
| fungi | restarting it | 18:08 |
| mgagne | fungi: thanks! | 18:08 |
| clarkb | fungi: ff91e70e4251ec2740fa000f3ada250f83b87dab in config | 18:09 |
| fungi | looks like gerritbot is still running... | 18:09 |
| fungi | gerrit2 19137 0.1 0.0 313684 20076 ? Sl Mar04 4:23 /usr/bin/python /usr/local/bin/gerritbot /etc/gerritbot/gerritbot.config | 18:09 |
| clarkb | fungi: so I think we should save $? and can check that return code against 0 in the same check for 0 tests | 18:09 |
| clarkb | sdague: ^ | 18:09 |
| fungi | strace says "futex(0x18011c0, FUTEX_WAIT_PRIVATE, 0, NULL" and just sits there | 18:09 |
| fungi | having a look in the logs | 18:10 |
| clarkb | fungi: however that may only be necessary for the old foundcount variable | 18:10 |
| *** zns has joined #openstack-infra | 18:11 | |
| clarkb | fungi: I have +2'd sdague's change. Inline comment explains why I Think it is ok | 18:11 |
| clarkb | that said | 18:11 |
| clarkb | how did set -e trip if that worked as expected?" | 18:12 |
| fungi | clarkb: okie dokie. i will likely follow suit once i read it | 18:12 |
| clarkb | gah I should think these things before I +2 | 18:12 |
| clarkb | fungi: hold on | 18:12 |
| clarkb | fungi: why caused set -e to errexit in the first place? | 18:12 |
| clarkb | we probably need to guard against that | 18:12 |
| clarkb | fungi: oh wait I get it, errexit happened later | 18:12 |
| clarkb | so ya +2 | 18:12 |
| jogo | sdague: have you been staring at the gate recently? | 18:12 |
| fungi | here's the last thing i find in gerritbot's log... http://paste.openstack.org/show/72671/ | 18:13 |
| fungi | seeing if there's anything else more enlightening in the debug log | 18:13 |
| clarkb | I hear a mordred around the corner | 18:13 |
| jomara | fungi: i think everything is good with my set of patches EXCEPT a mistake i made merging patch 4 (of 5); i need to fix it and reupload (just a conflict resultion error), but *WHAT* should i run git review on? P5, with P4 correct in its history? | 18:13 |
| jomara | or just p4 | 18:14 |
| fungi | jomara: if you still have the topic branch you were using, just rebase -i <commit_before_the_one_you_want_to_change> | 18:14 |
| fungi | jomara: then change the "pick" next to the change you need to modify to "edit" instead | 18:15 |
| fungi | jomara: then make the modifications you need to that change and do git commit -a followed by git rebase --continue | 18:15 |
| fungi | jomara: after that you can 'git log -p' to see whether the changes look the way you intend | 18:16 |
| jomara | fungi: oh awesome | 18:16 |
| *** Ryan_Lane1 has quit IRC | 18:16 | |
| *** bhuvan_ has joined #openstack-infra | 18:16 | |
| *** bhuvan has joined #openstack-infra | 18:16 | |
| *** sabari has joined #openstack-infra | 18:17 | |
| fungi | jomara: and if they look right, run 'git review' and say yes to the multi-change prompt. it will list all the changes which are still outstanding but will actually only update gerrit with the ones you changed (the edited change and any after it) | 18:17 |
| fungi | jomara: that will properly preserve the dependency chain in gerrit | 18:17 |
| *** malini has joined #openstack-infra | 18:18 | |
| jomara | fungi: wow! excellent | 18:19 |
| fungi | jomara: in general, git rebase -i is a great tool for safely adjusting the order or content of changes in a topic branch (you can even squash, remove or insert changes that way too) | 18:19 |
| clarkb | just be careful you don't end up with na extra goto fail >_> | 18:19 |
| *** Ryan_Lane1 has joined #openstack-infra | 18:19 | |
| fungi | jomara: experiment with it when you get time, but yeah it's advanced git voodoo | 18:20 |
| jomara | that would probably would have been superior for my method of "rebasing" | 18:20 |
| jomara | had i done htat from the start | 18:20 |
| fungi | jomara: as with any git voodoo, probably best to first get used to using git stash, git reflog and git reset | 18:21 |
| fungi | since those are the best way to save your bacon when you mess up | 18:21 |
| jomara | fungi: to be 100% clear so i dont make mistakes, i rebase -i from the TOP commit, or from the commit in question(that needs edit) | 18:21 |
| clarkb | fungi: re gerritbot, should we catch ServerNotConnected in run() and just reconnect if that happens? | 18:21 |
| fungi | clarkb: i think that would be a good idea | 18:21 |
| *** mriedem has joined #openstack-infra | 18:22 | |
| clarkb | fungi: I don't think there is much reason to try and be more graceful about it nor do I think we can be more graceful | 18:22 |
| fungi | jomara: rebase on the parent sha of the commit you want to change | 18:22 |
| malini | hello..I am trying to add a new job for marconi in devstack's experimental queue. can somebody point me to a good example? | 18:22 |
| *** gokrokve has quit IRC | 18:22 | |
| clarkb | malini: will this job need to do things like patch devstack/tempest before the job runs? | 18:22 |
| clarkb | s/before the job/before the tests/ | 18:23 |
| *** mriedem1 has quit IRC | 18:23 | |
| fungi | jomara: also, read all the comments and output messages you get from those git commands very carefully. they contain pertinent instructions, but they're very easy to ignore | 18:23 |
| *** reed has quit IRC | 18:23 | |
| *** reed has joined #openstack-infra | 18:23 | |
| fungi | git is quite happy to warn you about how to proceed and then let you shoot yourself in the foot when you don't heed its warnings | 18:24 |
| *** Ryan_Lane has quit IRC | 18:24 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 18:24 | |
| *** Ryan_Lane has quit IRC | 18:24 | |
| *** Ryan_Lane has joined #openstack-infra | 18:24 | |
| *** Ryan_Lane1 has joined #openstack-infra | 18:24 | |
| malini | clarkb: We already have an experimental job that runs tempest on devstack. I am looking for something that can be used to see the impact, on any devstack changes that I make | 18:25 |
| malini | hope tht makes sense | 18:26 |
| *** openstackgerrit has joined #openstack-infra | 18:26 | |
| clarkb | malini: it does, but to find a good example I need to know if you are going to run stock devstack with stock tempest or if you need to manipulate those projects before running tests | 18:26 |
| *** jcooley_ has joined #openstack-infra | 18:27 | |
| fungi | clarkb: also worth looking into for gerritbot, i think its initscript may not be looking in the right place for the pidfile because stop and restart always complain that no pidfile was found | 18:27 |
| malini | clarkb: when you say stock devstack, will it include changes in any proposed patch? | 18:27 |
| clarkb | malini: yes | 18:28 |
| malini | clarkb: & what would be an example of manipulating the project? | 18:28 |
| jomara | fungi: ACK, thanks | 18:28 |
| clarkb | malini: some projects add stuff to devstack/lib/projectname since devstack won't carry those things in tree | 18:29 |
| clarkb | malini: others add tests to tempest that tempset won't carry in tree | 18:29 |
| fungi | clarkb: actually, skimming the initscript it does seem to be checking the right path for the pidfile, and the current pidfile contains the pid of the process, so maybe something is disappearing the pidfile. i'll try to remember to check for it *before* restarting next time | 18:29 |
| clarkb | fungi: ok | 18:29 |
| clarkb | pypi is down to its static mirror | 18:30 |
| clarkb | says all systems operational though | 18:30 |
| malini | clarkb: Sounds like stock tempest + stock devstack is all I need, | 18:30 |
| *** resker has quit IRC | 18:30 | |
| fungi | clarkb: pypi.python.org? | 18:30 |
| clarkb | dstufft: ^ you are probably aware of anything going on but if you aren't | 18:30 |
| clarkb | fungi: ya | 18:30 |
| clarkb | malini: great, in that case your job probably just needs to enable and disable specific features? | 18:31 |
| *** johnthetubaguy has quit IRC | 18:31 | |
| *** esker has joined #openstack-infra | 18:31 | |
| clarkb | that already exist in devstack/tempest | 18:31 |
| dstufft | hm | 18:31 |
| fungi | i *could* pull up my maildir for the python packaging list, but EEFFORT | 18:31 |
| dstufft | pingdom hasn't said anything yet | 18:31 |
| malini | clarkb: yes..I need to enable marconi | 18:32 |
| *** amcrn_ has joined #openstack-infra | 18:32 | |
| *** amcrn has quit IRC | 18:32 | |
| *** jcoufal_ has joined #openstack-infra | 18:33 | |
| malini | clarkb: some background.. marconi is not already part of tempest..We are seeing some weird issues in tempest-marconi experimental jobs. it was suggested tht having an experimental job in devstack queue for marconi will make troubleshooting easier | 18:33 |
| *** jcoufal has quit IRC | 18:33 | |
| dstufft | ah there's pingdom | 18:33 |
| clarkb | malini: openstack-infra/config c515a55a2d40d87510b3b6d10915b426de8f5c5d is probably pretty close | 18:33 |
| dstufft | having a service that lots of people depend on is way more responsive than pingdom | 18:34 |
| clarkb | malini: that is ironics change to add tests | 18:34 |
| *** Ryan_Lane1 has quit IRC | 18:34 | |
| clarkb | dstufft: glad I could help :) | 18:34 |
| *** Ryan_Lane1 has joined #openstack-infra | 18:34 | |
| malini | clarkb: thanks a lot!! appreciate your help! | 18:34 |
| *** resker has joined #openstack-infra | 18:34 | |
| *** esker has quit IRC | 18:35 | |
| fungi | dstufft: we keep trying to tell people that too, but they seem to think setting up an actual monitoring system would be less embarrassing ;) | 18:35 |
| markmcclain1 | would it be possible to get a review promoted on next reset? | 18:36 |
| markmcclain1 | https://review.openstack.org/#/c/78077/ | 18:36 |
| fungi | markmcclain1: probably. what are the reasons? | 18:36 |
| clarkb | dstufft: its back | 18:36 |
| clarkb | or at least useable | 18:36 |
| dstufft | fungi: I don't think we've ever had the monitoring tell us before someone else did | 18:36 |
| fungi | markmcclain1: looks like it's probably a stability fix? | 18:36 |
| markmcclain1 | fungi: we encounter a race, so added better error handling | 18:36 |
| dstufft | yea one our PGpool nodes restarted and when it came back up it got confused | 18:36 |
| markmcclain1 | fungi: yep | 18:36 |
| dstufft | we're bypassing the pool right now and connecting directly to the DB | 18:37 |
| fungi | markmcclain1: right on. in it goes on the next reset | 18:37 |
| markmcclain1 | fungi: thanks | 18:37 |
| pleia2 | fungi: could use your security brain for a chat at some point re: https://review.openstack.org/#/c/60375/ | 18:37 |
| pleia2 | fungi: modules/openstack_project/files/git/bin/about-formatting.sh is kind of scary and awful, and selinux will never love it | 18:37 |
| fungi | pleia2: i left it in my other jacket, but i'll see what i can do | 18:38 |
| *** ildikov_ has quit IRC | 18:38 | |
| pleia2 | :) | 18:38 |
| pleia2 | fungi: I'm wondering if we should come up with a saner solution than what ships with cgit (about-formatting.sh is slightly modified from upstream) or just live with it and turn off selinux | 18:38 |
| pleia2 | we could also allow apache user to execute bash scripts, but at that point we might as well turn off selinux because all is lost | 18:39 |
| clarkb | fungi: so we already have reconnect logic in the bot | 18:40 |
| clarkb | fungi: I am going to dig in logs more | 18:40 |
| fungi | clarkb: okay | 18:40 |
| malini | clarkb: what do jobs like 'gate-climate-devstack-dsvm' do? | 18:40 |
| malini | clarkb: just enable climate in devstack ? | 18:40 |
| clarkb | malini: they use a pre test hook to patch devstack, which is then run and if successful the devstack exercises are run last | 18:41 |
| clarkb | malini: which is why I asked if you would need to patch devstack or tempest because if you do then the test hook stuff becomes important | 18:41 |
| malini | clarkb: what is a test hook? | 18:42 |
| fungi | markmcclain1: if you're lucky and there's no gate reset on the 9 changes being tested ahead of it, 78077,2 is estimated to land in about 20 minutes anyway. but i'll bump it ahead if we do get a reset before then | 18:42 |
| markmcclain1 | fungi: a little luck would be good | 18:43 |
| malini | clarkb: I know thts a lot of questions..But we already have the tempest + devstack job similar to ironic | 18:43 |
| *** UtahDave has joined #openstack-infra | 18:43 | |
| malini | clarkb: But marconi crashes when tht job runs..We are trying to figure out what causes tht crash | 18:44 |
| *** bhuvan_ has quit IRC | 18:44 | |
| *** bhuvan has quit IRC | 18:44 | |
| fungi | pleia2: i'm re-reviewing 60375 now... i'll have some pointers for you here in a bit, hopefully | 18:44 |
| clarkb | malini: in the test runner framework we have a project called devstack-gate. This is the thing that configures a localrc appropriate for each job. It then runs devstack and tempest is the typical configuration. Because some projects may need to do other things like patch devstack or tempest, devstack gate has a pre test hook in which you can do stuff before devstack and tempest run | 18:44 |
| clarkb | malini: it also has a test hook if you want to do tests that are not tempest and it has a post test hook if you want to run more tests after tempest or clean up particular things or whatever | 18:44 |
| *** talluri has quit IRC | 18:44 | |
| pleia2 | fungi: thank you | 18:44 |
| clarkb | malini: they are just hooks into the various steps of running integration tests | 18:45 |
| *** talluri has joined #openstack-infra | 18:45 | |
| clarkb | fungi: aha! we need a new gerritbot release | 18:46 |
| clarkb | fungi: the retry logic isn't tagged yet | 18:46 |
| * fungi facepalms | 18:47 | |
| *** Sukhdev has quit IRC | 18:47 | |
| fungi | looking at the delta since the last tag... | 18:47 |
| fungi | oh, wow... lotsastuff | 18:48 |
| fungi | 1.5 years worth of fixes | 18:48 |
| clarkb | ya | 18:49 |
| fungi | whoever first said "release early and often" should not look at this repo :/ | 18:49 |
| *** khyati has quit IRC | 18:49 | |
| *** gema has quit IRC | 18:49 | |
| *** e0ne has joined #openstack-infra | 18:49 | |
| clarkb | fungi: now we can move onto the next thing :) | 18:49 |
| fungi | clarkb: for our own sanity, let's shoot for tagging the tip of master as something like 0.2.0 on monday | 18:49 |
| clarkb | I won't tag a release just yet, but if everyone is on board I am happy to do it | 18:50 |
| clarkb | fungi: ++ | 18:50 |
| wendar | Is logs/devstacklog.txt indexed? | 18:50 |
| malini | clarkb: thanks for the clarification! let me dig in a lil deeper on what will help us the most! | 18:50 |
| *** talluri has quit IRC | 18:50 | |
| clarkb | wendar: typically I could answer that question, but I am behind on that stuff. Let me check | 18:50 |
| *** afazekas has quit IRC | 18:50 | |
| clarkb | wendar: yes it should be | 18:51 |
| *** pblaho has quit IRC | 18:51 | |
| clarkb | looks like it is | 18:51 |
| clarkb | I have almost a quarter of a million hits for filename:"logs/devstacklog.txt" over the last 15 minutes | 18:52 |
| fungi | markmcclain1: in good news, all the neutron changes ahead of 78077,2 have completed and passed all their tests, so at least they're not going to cause the next reset (if there is one) | 18:52 |
| *** resker has quit IRC | 18:53 | |
| *** esker has joined #openstack-infra | 18:53 | |
| * clarkb moves onto catching up on zuul stuff then normal reviews | 18:54 | |
| *** rlandy has quit IRC | 18:55 | |
| *** krotscheck has quit IRC | 18:55 | |
| *** rlandy has joined #openstack-infra | 18:56 | |
| *** gema has joined #openstack-infra | 18:56 | |
| *** briancurtin has quit IRC | 18:56 | |
| malini | clarkb: what does 'dsvm' stand for? | 18:58 |
| clarkb | malini: devstack VM | 18:58 |
| *** esker has quit IRC | 18:58 | |
| clarkb | malini: its an string we can key off of to make sure we do bookkeeping around those machiens properly | 18:58 |
| SpamapS | ohai.. I'm wondering if we can have https://review.openstack.org/78310 bumped in the zuul queue (once it passes checks) ... The bug it fixes is causing all of TripleO CI to fail. | 18:58 |
| fungi | clarkb: or at least was, but now that single-use is default, it's not really as important as it was | 18:59 |
| wendar | clarkb: great, thanks! | 18:59 |
| *** jcooley_ has quit IRC | 18:59 | |
| *** skraynev_afk is now known as skraynev | 18:59 | |
| clarkb | fungi: good point | 18:59 |
| malini | clarkb: thanks! now that I know it, it should have been obvious :) | 18:59 |
| *** jeremyb has quit IRC | 18:59 | |
| *** jcooley_ has joined #openstack-infra | 18:59 | |
| fungi | malini: we used to spell it out, but job names got way, way, way too long | 18:59 |
| fungi | SpamapS: 78310 isn't going to make it into the gate until 77920 does | 19:00 |
| fungi | SpamapS: see gerrit change dependencies there | 19:01 |
| sdague | jogo: just got back from lunch, what's the question? | 19:01 |
| sdague | wendar: yes, logs/devstacklog.txt starting getting indexed last week some time | 19:01 |
| *** dripton_ is now known as dripton | 19:02 | |
| clarkb | sdague: I am going to jinx it, but the cluster seems to have kept up after that initial slowdown on monday. I added a few additional gearman workers but other than that haven't touched it | 19:02 |
| *** openstackgerrit has quit IRC | 19:02 | |
| wendar | sdague: thanks! | 19:03 |
| *** nati_uen_ has joined #openstack-infra | 19:03 | |
| clarkb | sdague: I think we are just very near the event horizon when things realy pick up. I will spend more cycles on autoscaling those nodes when I have time to sit and hack on stuff | 19:03 |
| jogo | sdague: gate backed up 15 hours, have you been looking at it? if not thats fine | 19:03 |
| jogo | I am working with wendar on it | 19:03 |
| fungi | pleia2: looking at 60375, i don't immediately see why the selinux rules for it would be particularly challenging to write... should just be able to allow it to run /usr/local/bin/rst2html and things in /usr/libexec/cgit/filters/html-converters right? | 19:03 |
| sdague | jogo: so realistically, a big amount of our reset is the zuul git merge timeout | 19:03 |
| *** jcooley_ has quit IRC | 19:04 | |
| *** jcooley_ has joined #openstack-infra | 19:04 | |
| sdague | after that, it's mostly the normal races that we hit | 19:04 |
| sdague | a few neutron ones | 19:04 |
| sdague | grenade issue where services don't shut down on old side | 19:04 |
| *** svpress has joined #openstack-infra | 19:05 | |
| fungi | sdague: jogo: and i should check the zuul logs to see if it's happened since i did a repack and pack-refs on the gerrit local mirror | 19:05 |
| sdague | fungi: yeh, that would be cool | 19:05 |
| sdague | jogo: though the classifaction rate has dropped a lot - http://status.openstack.org/elastic-recheck/data/uncategorized.html | 19:05 |
| *** jcooley_ has quit IRC | 19:05 | |
| fungi | it may no longer be biting us | 19:05 |
| *** mrodden has joined #openstack-infra | 19:05 | |
| sdague | so there may be some new issues, or moving around issues | 19:05 |
| sdague | fungi: definitely possible | 19:05 |
| sdague | I think it was a big reason for not much overnight progress though | 19:06 |
| pleia2 | fungi: so the trouble I'm running into is that about-formatting.sh is calling bash to run those, which selinux isn't happy with (I can run it again and grab the selinux errors) | 19:06 |
| fungi | pleia2: that might help narrow it down | 19:06 |
| sdague | jogo: but honestly, ignore everything older than the 21st | 19:06 |
| *** jeremyb has joined #openstack-infra | 19:06 | |
| *** nati_ueno has quit IRC | 19:06 | |
| pleia2 | fungi: ok, will do | 19:06 |
| sdague | because we were in a weird place on log indexing then | 19:07 |
| *** zns has quit IRC | 19:07 | |
| *** zns has joined #openstack-infra | 19:07 | |
| fungi | pleia2: one (probably crazy) option would be to run a shell compiler on it at install time, but that's really just papering over the underlying issue i think | 19:07 |
| jogo | sdague: ack, wendar has a patch to ignore old hits | 19:09 |
| *** packet has joined #openstack-infra | 19:09 | |
| sdague | right, I was +2 on that | 19:09 |
| sdague | so you should approve it :) | 19:09 |
| *** zns has quit IRC | 19:09 | |
| jogo | done | 19:10 |
| *** zns has joined #openstack-infra | 19:10 | |
| fungi | markmcclain1: it merged on its own without needing help | 19:11 |
| *** mbacchi has quit IRC | 19:11 | |
| clarkb | fungi: woot | 19:11 |
| markmcclain1 | fungi: awesome | 19:11 |
| sdague | jogo: we actually started at 75 at the beginning of the day | 19:11 |
| clarkb | 74144 is an unhappy change though | 19:11 |
| sdague | so the gate is shrinking | 19:11 |
| sdague | which is good | 19:11 |
| *** khyati has joined #openstack-infra | 19:12 | |
| *** mrodden has quit IRC | 19:12 | |
| sdague | markmcclain1: whats up with neutron patch failing in the gate? | 19:13 |
| sdague | alembic explode? | 19:13 |
| fungi | sdague: first guess is that it functionally conflicts with one which was approved ahead of it | 19:13 |
| sdague | it's alembic | 19:13 |
| clarkb | fungi: thinking about the window race. Wouldn't the merger already have the refs for the things that merged so that the zuul ref would appropriately be based on that? | 19:13 |
| fungi | but doesn't merge-conflict | 19:13 |
| wendar | clarkb: 74144, 77927, 74156, and 67862 are all the same error pattern | 19:14 |
| fungi | clarkb: any zuul ref would be based on that, sure. it's non-zuul refs i'm concerned with (integration tests pulling from our git.o.o farm) | 19:14 |
| clarkb | fungi: actually no because we may have any number of mergers | 19:14 |
| clarkb | fungi: right but those don't matter | 19:14 |
| markmcclain1 | sdague: yep.. we had a race for a down revision | 19:14 |
| fungi | clarkb: why don't they matter? | 19:14 |
| clarkb | fungi: because any project that was in gating should be updated via zuul refs | 19:14 |
| clarkb | fungi: anything else has been static from zuul's perspective | 19:15 |
| sdague | that's something we probably need a better resolution on doing before moving all the other projects to alembic, because in "theory" it supports this nice dependency graph, but in practice it gets funky when more than one in flight | 19:15 |
| fungi | clarkb: _integration_ tests | 19:15 |
| clarkb | fungi: right | 19:15 |
| *** mbacchi has joined #openstack-infra | 19:15 | |
| markmcclain1 | sdague: longer term I'm thinking we might need to make a prep script that auto sequences the migrations | 19:15 |
| clarkb | fungi: lets say we are testing nova nova cinder swift | 19:15 |
| *** mrodden has joined #openstack-infra | 19:15 | |
| fungi | clarkb: the change at issue may not be for the project being tested, or for any currently in the queue ahead of it | 19:15 |
| clarkb | fungi: glance won't matter because it was never changed | 19:15 |
| markmcclain1 | sdague: unless the patchset declares something else specific | 19:15 |
| sdague | markmcclain1: yeh, that makes sense | 19:15 |
| sdague | ok, I'm going to hide from computers for a bit. I was up super early this morning. | 19:16 |
| clarkb | fungi: we are only concered with nova cinder and swift, and since they have been involved in the merger their refs should be up to date. However this may not be true with multiple mergers | 19:16 |
| clarkb | fungi: now say the first two nova changes merge and we add a neutron change. then we have cinder swift nova. they should still get the proper nova refs because the zuul ref for nova will exist right? | 19:17 |
| jogo | sdague: we are talking about neutron issues in neutron room | 19:17 |
| clarkb | maybe thats the bit I am missing, I thought zuul refs are created for projects as long as they were in the gate at some point | 19:17 |
| clarkb | fungi: and maybe ^ is the fix if they don't | 19:17 |
| sdague | jogo: ok, well I'll let you handle it, I'm going to run away for a bit :) | 19:17 |
| fungi | clarkb: what if the change which just merged was for glance, and fixes an integration bug, then the window slides to start testing a swift change (and there are no swift changes ahead of it) and it pulls glance from the git.o.o mirror because there are no zuul refs for glance | 19:17 |
| *** markwash has joined #openstack-infra | 19:17 | |
| clarkb | fungi: I think htere should be a zuul ref for glance | 19:17 |
| clarkb | fungi: bceause glance was in the pipeline prior | 19:18 |
| jogo | sdague: sounds good, well I really just showing wendar how to take care of it | 19:18 |
| sdague | coolio | 19:18 |
| fungi | clarkb: are you suggesting that we add that capability to zuul? i don't believe it currently adds a zuul ref for projects which don't have changes ahead of your change | 19:18 |
| clarkb | fungi: I think zuul already does this | 19:18 |
| clarkb | fungi: but only in the case wheer that project had a change processed at some point in the running memory of zuul | 19:19 |
| clarkb | if you haven't been tested since the last restart it won't do that | 19:19 |
| clarkb | if however zuul doesn't have this behavior then I guess I am proposing this is the way zuul deal with the race | 19:19 |
| clarkb | this is easy enough to check, /me waits for a window shift | 19:20 |
| fungi | clarkb: git refs are built for the change being tested... so i don't believe it will build a ref for the change being tested on another project which has no changes ahead in the queue, but i could of course be wrong | 19:21 |
| fungi | the setup-workspace.log would have far fewer fallbacks to teh git farm if theat were really the case | 19:21 |
| *** MarkAtwood has joined #openstack-infra | 19:21 | |
| *** svpress has quit IRC | 19:22 | |
| clarkb | fungi: you are correct | 19:23 |
| clarkb | git fetch http://zm01.openstack.org/p/openstack/requirements refs/zuul/master/Za2b6a99161fb4dceb2d336d2ff55fca4 fails | 19:23 |
| jeblair | clarkb: zuul builds refs for all projects ahead of (and of course including) the changes in the queue | 19:23 |
| *** briancurtin has joined #openstack-infra | 19:23 | |
| jeblair | clarkb: at the time it builds the refs | 19:23 |
| *** gokrokve has joined #openstack-infra | 19:23 | |
| clarkb | jeblair: ooohhhhh now I see why the window is important | 19:23 |
| jeblair | clarkb: so in fungi's situation, the refs for change #20 in the queue, just after the queue has shifted up, do not include the project that just popped off the queue cause it merged | 19:23 |
| *** coolsvap has quit IRC | 19:24 | |
| jeblair | clarkb: yeah, it actually forces the timing of the situation to be near worst-case | 19:24 |
| fungi | clarkb: technically we could also hit it without the window behavior, but it increases the chances due to synchronization | 19:24 |
| fungi | that | 19:24 |
| clarkb | ya | 19:24 |
| fungi | sdague: i repacked the gerrit local mirror at 16:29 and since then we hit the replication timeout once, https://review.openstack.org/78028 | 19:27 |
| fungi | so that's one occurrence in 3 hours. not great, but not terrible | 19:27 |
| jeblair | fungi: i don't expect repacking to help significantly; that's why i did those timing tests yesterday | 19:27 |
| fungi | also, very tiny sample size, so ymmv | 19:28 |
| jeblair | the timeout change has propogated so we can restart at will | 19:29 |
| *** bhuvan has joined #openstack-infra | 19:29 | |
| fungi | probably best to at least let these next 6 which are about to merge make it in... none are nova changes | 19:29 |
| *** bhuvan_ has joined #openstack-infra | 19:29 | |
| clarkb | there are about 4 changes that can potentially merge shortly | 19:29 |
| *** ociuhandu has quit IRC | 19:29 | |
| clarkb | maybe after those we do the zuul restart? | 19:29 |
| clarkb | fungi: you win | 19:29 |
| clarkb | :P | 19:30 |
| fungi | or alternatively wait for the next reset | 19:30 |
| clarkb | nah everything behind those that may merge have an unknown ttm | 19:30 |
| *** rlandy has quit IRC | 19:30 | |
| *** krotscheck has joined #openstack-infra | 19:31 | |
| fungi | okay, the gate just did some very weird reshuffling and restarted testing on things | 19:31 |
| devananda | irc bots down? | 19:31 |
| hub_cap | devananda: that or they dislike u | 19:31 |
| hub_cap | id say its 50/50 | 19:32 |
| clarkb | fungi: jeblair: I think now is good zuul restart opportnity | 19:32 |
| jeblair | i'll restart now | 19:32 |
| fungi | oh yay! 1902 <-- openstackgerrit (~openstack@review.openstack.org) has quit (Ping timeout: 240 seconds) | 19:32 |
| fungi | jeblair: clarkb yes, good time for a zuul restart | 19:32 |
| jeblair | it is stopped | 19:32 |
| *** zns has quit IRC | 19:32 | |
| *** jcooley_ has joined #openstack-infra | 19:33 | |
| jeblair | i'm attempting to set used nodes to delete | 19:33 |
| fungi | though figuring out what that was it just did while it was merging those last few changes would be nice. it seemed to spontaneously start retesting changes behind others it merged | 19:33 |
| clarkb | fungi: I think the ceilomter change failed | 19:34 |
| jeblair | starting zuul | 19:35 |
| jeblair | i only managed to change about 75 nodes to delete due to row locks | 19:35 |
| jeblair | still it's something | 19:35 |
| fungi | clarkb: if it did, then why didn't it restart all the changes immediately behind it? seemed it merged several changes after the ceilo change | 19:35 |
| clarkb | fungi: the failing neutron changes were in that gap | 19:36 |
| clarkb | could be a change after that gap failed too | 19:36 |
| clarkb | but I thought those had completed | 19:36 |
| fungi | clarkb: the one right before 77719,1 was a tempest change | 19:36 |
| clarkb | jeblair: so process is dump queues with zuul tool, stop zuul, use nodepool to delete used nodes, start zuul, use tool to rebuild quues? | 19:36 |
| fungi | but it restarted tests on changes further down before it was done reflecting that several changes ahead of the tempest change merged. i'm digging in the debug log for enlightenment | 19:37 |
| jeblair | clarkb: yeah. nodepool is optional, it's just an optimization | 19:37 |
| jeblair | clarkb: then restart zuul mergers because they have a bug | 19:37 |
| clarkb | oh right | 19:37 |
| jeblair | which i have also just done | 19:37 |
| *** ildikov_ has joined #openstack-infra | 19:40 | |
| *** jcoufal_ has quit IRC | 19:41 | |
| clarkb | jeblair: is that bug something you have debugged yet? would you consider it relatively important to fix (eg should I take a look at it?) | 19:43 |
| jeblair | clarkb: i have not, and it's probably relatively important. it's certainly annoying. | 19:43 |
| *** rlandy has joined #openstack-infra | 19:43 | |
| jeblair | clarkb: it might be in the zuul merger server, or it might be gearman | 19:43 |
| *** nicedice has joined #openstack-infra | 19:43 | |
| jeblair | clarkb: it only re-registers one of the 2 functions it has on reconnection | 19:44 |
| jeblair | s/gearman/gear/ | 19:44 |
| fungi | first breadcrumb... 2014-03-05 19:30:00,991 DEBUG zuul.DependentPipelineManager: Canceling jobs for change <Change 0x7fda28fc5410 77719,1>, behind change <Change 0x7fda29c17750 75588,6> | 19:45 |
| fungi | d'oh! | 19:46 |
| fungi | 75588,6 got sniped by a new patchset as it was in the process of getting merged | 19:46 |
| clarkb | fungi: nice! | 19:47 |
| jeblair | ah, it's a requirements update patch | 19:47 |
| fungi | there's an interesting dilemma | 19:47 |
| fungi | if we have those in a long gate and then a requirements change merges, they will all get sniped | 19:48 |
| *** jcooley_ has quit IRC | 19:48 | |
| jeblair | so maybe the proposal job should not propose if the existing change is approved, and instead it should base a new change on that patch and propose it. | 19:48 |
| fungi | that's precisely what i was about to suggest | 19:48 |
| clarkb | ya | 19:48 |
| fungi | fairly lhf bug on that script | 19:48 |
| *** jcooley_ has joined #openstack-infra | 19:48 | |
| fungi | but also, things working as designed, if not optimally designed ;) | 19:49 |
| jeblair | gate-python-openstackclient-python33: NOT_REGISTERED (non-voting) | 19:49 |
| jeblair | it looks like there are no current 33 or pypy jobs registered | 19:50 |
| jeblair | possibly just because nodepool hasn't gotten around to creating those nodes | 19:50 |
| *** kashyap has quit IRC | 19:51 | |
| *** hashar has joined #openstack-infra | 19:51 | |
| StevenK | The check-tripleo and experimental-tripleo queues are empty since the zuul restart. | 19:51 |
| jeblair | StevenK: yes | 19:51 |
| clarkb | jeblair: that sounds correct. Maybe we should have gearman plugin register all jobs it knows about regardless of slaves that are connected | 19:51 |
| StevenK | jeblair: Intended behaviour? | 19:51 |
| jeblair | StevenK: yes; you can recheck if it's important | 19:52 |
| ttx | jeblair: checking in, did you manage to restart zuul ? Looks like you did, given the enqueue timers in the gate | 19:52 |
| *** ociuhandu has joined #openstack-infra | 19:52 | |
| StevenK | jeblair: Fair enough. | 19:52 |
| jeblair | ttx: just now | 19:53 |
| ttx | jeblair: cool | 19:53 |
| jeblair | clarkb: then it could end up with a job it can't run | 19:53 |
| jeblair | clarkb: i think that would be worse | 19:53 |
| *** vkozhukalov has quit IRC | 19:53 | |
| clarkb | jeblair: hrm good point | 19:53 |
| *** Sukhdev has joined #openstack-infra | 19:57 | |
| fungi | 62626,3 (in the gate) managed to get a py3k-precise node for a pypy job | 19:59 |
| *** rossella-s has quit IRC | 19:59 | |
| jeblair | pvo: how's that rate limit increase on /servers coming? we get to run 1853 more jobs and then we're done for the day. :( | 19:59 |
| *** rlandy has quit IRC | 19:59 | |
| fungi | Took 2 min 2 sec on py3k-precise-rax-iad-2171581 | 19:59 |
| fungi | jeblair: when we hit the limit, it's time to go out for beer | 20:00 |
| jeblair | pvo: could we perhaps switch back to unlimited until the new class is set up? | 20:00 |
| jeblair | fungi: we've hit dfw, iad has 33 left, ord has 1820 | 20:00 |
| fungi | beer's close then! | 20:00 |
| jeblair | i'm going to manually set ord to 0 in the nodepool config so it stops spinning its wheels | 20:01 |
| fungi | jeblair: what do you think about considering NOT_REGISTERED status as fodder for a job restart? | 20:02 |
| fungi | or are there other possible ways for that to spin madly out of control? | 20:02 |
| *** ociuhandu has quit IRC | 20:02 | |
| jeblair | fungi: this is very rare; mostly it hits when there really isn't a job for something. | 20:02 |
| fungi | good point | 20:02 |
| jeblair | fungi: (which is hard in our environment, but easily possible elsewhere) | 20:02 |
| clarkb | we have node minimums to at least 1 right? this is just a race around actually getting that node up | 20:04 |
| fungi | perhaps the jenkins-gearman plugin could remember what it had registered prior to reconnect and re-register those? but it's probably not worth the effort to implement at this stage | 20:04 |
| jeblair | fungi: that's what clarkb was suggesting, but registering a job means you can run it. if you register a job you might get it assigned. | 20:05 |
| *** rossella-s has joined #openstack-infra | 20:05 | |
| jeblair | clarkb: yeah, not being able to build nodes in rackspace hurts our ability to bring up such a node. | 20:05 |
| jeblair | clarkb: since they are only spec'd for rox. | 20:05 |
| jeblair | rax | 20:05 |
| fungi | clarkb: right, i suspect that if nodepool prioritized the first node of each type even more so than just meeting the minimum in general, we could mitigate that somewhat | 20:06 |
| fungi | clarkb: since right now py3k-precise nodes are much lower minimums than other node types | 20:06 |
| *** skraynev is now known as skraynev_afk | 20:06 | |
| pleia2 | fungi: selinux fun! https://etherpad.openstack.org/p/cgit-about-selinux | 20:07 |
| pleia2 | fungi: so now I actually think it's an issue with /tmp | 20:07 |
| fungi | so they probably get shorted a little if nodepool has to turn over a ton of new nodes all at once | 20:07 |
| fungi | pleia2: selinux forces you to learn all sorts of behaviors you never really wanted to know about random programs | 20:08 |
| pleia2 | fungi: yeah :\ | 20:08 |
| pleia2 | like "all the things that write to /tmp by default" | 20:08 |
| *** amcrn_ has quit IRC | 20:09 | |
| mriedem | anyone here ever seen this kind of failure in devstack-gate-cleanup-host? looks like a problem reporting test results with nose? | 20:11 |
| mriedem | http://logs.openstack.org/96/77396/8/check/check-sqlalchemy-migrate-devstack-dsvm/0928185/logs/devstack-gate-cleanup-host.txt | 20:11 |
| mriedem | sqlalchemy-migrate non-voting tempest job has been failing with that ever sense we got it up and running this week | 20:11 |
| *** zns has joined #openstack-infra | 20:11 | |
| fungi | pleia2: yep, so it does look like it's actually trying to use normal tempfile/tempdir patterns to formulate its output | 20:11 |
| *** vogxn has quit IRC | 20:11 | |
| mriedem | *since | 20:11 |
| *** rlandy has joined #openstack-infra | 20:11 | |
| jeblair | there goes iad | 20:12 |
| clarkb | fungi: should we be adding those images to the hpcloud as well? | 20:13 |
| fungi | lights winking out one by one | 20:15 |
| fungi | clarkb: seems fine--we've just not tested them with hopcloud's base images | 20:15 |
| jeblair | wow, so dfw just jumped up to 5000 remaining | 20:15 |
| *** Ryan_Lane has quit IRC | 20:15 | |
| jeblair | so i put it back in | 20:15 |
| fungi | clarkb: also, i guess that would just be az1 and az3 (any news on being able to build images in az2?) | 20:15 |
| *** ArxCruz has quit IRC | 20:15 | |
| *** akerr has joined #openstack-infra | 20:15 | |
| clarkb | fungi: no news but I got a name to ping and will be checking in on it nowish | 20:15 |
| clarkb | but that means hipchat | 20:16 |
| fungi | jeblair: i'm guessing pvo rolled our counter over! ;) | 20:16 |
| *** ociuhandu has joined #openstack-infra | 20:16 | |
| fungi | clarkb: i would suggest a shower afterward, but really there is no water hot enough for that | 20:16 |
| jeblair | iad is still at 0 though | 20:17 |
| akerr | maybe I'm missing something, but https://review.openstack.org/76468 was approved about 7 hours ago but Jenkins has not yet started gate checks. Any ideas why? | 20:17 |
| wendar | jeblair: I've got a question about the version of javascript we're running in elastic-recheck, jogo suggested you'd be a good person to ask? | 20:18 |
| clarkb | akerr: good question, nothing is readily apparent to me. May need to dig in logs | 20:19 |
| jeblair | wendar: i'm not sure we get to control the version of javascript. i'm pretty sure that's determined by browsers. | 20:19 |
| clarkb | but first I need to hipchat | 20:19 |
| wendar | jeblair: library versions, specifically jQuery | 20:19 |
| wendar | jeblair: we're loading 1.7.1 from http://status.openstack.org/jquery.min.js | 20:20 |
| wendar | jeblair: it's old, and uses functions that are deprecated in modern browsers | 20:20 |
| fungi | akerr: i think it was approved when 78093 was still a draft? | 20:21 |
| *** rf01 has joined #openstack-infra | 20:21 | |
| clarkb | oooohhhhhh | 20:21 |
| clarkb | drafts | 20:21 |
| jeblair | wendar: it's what is in ubuntu lts; it's likely to be upgraded when the new lts comes out. | 20:21 |
| fungi | akerr: infrequently encountered zuul bug. if any changes which depend (directly or indirectly) on your change are a gerrit draft, zuul won't enqueue them | 20:21 |
| wendar | jeblair: okay, so avoid deprecated functions in any way possible until sometime after April this year | 20:22 |
| akerr | fungi: so is there a way to tickle jenkins to start the gate? | 20:22 |
| fungi | akerr: one of the reasons we wish we could just turn off gerrit drafts entirely (but in this case it's actually triggering a legitimate bug in zuul we should fix some time when we're not busy... hah, hah) | 20:22 |
| fungi | akerr: recheck no bug on your change | 20:22 |
| wendar | jeblair: I'll see if I can override the default behavior to avoid running that function. | 20:23 |
| fungi | akerr: or get a core to reapprove it | 20:23 |
| fungi | akerr: or you could reverify against the bug we have open for zuul for that corner case. let me find it | 20:23 |
| akerr | fungi: already did recheck no bug. Will that also continue on into the gate process after the check pipeline finishes? | 20:24 |
| fungi | akerr: yes | 20:24 |
| akerr | fungi: great, thanks | 20:24 |
| fungi | akerr: for reference, bug 1260836 | 20:24 |
| akerr | fungi: good to know. thanks for the help :) | 20:24 |
| pleia2 | fungi: thanks for having a look, I'll grab one of the redhat guys here at the sprint to have a browse to see how to solve this | 20:24 |
| fungi | and if uvirtbot were around, she would retort: https://launchpad.net/bugs/1260836 | 20:25 |
| *** Ryan_Lane has joined #openstack-infra | 20:25 | |
| *** arborism has joined #openstack-infra | 20:25 | |
| fungi | pleia2: you bet. there's probably a canned selinux pattern/recipe for "expect this to use tempfiles and tempdirs in /tmp" | 20:25 |
| fungi | pleia2: or there may even be one specifically for rst2html since it is what seems to be making use ot /tmp for it's scratch space | 20:26 |
| pleia2 | fungi: it calls a few scripts aside from just rst2html, md2html and a few others as well, I'll have a look around | 20:27 |
| pleia2 | finding a lot about audit2allow, which doesn't really help a whole lot in puppet land | 20:27 |
| pleia2 | (and makes me realize lots of people don't grok selinux) | 20:27 |
| fungi | pleia2: yes, people don't grok computers (okay, well, technology in general) | 20:27 |
| pleia2 | hehe | 20:28 |
| *** jcoufal has joined #openstack-infra | 20:28 | |
| fungi | pleia2: if you ever need a reminder of that, just take a quick stroll through ubuntu forums ;) | 20:28 |
| pleia2 | hah | 20:28 |
| *** markmcclain1 has quit IRC | 20:29 | |
| fungi | i have to -site:ubuntuforums.org a lot of my web searches just to filter out the awful, awful advice | 20:29 |
| *** akerr has left #openstack-infra | 20:30 | |
| clarkb | fungi: use sudo | 20:30 |
| clarkb | for everything | 20:30 |
| fungi | clarkb: sudo bash. that makes it all work | 20:30 |
| pleia2 | and chmod 777 | 20:32 |
| *** jcooley_ has quit IRC | 20:32 | |
| fungi | pleia2: definitely. how else can you anonymously ftp scripts to your webserver and then run them? | 20:32 |
| pleia2 | D: | 20:33 |
| fungi | pleia2: protip... run apache as root and serve up the contents of /etc so you can back it up more easily! | 20:34 |
| pleia2 | this is why we can't have nice things | 20:35 |
| pleia2 | our bot has disappeared | 20:35 |
| fungi | oh, right, i was going to restart it _again_ | 20:35 |
| fungi | this time i'll see what happened to its pidfile first | 20:35 |
| clarkb | fungi: hold on | 20:35 |
| fungi | clarkb: will do | 20:36 |
| clarkb | fungi: what do you think about tagging a release now? | 20:36 |
| clarkb | fungi: and using that? | 20:36 |
| clarkb | jeblair: ^ | 20:36 |
| fungi | clarkb: i s'pose we could... but we might then find ourselves rolling it back if something in those 1.5 years of commits is weird | 20:36 |
| jeblair | clarkb: ok i guess | 20:36 |
| jeblair | fungi: you've convinced me :) | 20:36 |
| clarkb | fungi: yeah I am almost positive we will but fixing problems when we need to upgrade seems silly | 20:37 |
| jeblair | i did upgrade irclib on that host, but i expecetd that to make it better, not worse. | 20:37 |
| *** lcestari_ has joined #openstack-infra | 20:37 | |
| fungi | clarkb: so, interesting to note, the pidfile is indeed missing now. it was there when i first started the bot, for at least several minutes thereafter | 20:37 |
| *** Ryan_Lane has quit IRC | 20:38 | |
| jeblair | iad has mysteriously jumped to 5000; so i'll put it back in | 20:38 |
| *** rf01 has quit IRC | 20:38 | |
| jeblair | and restarted puppet | 20:38 |
| fungi | these are the sorts of mysteries i enjoy | 20:39 |
| *** jp_at_hp1 has quit IRC | 20:39 | |
| *** lcestari has quit IRC | 20:39 | |
| fungi | mysterious extra quota | 20:39 |
| clarkb | fungi: for https://review.openstack.org/#/c/72751/ and the associated bug how did you intend on soliciting opinions? | 20:41 |
| clarkb | fwiw I agree with you, just use a .gitreview | 20:42 |
| fungi | clarkb: i was going to ask you, jeblair, mordred and SergeyLukjanov once firefighting slowed a bit | 20:42 |
| clarkb | but the code doesn't seem particularly fragile for those of us that use a .gitreview so I am not worried about merging the patch | 20:43 |
| fungi | clarkb: agreed, my concerns are more around whether this will lead to other confusion around what workflows are expected to be supported by the utility | 20:44 |
| fungi | we already have enough people opening bugs against git-review because they read bad workflow advice in a wikimedia forum about how to use it, and it didn't work as described/elicited errors/broke | 20:46 |
| *** denis_makogon_ has joined #openstack-infra | 20:46 | |
| *** yolanda_ has quit IRC | 20:46 | |
| clarkb | :( this is true | 20:46 |
| *** fbo_away is now known as fbo | 20:46 | |
| fungi | i worry that the more flexible we make it, the more it will confuse new users (particularly outside the openstack developer community) | 20:46 |
| * SergeyLukjanov returns back from reading scrollback | 20:47 | |
| SergeyLukjanov | fungi, agree with making git-review simple and hopefully well tested | 20:47 |
| clarkb | fungi: so I hadn't considered that angle and I think it is an important one | 20:48 |
| SergeyLukjanov | fungi, clarkb, are there known usages of git-review outside of OpenStack? | 20:48 |
| SergeyLukjanov | I mean not single person but projects with recommendation to use it | 20:49 |
| fungi | SergeyLukjanov: i think almost all projects using gerrit implementations have at least a few developers using git-review to interact with it | 20:49 |
| fungi | SergeyLukjanov: but wmf is the biggest use outside openstack which i'm vurrently aware of | 20:49 |
| fungi | currently | 20:50 |
| SergeyLukjanov | wow | 20:50 |
| *** krotscheck has quit IRC | 20:51 | |
| clarkb | fwiw someone is apparently looking at the ticket for me now. hopefully get an answer back about stuff | 20:51 |
| *** krotscheck has joined #openstack-infra | 20:51 | |
| jeblair | ttx: yay ff! | 20:51 |
| *** gokrokve has quit IRC | 20:51 | |
| jeblair | ttx: any chance i can convince you to link to git.o.o instead of github? :) eg https://git.openstack.org/cgit/openstack/nova/tree/?h=milestone-proposed | 20:51 |
| fungi | clarkb: sorry about the hipchat taint, but it's for a good cause | 20:53 |
| jeblair | (i'm very proud of our git server farm, and it's so much prettier and openstacky than github) | 20:53 |
| clarkb | pythoneers https://review.openstack.org/#/c/73216/2 will that work as expected? The decorator will give us a differently named closure in each case? | 20:53 |
| fungi | clarkb: i've had continuous loops on nodepool.o.o trying (and failing) to build new bare-precise and devstack-precise images in hpcloud-az2 for over 36 hours straight now | 20:53 |
| *** rpodolyaka has joined #openstack-infra | 20:54 | |
| rpodolyaka | Hey, all! Has anyone seen this issue with python33 gates? https://review.openstack.org/#/c/74314/ | 20:55 |
| fungi | rpodolyaka: yeah, that was a side effect of the zuul restart. should hopefully have only affected a handful of changes being checked | 20:56 |
| *** markmcclain has joined #openstack-infra | 20:56 | |
| fungi | rpodolyaka: you can just recheck no bug on that | 20:56 |
| *** mrda_away is now known as mrda | 20:56 | |
| fungi | rpodolyaka: zuul tried to start that job before there were any available nodes to run it, so no workers had registered its existence | 20:57 |
| *** markmcclain has quit IRC | 20:57 | |
| *** markmcclain has joined #openstack-infra | 20:57 | |
| rpodolyaka | fungi: cool, thanks! just didn't want to force recheck in case it wasn't supposed to work :) | 20:57 |
| clarkb | fungi: jeblair: in a nodepool image-list which number in the row maps to the server instance uuid? | 20:58 |
| fungi | rpodolyaka: it was supposed to work. we didn't expect it, but figured it out fairly quickly when we spotted it | 20:58 |
| clarkb | or do I need to do a nodepool list? | 20:58 |
| fungi | clarkb: nodepool list doesn't give you the template nodes, i don't think | 20:59 |
| *** vkozhukalov has joined #openstack-infra | 20:59 | |
| *** denis_makogon has quit IRC | 20:59 | |
| *** denis_makogon_ is now known as denis_makogon | 20:59 | |
| clarkb | doesn't appear to. Is it server ID and az2 doesn't do uuids? | 20:59 |
| SergeyLukjanov | clarkb, it'll exec addOnException for each attach_on_exception usage and (http://testtools.readthedocs.org/en/latest/for-test-authors.html#addonexception) it'll be triggered only in case of exception | 20:59 |
| *** dmakogon_ has joined #openstack-infra | 20:59 | |
| fungi | clarkb: i think you'll have to couple nodepool image-list column 4 with nova list and be quick about it | 21:00 |
| clarkb | SergeyLukjanov: ah addDetail is what sorts it for us, thank you for pointing that out | 21:00 |
| clarkb | fungi: :( | 21:00 |
| fungi | clarkb: i'm checking to be sure | 21:00 |
| clarkb | fungi: I think you are correct | 21:00 |
| clarkb | fungi: because the building image state doesn't come with a Server ID | 21:01 |
| clarkb | fungi: acutally I have a server id now | 21:01 |
| fungi | clarkb: oh, the server id column does get it for you (you can see in the nodepool image-list output the rax ids are uuid and hpcloud are int) | 21:01 |
| SergeyLukjanov | clarkb, hm, looks like addDetail could be used | 21:01 |
| fungi | yep | 21:01 |
| *** bhuvan_ has quit IRC | 21:02 | |
| *** bhuvan has quit IRC | 21:02 | |
| clarkb | SergeyLukjanov: yeah that might be cleaner but the code as is should work fine | 21:02 |
| clarkb | SergeyLukjanov: I think | 21:02 |
| fungi | i like having people with fancy python learnin' around. i start to get lost with decorators still | 21:02 |
| *** yassine has quit IRC | 21:02 | |
| SergeyLukjanov | clarkb, addDetail is already used inside the attach_file | 21:02 |
| clarkb | SergeyLukjanov: yup | 21:02 |
| *** yassine has joined #openstack-infra | 21:03 | |
| clarkb | SergeyLukjanov: the explicit addDetail is for catchign non existant files | 21:03 |
| *** mrodden has quit IRC | 21:03 | |
| SergeyLukjanov | clarkb, yup | 21:04 |
| rpodolyaka | hmm, it seems that openstack_citest MySQL user still has not enough permissions to actually use created databases - http://logs.openstack.org/63/74963/4/check/gate-oslo-incubator-python27/af7b948/console.html | 21:05 |
| clarkb | rpodolyaka: on some nodes that is correct | 21:05 |
| clarkb | rpodolyaka: hpcloud az2 nodes in particular | 21:05 |
| SergeyLukjanov | clarkb, oh, I've just find the content_from_file in testtools.content | 21:06 |
| clarkb | which that test ran on | 21:06 |
| clarkb | SergeyLukjanov: oh heh I approved the change already though :) | 21:06 |
| *** MarkAtwood has quit IRC | 21:06 | |
| SergeyLukjanov | clarkb, I think it's good enough | 21:06 |
| SergeyLukjanov | clarkb, oh, nevermind content_from_file is used under attach_file and looks like couldn't handle lack of file | 21:07 |
| jeblair | fungi: thanks for taking care of the static volumes; the new space is showing up in cacti now | 21:08 |
| rpodolyaka | clarkb: is there something we can do to make it work on those nodes? | 21:08 |
| fungi | jeblair: sure, my pleasure | 21:08 |
| *** mrodden has joined #openstack-infra | 21:08 | |
| *** Ryan_Lane has joined #openstack-infra | 21:08 | |
| *** aysyanne has quit IRC | 21:08 | |
| *** MarkAtwood has joined #openstack-infra | 21:09 | |
| fungi | rpodolyaka: we have a case open with hpcloud support to fix whatever has caused us to no longer be able to build new server images in that zone | 21:09 |
| *** rf0 has joined #openstack-infra | 21:09 | |
| fungi | rpodolyaka: at the moment our options are to either wait, or stop using that zone (which reduces our test capacity by about 200 servers) | 21:09 |
| *** andreaf has joined #openstack-infra | 21:09 | |
| *** bhuvan has joined #openstack-infra | 21:09 | |
| *** bhuvan_ has joined #openstack-infra | 21:09 | |
| rpodolyaka | fungi: got it. Thanks for the info! | 21:10 |
| jeblair | so on the subject of using the new hpcloud regions... do you think we can specify a kernel command line parameter? if so, we could reduce the available ram on them | 21:10 |
| *** thomasem_ has joined #openstack-infra | 21:11 | |
| jeblair | (so we can use 30g nodes or whatever but only have linux allocate 8g) | 21:11 |
| *** jeremyb has quit IRC | 21:11 | |
| *** jeremyb has joined #openstack-infra | 21:11 | |
| mriedem | sdague: did you ever find anything in the devstack logs on the sqla-migrate tempest fails? | 21:12 |
| sdague | mriedem: did not | 21:12 |
| krotscheck | Anyone know where I can find an expert on sqllite and alembic? I'm hitting a wall and need some pointers. | 21:12 |
| sdague | the thing I thought would fix it, didn't | 21:12 |
| *** SumitNaiksatam_ has joined #openstack-infra | 21:12 | |
| *** thomasem has quit IRC | 21:12 | |
| *** Ryan_Lane has quit IRC | 21:12 | |
| mriedem | sdague: the only thing i can think is there is something missing in the job config for how it's supposed to run tests and gather results? | 21:13 |
| mriedem | but that's just based on the nosetests*.xml errors | 21:13 |
| rpodolyaka | krotscheck: what's the problem you are hitting? | 21:13 |
| sdague | mriedem: devstack is dying in a weird point, and I don't know why | 21:13 |
| mriedem | sdague: oh on setup? | 21:14 |
| mriedem | the ceilometer thing? | 21:14 |
| *** _david_ has joined #openstack-infra | 21:14 | |
| clarkb | jeblair: that doesn't solve the problem of them not being useful for tempest but could make them usable for unittesting | 21:14 |
| clarkb | jeblair: however for unittests we can probably just get away with normal 8GB nodes | 21:14 |
| *** cody-somerville has joined #openstack-infra | 21:14 | |
| jeblair | clarkb: why doesn't that make them useful for tempest? | 21:14 |
| *** bhuvan_ has quit IRC | 21:14 | |
| *** bhuvan has quit IRC | 21:14 | |
| *** jcoufal has quit IRC | 21:14 | |
| _david_ | zaro: Forking Gerrit? Havn't we said we don't want to do it? | 21:15 |
| krotscheck | rpodolyaka: Some of my tests are hitting a UserWarning during a particular migration step, which seems to make the given migration repeat itself. I've got a paste of two relevant log statements here: http://paste.openstack.org/show/72706/ | 21:15 |
| *** jcoufal has joined #openstack-infra | 21:15 | |
| clarkb | jeblair: they are too slow | 21:15 |
| *** jhesketh has joined #openstack-infra | 21:15 | |
| jeblair | clarkb: i thought the 30g nodes were fast enough | 21:15 |
| clarkb | jeblair: when sdague last pulled numbers they weren't | 21:15 |
| krotscheck | rpodolyaka: Essentially, the warning caused by migration 21645ef1040f is making alembic try to repeat the migration in the subsequent test. | 21:16 |
| jeblair | clarkb: afaik when sdague last pulled numbers they were | 21:16 |
| jhesketh | Morning | 21:16 |
| *** rpodolyaka has quit IRC | 21:16 | |
| *** SumitNaiksatam has quit IRC | 21:17 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 21:17 | |
| *** lcostantino has quit IRC | 21:17 | |
| *** yassine has quit IRC | 21:17 | |
| *** yassine has joined #openstack-infra | 21:18 | |
| clarkb | woot should have more infos about this az2 thing by tomorrow morning | 21:18 |
| *** MarkAtwood has quit IRC | 21:19 | |
| sdague | jeblair: do we have some nodes in rotation? I can scrape again | 21:20 |
| jeblair | sdague: yeah, we always have a few running check | 21:20 |
| jhesketh | clarkb, jeblair: I have a few patches for zuul if you guys have time to look at them whenever's convenient please :-) (includes the requested footer-message) | 21:20 |
| clarkb | I just pulled up jobs that are running on 5 region-b nodes | 21:20 |
| *** MarkAtwood has joined #openstack-infra | 21:20 | |
| *** MarkAtwood has quit IRC | 21:20 | |
| clarkb | when they finish I should haev a small sample | 21:20 |
| *** MarkAtwood has joined #openstack-infra | 21:21 | |
| sdague | yeh, they look fine | 21:21 |
| sdague | http://dague.net/testing/check-tempest-dsvm-full.png | 21:22 |
| sdague | faster than hp azs by a little, slightly slower than rax (though those are running 2x cores) | 21:23 |
| sdague | how many cores in region-b? | 21:23 |
| jeblair | 8 | 21:23 |
| sdague | and check-rax is the only performance node region? | 21:24 |
| clarkb | sdague: yes | 21:24 |
| jeblair | sdague: all of rax are performance nodes now | 21:24 |
| clarkb | jeblair: are they? | 21:24 |
| sdague | jeblair: really | 21:24 |
| jeblair | clarkb: my goodness i hope so | 21:24 |
| *** andre__ has quit IRC | 21:24 | |
| sdague | check-rax is definitely different than the others | 21:24 |
| *** yassine has quit IRC | 21:24 | |
| jeblair | i'm not sure what check-rax is | 21:25 |
| *** yassine has joined #openstack-infra | 21:25 | |
| *** yassine has quit IRC | 21:25 | |
| clarkb | so they are | 21:25 |
| clarkb | sdague: I remember now | 21:25 |
| sdague | this is a scrape of what's in jenkins | 21:25 |
| clarkb | sdague: its the only pvhvm image | 21:25 |
| sdague | check-rax-iad | 21:25 |
| clarkb | everything is performance but only that image is pvhvm | 21:25 |
| sdague | oh | 21:25 |
| sdague | ok, that makes sense | 21:25 |
| fungi | clarkb: yeah, we wanted to see whether the pvhvm images worked and were any more performant for our jobs | 21:25 |
| jeblair | clarkb: oh, you mean devstack-precise-check in iad. yes. | 21:25 |
| clarkb | so we should switch to pvhvm for everything too, but need to sort out the oddness in images | 21:25 |
| sdague | so the fact that we're running cores / 2 workers in tempest | 21:26 |
| sdague | puts region-b into the useful camp | 21:26 |
| sdague | at static 2 way, it was slow, IIRC | 21:26 |
| clarkb | sdague: gotcha | 21:26 |
| jeblair | but definitely all rax are performance nodes | 21:26 |
| clarkb | jeblair: yup | 21:26 |
| sdague | jeblair: ok, so just need the pvhvm image | 21:27 |
| sdague | that will speed things up 15 - 20% in rax | 21:27 |
| jeblair | rax nodes are 8 cores too | 21:27 |
| sdague | jeblair: yep | 21:27 |
| clarkb | fungi: what were the problems with pvhvm again? | 21:27 |
| sdague | core for core they are slower than the old hp nodes | 21:27 |
| sdague | but they make up for it with the cores / 2 approach | 21:27 |
| fungi | clarkb: no substantial problems once we worked out things like missing some packages we took for granted on the normal images | 21:28 |
| jeblair | so, the question is, can we set "mem=8G" on the kernel command line in hpcloud region b and have that give us what we want | 21:28 |
| fungi | clarkb: the bigger challenge i think is the transition from non-pvhvm to pvhvm... i assume we'll need separate node labels and a couple changes to job configuration to swap them around | 21:29 |
| jeblair | i think we should probably focus on that before pvhvm... | 21:29 |
| clarkb | fungi: why is that? | 21:29 |
| clarkb | fungi: can't we change the base image name and have nodepool transition over time? | 21:29 |
| fungi | clarkb: oh, nevermind. we don't need a different flavor to use the pvhvm base images do we? | 21:29 |
| clarkb | jeblair: as an alternative we can have the root cgroup set a system wide memory limit | 21:30 |
| jeblair | fungi: it's a different base image but not a flavor | 21:30 |
| clarkb | fungi: I don't think so | 21:30 |
| fungi | it was just to go from non-performance to performance that we needed to yse a separate flavor | 21:30 |
| clarkb | yup | 21:30 |
| fungi | and separate image for that flavor | 21:30 |
| jeblair | clarkb: do you think that will work? that sounds much more hairy than limiting the ram | 21:30 |
| clarkb | jeblair: it probably is more hairy but gives us more flexibility too if we need it | 21:31 |
| jeblair | clarkb: how so? | 21:31 |
| clarkb | jeblair: doing something in the nodepool script to update grub seems simpler | 21:31 |
| clarkb | jeblair: we can set hard limits and soft limits and restrict based on user | 21:31 |
| *** rfolco has quit IRC | 21:31 | |
| sdague | jeblair: so the concern is we use too much memory for guests, right | 21:31 |
| clarkb | jeblair: whcih probably don't help this use case much | 21:31 |
| jeblair | clarkb: yeah, but the point is to try to make them as close to rax as possible | 21:31 |
| jeblair | sdague: yeah, i don't want us to slip in a change that makes it so we can only run tests on nodes with 30g of ram | 21:31 |
| jeblair | sdague: then we basically lose rax, and, frankly, sanity at that point. | 21:32 |
| sdague | agreed | 21:32 |
| *** bhuvan has joined #openstack-infra | 21:33 | |
| *** bhuvan_ has joined #openstack-infra | 21:33 | |
| fungi | who still has sanity here? i thought there was a sign saying to check it at the door | 21:33 |
| clarkb | jeblair: I think we should give your kernel parameter a shot | 21:33 |
| clarkb | jeblair: we should be able to do that in the nodepool scripts for all nodes | 21:33 |
| jeblair | clarkb: yeah | 21:33 |
| clarkb | and start with testing it on the region-b nodes | 21:33 |
| fungi | clarkb: jeblair: are hpcloud's new regions using an in-image bootloader configuration? | 21:34 |
| fungi | for pvgrub or something similar? | 21:34 |
| clarkb | fungi: I have no idea, I haven't looked at image internals much | 21:35 |
| sdague | jeblair: or just allocate a giant tmpfs to eat up the memory :) | 21:36 |
| fungi | clarkb: that's likely to present the only real challenge to that plan is if they use external bootloader/config | 21:36 |
| jeblair | sdague: clever :) | 21:36 |
| clarkb | fungi: jeblair: give me a couple minutes and I will have a node for us to experiment on | 21:36 |
| jeblair | clarkb: ok cool | 21:36 |
| clarkb | sdague: and put our git repos in it | 21:36 |
| fungi | sdague: a giant tmpfs with a 22gb copy of /dev/zero on it | 21:36 |
| *** e0ne has quit IRC | 21:36 | |
| *** zns has quit IRC | 21:37 | |
| *** sweston has quit IRC | 21:37 | |
| sdague | fungi: sure, I actually wonder if we could tweak nova scheduler for this in a sane way as well. | 21:37 |
| fungi | sdague: does the api present ways to control that? and does novaclient support them? | 21:41 |
| sdague | fungi: no. I'm just thinking aloud | 21:42 |
| sdague | which I should stop doing, as I have to run away shortly for our lug meeting | 21:43 |
| jeblair | my understanding is the the new hpcloud regions have cpu throttling that differs from the old one | 21:43 |
| sdague | jeblair: interesting | 21:43 |
| sdague | that sort of makes sense, that came in on flavors in grizzly | 21:44 |
| clarkb | image is booting now, I will add keys to the ubuntu account when it is up | 21:44 |
| sdague | the cgroups bits | 21:44 |
| *** jooools has quit IRC | 21:44 | |
| *** MarkAtwood has quit IRC | 21:44 | |
| clarkb | jeblair: right. which would be fine if they still allowed bursting but that doesn't seem to be a case | 21:44 |
| jeblair | it's also possible that just the vcpu/ram ratios are simply other than what we would like. or both of those things. | 21:44 |
| clarkb | you get an allocation and thats it | 21:44 |
| * fungi is about to disappear soon for dinner | 21:45 | |
| fungi | but will catch up and help experiment on returning | 21:46 |
| clarkb | fungi: but you'll love the name of this machine :) | 21:46 |
| * fungi can't wait | 21:47 | |
| clarkb | fungi: jeblair ubuntu@15.126.252.128 | 21:47 |
| clarkb | someone should start a root screen we can attach to | 21:47 |
| clarkb | but not me because I never get terminal sizes right | 21:48 |
| nibalizer | clarkb: ^A F | 21:48 |
| nibalizer | resizes screen to your current windows size | 21:49 |
| fungi | nibalizer: oh! good tip | 21:49 |
| clarkb | well I started one | 21:49 |
| nibalizer | also thats awesome that you guys use screen -x, one of my favorite things to do | 21:49 |
| jeblair | yay i joined | 21:49 |
| nibalizer | also ^A * to see who all is attached | 21:49 |
| *** rf0 has quit IRC | 21:50 | |
| *** arborism is now known as amcrn | 21:50 | |
| fungi | okay, in there | 21:51 |
| jeblair | fungi: want to look at anything before rebooting? | 21:51 |
| fungi | jeblair: nope, checked scrollback | 21:52 |
| fungi | mostly just curious to see whether the bootloader gives a crap that there's a grub config or goes all honeybadger on it | 21:52 |
| clarkb | I think the rounding happened a little funny but it booted with 8G of memory just fine | 21:52 |
| jeblair | we could go for exact # of bytes from rax | 21:53 |
| fungi | lgtm | 21:53 |
| clarkb | jeblair: I think its close enough this way and a lot more readable | 21:53 |
| fungi | lather, rinse, repeat, but yeah this looks like it's a way forward | 21:53 |
| jeblair | clarkb: ok. so next, want to take a snapshot, boot a new 30g from it and make sure the new one comes up with 8g? | 21:54 |
| clarkb | jeblair: yup I can do that now | 21:54 |
| SpamapS | hmm.. | 21:54 |
| SpamapS | why does this show "NOT_REGISTERED" for checks? https://review.openstack.org/#/c/78426/ | 21:54 |
| clarkb | I am using the horizon ui because I haven't sorted out all the neutron stuff in region b fwiw | 21:54 |
| fungi | nibalizer: i also love using screen as a serial terminal emulator. as much as i prefer tmux these days, that's something it doesn't (and won't, according to the devs) support | 21:54 |
| *** vkozhukalov has quit IRC | 21:54 | |
| fungi | SpamapS: recheck no bug | 21:54 |
| *** thomasem_ has quit IRC | 21:55 | |
| jeblair | SpamapS: fungi: 1 sec | 21:55 |
| *** esker has joined #openstack-infra | 21:55 | |
| SpamapS | doh | 21:55 |
| nibalizer | fungi: screen /dev/ttyUSB0 is basically how we got our robot working in undergrad | 21:55 |
| SpamapS | alt-tab fail | 21:55 |
| clarkb | jeblair: I will also try booting an 8GB image and maybe a 4GB image to se ehow the kernel reacts when closer to the limit | 21:55 |
| nibalizer | :P | 21:55 |
| fungi | SpamapS: a few of the jobs which wanted py3k-precise nodes got started before there were any built to run them when we restarted zuul earlier | 21:55 |
| jeblair | we restarted zuul and there are 0 tripleo nodes online | 21:55 |
| * SpamapS checks for repeate of quota fail | 21:55 | |
| jeblair | SpamapS: so zuul doesn't think the jobs exist | 21:55 |
| *** mfer has quit IRC | 21:56 | |
| fungi | jeblair: SpamapS: oh! i should read more closely. my brain through python33 in there | 21:56 |
| jeblair | SpamapS: once those nodes come online, they will be registered and zuul will resume it's regularly scheduled behavior of waiting indefinitely | 21:56 |
| fungi | threw | 21:56 |
| jeblair | fungi: i now associate SpamapS with "tripleo issue" :) | 21:56 |
| fungi | jeblair: good point | 21:56 |
| * SpamapS feels kinda funny now | 21:56 | |
| fungi | SpamapS: did you let mordred buy your drinks again? bad idea | 21:57 |
| *** jgrimm has joined #openstack-infra | 21:57 | |
| *** jgrimm has quit IRC | 21:57 | |
| *** mwagner_lap has quit IRC | 21:57 | |
| *** Sukhdev has quit IRC | 21:57 | |
| SpamapS | fungi: he _BUYS_ them ? | 21:58 |
| *** MarkAtwood has joined #openstack-infra | 21:58 | |
| SpamapS | I only see the template node ACTIVE | 21:58 |
| SpamapS | quotas are in fact inaccurate... | 21:58 |
| SpamapS | but there should be space available | 21:59 |
| jeblair | SpamapS: we're probably within nodepool's 8 hour window | 21:59 |
| fungi | SpamapS: we've got a ton of nodes stuck in a building state for various lengths of time from when your cloud was doing weird things | 21:59 |
| jeblair | SpamapS: so 8 hours after that^ nodepool should start deleting those | 22:00 |
| *** mgagne has quit IRC | 22:00 | |
| jeblair | which may be as soon as 30 mins from now | 22:01 |
| fungi | there are a few building close to the 8 hour mark, so those will likely start to get turned over in greater and greater numbers | 22:01 |
| SpamapS | fungi: many have deleted now that we kicked nova-compute on all of the nodes | 22:01 |
| *** fbo is now known as fbo_away | 22:01 | |
| SpamapS | Ok quotas are accurate now | 22:01 |
| *** mbacchi has quit IRC | 22:02 | |
| clarkb | fungi: jeblair ubuntu@15.126.224.0 booted off a snapshot of the first node | 22:02 |
| clarkb | free reports proper values. moving on to booting off the same snapshot using smaller flavors to see how that goes | 22:02 |
| ttx | jeblair: been trying but then that makes lines > 78char | 22:02 |
| *** bhuvan__ has joined #openstack-infra | 22:02 | |
| *** bhuvan___ has joined #openstack-infra | 22:02 | |
| jeblair | ttx: drat! | 22:03 |
| fungi | SpamapS: you also had about 100 stuck in delete state, so i'm manually deleting those quickly | 22:03 |
| SpamapS | fungi: I show 5 in deleting | 22:03 |
| jeblair | clarkb: awesome! | 22:03 |
| fungi | SpamapS: delete state in nodepool, rather than nova | 22:03 |
| SpamapS | ahhh | 22:03 |
| ttx | ok, time to cut ceilo | 22:03 |
| fungi | SpamapS: nodepool doesn't realize a lot of these don't exist, so it's probably being polite and trying to avoid overrunning the quota | 22:04 |
| *** thomasem_ has joined #openstack-infra | 22:04 | |
| jeblair | ttx: for some reason i thought you were saying goodbye just then. | 22:04 |
| SpamapS | fungi: makes sense. | 22:04 |
| clarkb | horizon is pretty shiny | 22:05 |
| fungi | SpamapS: it should eventually get around to re-trying to delete them, but i'm just speeding up the process | 22:05 |
| *** bhuvan_ has quit IRC | 22:06 | |
| *** bhuvan has quit IRC | 22:06 | |
| *** thomasm_ has joined #openstack-infra | 22:06 | |
| *** esker has quit IRC | 22:07 | |
| *** mfink has quit IRC | 22:08 | |
| *** thomasem_ has quit IRC | 22:09 | |
| clarkb | but now I can't find my floating ips | 22:09 |
| *** ociuhandu has quit IRC | 22:09 | |
| clarkb | oh right under access and security not manage networks | 22:10 |
| clarkb | fungi: jeblair ssh ubuntu@15.126.245.111 and ssh ubuntu@15.126.245.117 | 22:11 |
| clarkb | I think setting the limit like that is safe | 22:11 |
| clarkb | regardless of flavor size on hpcloud region b precise images | 22:11 |
| jeblair | clarkb: too bad it didn't give us more ram. ;) | 22:12 |
| *** sarob_ has joined #openstack-infra | 22:12 | |
| *** pdmars has quit IRC | 22:12 | |
| clarkb | should I go ahead and write a patch for nodepool scripts that does this and do it only for region b nodes ? | 22:12 |
| jeblair | clarkb: why don't we see if it is safe for rax as well... | 22:13 |
| clarkb | sure | 22:13 |
| jeblair | clarkb: i'll spin up a server | 22:13 |
| clarkb | ok | 22:13 |
| sdague | ttx: can you send a blanket email to the list about the criteria for FFE | 22:13 |
| sdague | we're seeing at least some people assume this just means "I didn't get my feature done yet" | 22:13 |
| clarkb | jeblair: can you test with pvhvm? | 22:14 |
| *** jnoller has quit IRC | 22:14 | |
| clarkb | jeblair: since we should move to those images anyways imo | 22:14 |
| jeblair | clarkb: i was thinking we should test w/o pvhvm so we don't have to tie these together. we can test both tho. | 22:15 |
| clarkb | jeblair: ok | 22:15 |
| *** flaper87 is now known as flaper87|afk | 22:15 | |
| *** jcoufal has quit IRC | 22:15 | |
| *** oubiwan__ has joined #openstack-infra | 22:16 | |
| ttx | sdague: hm, I must have posted that in the past | 22:16 |
| *** jeckersb is now known as jeckersb_gone | 22:16 | |
| ttx | sdague: will try to find something | 22:16 |
| sdague | ttx: yeh, but lots of new folks | 22:16 |
| sdague | see the cisco one on their scheduler | 22:16 |
| sdague | for nova | 22:16 |
| ttx | sdague: will post soemthing. Tomorrow. | 22:17 |
| sdague | ttx: sounds good | 22:17 |
| fungi | clarkb: did the floating ip quota in the new hp regions get addressed already? | 22:17 |
| clarkb | fungi: not that I am aware of. We will need to deal with quota for region b and region a separately | 22:18 |
| fungi | clarkb: vaguely remembering you said it was very, very low | 22:18 |
| clarkb | fungi: ya my personal account has like 50 | 22:18 |
| clarkb | fungi: but if we are using 30GB nodes * 50 that may be what we get who knows :) | 22:18 |
| *** JayF has joined #openstack-infra | 22:18 | |
| fungi | ha | 22:18 |
| jeblair | clarkb: 23.253.101.80 162.242.144.35 | 22:18 |
| fungi | you can always bring mordred to the plate on that one ;) | 22:18 |
| *** dkranz has quit IRC | 22:18 | |
| jeblair | i believe mordred suggested we run lots of 30g nodes | 22:19 |
| devananda | hi guys! question about an ironic "agent" -- from your perspective, what should determine whether it lives in a separate project or not? | 22:19 |
| jeblair | otherwise i wouldn't be proposing this | 22:19 |
| clarkb | jeblair: it is asking for my apssword on 23.253.101.80 | 22:19 |
| jeblair | clarkb: oh, root@ | 22:19 |
| clarkb | ah thanks | 22:19 |
| mordred | yeah | 22:19 |
| mordred | run lots of 30g nodes | 22:19 |
| fungi | devananda: is it likely to be installed and run outside of the systems which are installing and running ironic? | 22:19 |
| mestery | anteaya: Linux Foundation just got Jenkins running for the OpenDaylight Neutron plugin, just wanted to confirm the voting is being blocked on the OpenStack end since we just started. | 22:19 |
| clarkb | jeblair: those both look good to me | 22:20 |
| *** wchrisj_ has quit IRC | 22:20 | |
| fungi | devananda: will the people developing the agent differ substantially from the set of people developing ironic? | 22:20 |
| jeblair | clarkb: i haven't done anything yet | 22:20 |
| clarkb | jeblair: oh did you boot 8GB nodes then? | 22:20 |
| devananda | fungi: it is likely to be _only_ used within environemnts runnign ironic-api and ironic-conductor | 22:20 |
| jeblair | clarkb: yes, since that's what we'd be doing anyway | 22:21 |
| clarkb | jeblair: oh right duh | 22:21 |
| jeblair | clarkb: so we should see mem go from 8 something to 7something | 22:21 |
| devananda | fungi: the people developing them will be closely coupled, but possibly differnt teams | 22:21 |
| *** JoshNang has joined #openstack-infra | 22:21 | |
| *** jroll has joined #openstack-infra | 22:21 | |
| jeblair | clarkb: i have screen on the regular one | 22:21 |
| fungi | devananda: then it's probably not warranted to split it out. the other determining factor is if you want ironic and the agent to have different release cadences/models | 22:21 |
| jeblair | (i'm installing screen on pvhvm) | 22:21 |
| *** alex-out is now known as Alexandra | 22:21 | |
| devananda | fungi: at the moment, a release cadence more akin to the client cadence is more appealing | 22:22 |
| devananda | fungi: it is not a long-lived service which requires periodibig releases | 22:22 |
| fungi | devananda: yeah if you want to release ironic like nova but release the agent like python-novaclient then you want separate projects | 22:23 |
| jeblair | clarkb: screen is running on the pvhvm one | 22:23 |
| devananda | fungi: ack, thanks | 22:23 |
| ttx | yay, all done. Phew | 22:23 |
| fungi | (not saying that the ironic agent has anything to do with novaclient, just an example of the linear tagging and releasing from master model) | 22:23 |
| clarkb | jeblair: yup caught the tail end on the pvhvm one | 22:23 |
| jeblair | clarkb: screen back on the 1st one | 22:24 |
| jeblair | clarkb: looks like that doesn't work on rax | 22:24 |
| jeblair | clarkb: on the other hand, that's not terrible because at least it doesn't mess anything up | 22:24 |
| fungi | but does it break rax? | 22:24 |
| fungi | right | 22:24 |
| clarkb | jeblair: ya we can just set it and it will be ignored | 22:25 |
| *** reed has quit IRC | 22:25 | |
| *** vkozhukalov has joined #openstack-infra | 22:25 | |
| jeblair | clarkb: it _does_ work on pvhvm | 22:25 |
| jeblair | take a look | 22:25 |
| clarkb | looking | 22:25 |
| *** mrodden1 has joined #openstack-infra | 22:25 | |
| fungi | i think if it either works or is a no-op on rax then there's no reason to try to separate out the handling for it | 22:25 |
| jogo | sdague: http://logs.openstack.org/10/77710/3/gate/gate-grenade-dsvm/1b42df2/logs/new/screen-c-api.txt.gz?level=TRACE | 22:25 |
| clarkb | fungi: yeah | 22:25 |
| jogo | I thought keystone had a fix for that | 22:25 |
| jeblair | so basically, it's not going to break perf, and we should smoothly upgrade to it when we start using pvhvm | 22:25 |
| clarkb | and it looks like that is the case | 22:25 |
| jeblair | fungi, clarkb: agreed | 22:26 |
| clarkb | jeblair: did you want to snapshot those images and try booting from snapshot? | 22:26 |
| jeblair | let me image and build from images too | 22:26 |
| clarkb | jeblair: :) | 22:26 |
| *** mfink has joined #openstack-infra | 22:26 | |
| *** smarcet has quit IRC | 22:27 | |
| morganfainberg | jogo, *reading up* fix for? cert? | 22:27 |
| *** mrodden has quit IRC | 22:28 | |
| *** markmcclain has quit IRC | 22:28 | |
| *** spm has joined #openstack-infra | 22:28 | |
| *** spm has left #openstack-infra | 22:28 | |
| jogo | morganfainberg: yup | 22:28 |
| jogo | just saw it in gatge | 22:28 |
| morganfainberg | jogo, hmm. | 22:29 |
| morganfainberg | jogo, i think this is the fix https://review.openstack.org/#/c/77215/ | 22:29 |
| morganfainberg | jogo, i _think_ | 22:29 |
| morganfainberg | jogo, but just got in, so not released yet | 22:30 |
| *** sabari has quit IRC | 22:30 | |
| *** bhuvan__ has quit IRC | 22:30 | |
| jogo | morganfainberg: we don't use trunk clients | 22:30 |
| jogo | clarkb: ^ | 22:30 |
| morganfainberg | jogo, aye | 22:31 |
| jogo | I thought we used to, why not? | 22:31 |
| *** bhuvan___ has quit IRC | 22:31 | |
| clarkb | we do use trunk clients in integration testing | 22:31 |
| morganfainberg | jogo, i thought we only used release clients | 22:31 |
| morganfainberg | oh | 22:31 |
| jogo | morganfainberg: in that case that fix didn't fix it | 22:31 |
| morganfainberg | clarkb, hmm | 22:31 |
| morganfainberg | jogo, was that error ~3hrs ago? | 22:32 |
| morganfainberg | *looks at time* | 22:32 |
| *** pcm_ has quit IRC | 22:32 | |
| morganfainberg | jogo, it looks like that merged ~3h ago | 22:32 |
| *** mdenny has quit IRC | 22:32 | |
| morganfainberg | hm. yeah i would guess that didn't fix it | 22:32 |
| jogo | morganfainberg: too bad :/ | 22:33 |
| *** _david_ has quit IRC | 22:33 | |
| *** sabari has joined #openstack-infra | 22:33 | |
| *** dkliban has quit IRC | 22:34 | |
| *** ociuhandu has joined #openstack-infra | 22:34 | |
| *** rpodolyaka has joined #openstack-infra | 22:35 | |
| *** hashar has quit IRC | 22:35 | |
| clarkb | fungi: jeblair https://review.openstack.org/78440 | 22:35 |
| fungi | i guess openstackgerrit is gone again | 22:35 |
| clarkb | oui | 22:35 |
| jeblair | fungi: do you know what the issue is? | 22:35 |
| jogo | morganfainberg: I take it you can keep digging on this one | 22:36 |
| fungi | jeblair: which issue? | 22:36 |
| morganfainberg | jogo, this might be a case where shutil.move would have been a better use. | 22:36 |
| StevenK | fungi: We really really do miss it when it just ignores the world. | 22:36 |
| jeblair | fungi: openstackgerrit | 22:36 |
| morganfainberg | jogo, vs. os.rename | 22:36 |
| morganfainberg | jogo, the rest of the logic seems sound | 22:36 |
| morganfainberg | jogo, but i am looking at what might be going on | 22:36 |
| fungi | jeblair: oh, yes it's apparently losing the irc server connection and throwing a traceback into its log, then sittinf there | 22:36 |
| jeblair | fungi: we could downgrade irclib | 22:37 |
| fungi | jeblair: clarkb pointed out that we're running gerritbot release there, and we haven't released in 1.5 years, but there's a reconnect on disconnect patch rotting on master post the last tag | 22:37 |
| clarkb | that will need testing on centos6 too | 22:37 |
| mordred | morganfainberg: we'd LIKE to run a copy of the tests that also run with the released clients ... but there are some issues we haven't figured out yet | 22:37 |
| * clarkb fires up a centos6 image in hpcloud region b | 22:38 | |
| fungi | jeblair: so clarkb suggested releasing gerritbot and seeing if master works | 22:38 |
| jogo | morganfainberg: thanks! | 22:38 |
| morganfainberg | mordred, ++ | 22:38 |
| jeblair | fungi: okay. | 22:38 |
| morganfainberg | mordred, I rememebr this conversation actually | 22:38 |
| morganfainberg | mordred, :) i look forward to the day we get both | 22:38 |
| jeblair | fungi, clarkb: rax images work as expected | 22:38 |
| fungi | jeblair: i don't see any evidence to suggest for certain that this issue is new and stems from the irclib upgrade | 22:38 |
| clarkb | jeblair: we should spot check centos6 as well | 22:39 |
| clarkb | jeblair: I am doing that in hpcloud land | 22:39 |
| jeblair | clarkb: ack | 22:39 |
| fungi | could just be another bad day on freenode, or in rax-dfw or something i suppose | 22:39 |
| *** rossella-s has quit IRC | 22:39 | |
| *** mdenny has joined #openstack-infra | 22:40 | |
| *** mriedem has quit IRC | 22:40 | |
| clarkb | 2014-03-05 22:42:12,550 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [82/120s]: url error [[Errno 113] No route to host] apparently this isn't an az2 only problem | 22:42 |
| * clarkb deletes and tries again | 22:43 | |
| clarkb | oh it might have recovered | 22:43 |
| jeblair | clarkb: was that region-b? | 22:43 |
| clarkb | jeblair: ya | 22:43 |
| *** bhuvan has joined #openstack-infra | 22:43 | |
| mordred | morganfainberg: hrm. I think I just figured out how to do it easily... | 22:43 |
| *** bhuvan_ has joined #openstack-infra | 22:44 | |
| morganfainberg | mordred, cool! | 22:44 |
| clarkb | clarkb: the node brought up sshd but it isn't letting me in | 22:44 |
| clarkb | jeblair: ^ whoops. it wants a password | 22:44 |
| clarkb | so deleting it and trying again | 22:44 |
| jeblair | clarkb: yeah, that probably means it didn't recover from the cloudinit thing | 22:45 |
| *** david-lyle has quit IRC | 22:46 | |
| *** SumitNaiksatam has quit IRC | 22:46 | |
| *** dangers is now known as dangers_away | 22:46 | |
| *** thomasem has joined #openstack-infra | 22:46 | |
| *** mdenny has quit IRC | 22:47 | |
| clarkb | woo happened again | 22:48 |
| morganfainberg | jogo, i think we need some extra logging here. | 22:48 |
| *** SumitNaiksatam has joined #openstack-infra | 22:49 | |
| * clarkb tries a different flavor to get scheduled on a different hypervisor hopefully | 22:49 | |
| morganfainberg | jogo, since i can't tell what may or may not have failed. | 22:49 |
| *** jcooley_ has joined #openstack-infra | 22:49 | |
| *** rossella_s has joined #openstack-infra | 22:49 | |
| *** thomasem has quit IRC | 22:49 | |
| *** rossella_s has quit IRC | 22:49 | |
| *** thomasm_ has quit IRC | 22:50 | |
| morganfainberg | jogo, i'm thinking we should log a specific error if the cert is "invalid" and include what data is in the cert. | 22:50 |
| morganfainberg | jogo, it'll at least make it a bit more clear. i'll work on brewing up something to help with chasing this down. | 22:51 |
| *** sabari has quit IRC | 22:51 | |
| jogo | morganfainberg: thanks! | 22:51 |
| *** sabari has joined #openstack-infra | 22:51 | |
| morganfainberg | jogo, but afaict this _should_ work as is | 22:51 |
| *** rpodolyaka has quit IRC | 22:51 | |
| *** bada_ has joined #openstack-infra | 22:51 | |
| *** Ryan_Lane has joined #openstack-infra | 22:52 | |
| morganfainberg | so it is something else going on. | 22:52 |
| jogo | morganfainberg: so if you think you have a fix that won't break things, you can land it and wait 24 hours and see if bug went away | 22:52 |
| fungi | okay, i'm disappearing for a while to go grab dinner... back later | 22:52 |
| morganfainberg | jogo, wont fix it, but will log more data | 22:52 |
| *** sweston has joined #openstack-infra | 22:52 | |
| *** oubiwan__ is now known as oubiwann-ef | 22:53 | |
| morganfainberg | jogo, so we can see if there is something else going on that isn't aparant. | 22:53 |
| clarkb | lol amazon. They just refunded 3 cents to me card | 22:53 |
| morganfainberg | jogo, all i can tell you is that the cert file and ca files exist and we had a cert load error | 22:53 |
| *** bada has quit IRC | 22:53 | |
| clarkb | probably cost more to process the refund | 22:53 |
| jogo | in that cose more debug logs FTW | 22:53 |
| morganfainberg | jogo, yep! | 22:53 |
| clarkb | jeblair: booting centos nodes is hard. | 22:54 |
| clarkb | I have had 3 in a row fail. I passed UUIDS back to hpcloud hoping this helps them debug | 22:54 |
| jeblair | wow | 22:55 |
| jeblair | region-x may not be the fix to the az2 problem we had hoped | 22:55 |
| *** vkozhukalov has quit IRC | 22:55 | |
| jeblair | (though it's still good future proofing since mordred keeps threatening that azx will go away) | 22:56 |
| sweston | chmouel: do I have to activate the account? I am getting authentication errors :-(. Or should I give it some time? | 22:56 |
| clarkb | I am trying to boot an 8GB image now to see if things get placed differently | 22:56 |
| clarkb | jeblair: yup | 22:56 |
| *** vkozhukalov has joined #openstack-infra | 22:56 | |
| *** morgabra has joined #openstack-infra | 22:56 | |
| clarkb | no luck I am going to give up for now | 22:57 |
| clarkb | we don't actually boot centos nodes in hpcloud yet | 22:57 |
| clarkb | and judging by this experience won't be able to | 22:58 |
| jogo | clarkb: I am still getting 502s from logstash | 22:59 |
| *** markmcclain has joined #openstack-infra | 22:59 | |
| jeblair | clarkb: ack | 22:59 |
| clarkb | jogo: still as compared to what? I haven't had any trouble since the move | 23:00 |
| *** dcramer__ has quit IRC | 23:01 | |
| *** eharney has quit IRC | 23:01 | |
| clarkb | jogo: is the query malformed? | 23:02 |
| jogo | clarkb: nope | 23:02 |
| jogo | a refresh fixed it | 23:02 |
| clarkb | jogo: what query are you running? | 23:02 |
| *** reed has joined #openstack-infra | 23:02 | |
| jogo | ((message:"+ echo \'The following services are still running:") OR (message:"Error: Service" AND message:"is not running")) AND filename:"console.html" AND NOT build_name:"check-grenade-dsvm-partial-ncpu"AND build_queue:"gate" | 23:02 |
| clarkb | you might be hitting timeouts if you are doing something massive | 23:02 |
| *** reed has quit IRC | 23:02 | |
| *** reed has joined #openstack-infra | 23:02 | |
| *** openstack has joined #openstack-infra | 23:03 | |
| jogo | no changes needed | 23:04 |
| clarkb | jogo: what timeperiod areyou searching over? | 23:04 |
| reed | I know it's bad but I need to remove spam from planet; I'll auto-approve this if nobody does it in the next few seconds https://review.openstack.org/#/c/78443/1 | 23:04 |
| jogo | 24 hours | 23:04 |
| *** UtahDave has quit IRC | 23:05 | |
| clarkb | jogo: hrm not sure then | 23:05 |
| clarkb | I can dig through kibana/apache logs but those are typically not very helpful let me see what es logs say | 23:06 |
| *** bradm1 is now known as bradm | 23:07 | |
| *** rpodolyaka has joined #openstack-infra | 23:07 | |
| jogo | thanks | 23:07 |
| *** SumitNaiksatam has quit IRC | 23:09 | |
| clarkb | jogo: there are no tracebacks in ES log for that queriy | 23:09 |
| clarkb | jogo: I think it was something in kibana/apache that derped | 23:09 |
| *** rpodolyaka has quit IRC | 23:10 | |
| jogo | clarkb: ahh no worries | 23:10 |
| *** SumitNaiksatam has joined #openstack-infra | 23:11 | |
| *** ryanpetrello has quit IRC | 23:11 | |
| *** dkliban has joined #openstack-infra | 23:13 | |
| *** julim has quit IRC | 23:13 | |
| *** adrian_otto has joined #openstack-infra | 23:13 | |
| *** sabari has quit IRC | 23:14 | |
| adrian_otto | Sorry guys, I am seeing something perplexing. On https://review.openstack.org/77304 Jenkins voted −1, but the only tests that are not green are showing as NOT_REGISTERED. I don't understand what that means. Any clues? | 23:14 |
| adrian_otto | I'm expecting to find a link to a console log that indicates what's wrong. | 23:15 |
| *** jcoufal has joined #openstack-infra | 23:15 | |
| *** ryanpetrello has joined #openstack-infra | 23:15 | |
| *** ryanpetrello has quit IRC | 23:15 | |
| *** sabari has joined #openstack-infra | 23:15 | |
| clarkb | adrian_otto: the job was/is not registered with Gearman. It happened because we restarted zuul when jenkins did not have slaves to run some jobs | 23:16 |
| adrian_otto | so I should do a "recheck no bug"? | 23:16 |
| clarkb | the jenkins plugin will only register jobs if it has slaves to run those jobs | 23:16 |
| clarkb | adrian_otto: you can | 23:16 |
| clarkb | let me look in scroll back to see if there is abug | 23:16 |
| adrian_otto | should I expect that to work now? | 23:16 |
| *** jamielennox|away is now known as jamielennox | 23:17 | |
| clarkb | adrian_otto: yes | 23:17 |
| clarkb | recheck no bug looks appropriate | 23:17 |
| *** whoops has quit IRC | 23:19 | |
| adrian_otto | thanks clarkb | 23:19 |
| *** Ryan_Lane has quit IRC | 23:22 | |
| *** rpodolyaka has joined #openstack-infra | 23:22 | |
| *** jergerber has quit IRC | 23:24 | |
| *** bada_ has quit IRC | 23:25 | |
| *** bada has joined #openstack-infra | 23:25 | |
| *** rpodolyaka has quit IRC | 23:25 | |
| *** rcleere has quit IRC | 23:27 | |
| *** sabari has quit IRC | 23:27 | |
| *** bhuvan has quit IRC | 23:27 | |
| *** sabari has joined #openstack-infra | 23:28 | |
| *** CaptTofu has quit IRC | 23:28 | |
| *** bhuvan_ has quit IRC | 23:28 | |
| *** CaptTofu has joined #openstack-infra | 23:28 | |
| *** jhesketh has quit IRC | 23:28 | |
| *** vkozhukalov has quit IRC | 23:29 | |
| *** yamahata has quit IRC | 23:29 | |
| *** Ryan_Lane has joined #openstack-infra | 23:30 | |
| *** Ng has quit IRC | 23:31 | |
| *** Ng has joined #openstack-infra | 23:32 | |
| *** rlandy has quit IRC | 23:32 | |
| *** bhuvan has joined #openstack-infra | 23:32 | |
| *** bhuvan has quit IRC | 23:32 | |
| *** bhuvan_ has joined #openstack-infra | 23:32 | |
| *** bhuvan has joined #openstack-infra | 23:33 | |
| *** CaptTofu has quit IRC | 23:33 | |
| morganfainberg | jogo, https://review.openstack.org/#/c/78448/ | 23:33 |
| morganfainberg | jogo, at least this will give us extra logging in failure cases. this failure should be rare, but when it happens it'll give us something | 23:34 |
| *** sabari3 has joined #openstack-infra | 23:35 | |
| *** mrodden1 has quit IRC | 23:36 | |
| jogo | morganfainberg: woot | 23:36 |
| jogo | thanks! | 23:36 |
| *** bhuvan__ has joined #openstack-infra | 23:37 | |
| *** bhuvan___ has joined #openstack-infra | 23:38 | |
| *** sabari has quit IRC | 23:38 | |
| *** bhuvan_ has quit IRC | 23:39 | |
| *** bhuvan has quit IRC | 23:40 | |
| *** rpodolyaka has joined #openstack-infra | 23:40 | |
| *** rpodolyaka has quit IRC | 23:40 | |
| *** weshay has quit IRC | 23:42 | |
| *** bhuvan has joined #openstack-infra | 23:42 | |
| *** bhuvan_ has joined #openstack-infra | 23:43 | |
| SpamapS | fungi: could you check nodepoold <-> tripleo-cloud is talking properly? I misconfigured networking last night which would have led to "many problems" :-P | 23:43 |
| *** bhuvan___ has quit IRC | 23:43 | |
| *** bhuvan__ has quit IRC | 23:43 | |
| dhellmann | mordred: does anything actually use oslo.version? | 23:44 |
| *** lcostantino has joined #openstack-infra | 23:44 | |
| *** thedodd has quit IRC | 23:44 | |
| *** CaptTofu has joined #openstack-infra | 23:45 | |
| *** jhesketh__ has joined #openstack-infra | 23:45 | |
| *** jhesketh__ has quit IRC | 23:46 | |
| *** jhesketh has joined #openstack-infra | 23:46 | |
| *** stevebaker has quit IRC | 23:46 | |
| clarkb | dhellmann: mordred doesn't think so | 23:46 |
| *** stevebaker has joined #openstack-infra | 23:46 | |
| *** jhesketh_ has joined #openstack-infra | 23:46 | |
| dhellmann | clarkb: you're psychic now? ;-) -- thanks | 23:47 |
| clarkb | dhellmann: no mordred is sitting behind me | 23:47 |
| dhellmann | clarkb: I'm thinking of moving versiontuils.py there instead of a lib on its own | 23:47 |
| dhellmann | (I figured) | 23:47 |
| * dhellmann tries to keep mordred in sight at all times | 23:47 | |
| *** lcostantino has quit IRC | 23:48 | |
| *** bhuvan__ has joined #openstack-infra | 23:48 | |
| *** bhuvan___ has joined #openstack-infra | 23:48 | |
| *** krotscheck has quit IRC | 23:48 | |
| *** bhuvan has quit IRC | 23:50 | |
| *** bhuvan_ has quit IRC | 23:51 | |
| *** bhuvan has joined #openstack-infra | 23:53 | |
| *** bhuvan_ has joined #openstack-infra | 23:53 | |
| *** bhuvan___ has quit IRC | 23:54 | |
| *** bhuvan__ has quit IRC | 23:54 | |
| *** jcoufal has quit IRC | 23:55 | |
| *** dims has quit IRC | 23:56 | |
| SpamapS | clarkb: hey, since fungi is not responding, can you check if nodepoold can reach the tripleo check cloud? | 23:56 |
| * anteaya is finally caught up with scrollback *whew* | 23:58 | |
| clarkb | SpamapS: I see a bunch of nodes in building state and delete state | 23:58 |
| anteaya | mestery: confirmed: https://review.openstack.org/#/admin/groups/270,members | 23:58 |
| *** krotscheck has joined #openstack-infra | 23:58 | |
| *** bhuvan__ has joined #openstack-infra | 23:58 | |
| *** bhuvan___ has joined #openstack-infra | 23:58 | |
| *** bhuvan has quit IRC | 23:58 | |
| anteaya | mestery: OpenDaylight Jenkins is currently in the nov-voting group for 3rd party ci | 23:58 |
| *** jcoufal has joined #openstack-infra | 23:59 | |
| anteaya | mestery: new ci accounts now start in this group by default | 23:59 |
| *** bhuvan_ has quit IRC | 23:59 | |
| anteaya | mestery: you can comment on changes (one comment per patchset please) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
