| *** otter768 has quit IRC | 00:00 | |
| *** jpeeler has quit IRC | 00:00 | |
| *** annegentle has joined #openstack-infra | 00:01 | |
| *** agentle_ has joined #openstack-infra | 00:01 | |
| *** achanda has quit IRC | 00:01 | |
| *** markvoelker has joined #openstack-infra | 00:02 | |
| jogo | ianw: hopefully that will do the trick | 00:02 |
|---|---|---|
| *** agentle_ has quit IRC | 00:05 | |
| clarkb | my naive attempt at that fix fails elsewhere | 00:05 |
| clarkb | but I am not really debugging it | 00:05 |
| greghaynes | looks like theres already a pycparser fix up | 00:05 |
| ianw | that's a fair trap that using __slots__ without __weakref__ breaks weak references ... | 00:05 |
| greghaynes | not sure how long they will take to release | 00:06 |
| lifeless | fungi: ninjafixed? | 00:06 |
| *** annegentle has quit IRC | 00:06 | |
| * dhellmann joins the fun late | 00:06 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a centos-6 node label https://review.openstack.org/171286 | 00:06 |
| clarkb | oh my fix was incomplete | 00:06 |
| fungi | hopefully ^ that will get lucky and test with fixed pycparser | 00:07 |
| fungi | lifeless: the new pycparser release | 00:07 |
| fungi | oh, not tagged yet | 00:07 |
| fungi | poop | 00:07 |
| *** markvoelker has quit IRC | 00:08 | |
| fungi | nice. eliben says "I would advise cffi and other projects to lock their dependency on a "known-good" pycparser release and bump it occasionally after testing explicitly." | 00:08 |
| clarkb | https://github.com/eliben/pycparser/commit/978ad7a61d7e1cfe77dbacb81f1690e6bb89a11b#diff-82e74302e076fb19feab7ff44ffe1133L23 it may not be complete | 00:08 |
| clarkb | but I don't know anything about this project | 00:09 |
| *** swat30 has joined #openstack-infra | 00:09 | |
| *** luqas has joined #openstack-infra | 00:09 | |
| greghaynes | well, eliben says likely fixed | 00:09 |
| greghaynes | so *shrug* | 00:09 |
| greghaynes | but yea, *everything* is broken | 00:09 |
| greghaynes | its kind of amazing | 00:09 |
| lifeless | at 2.x I'd have expected some feeling for stability | 00:09 |
| lifeless | OTOH we are using a 0.x library for crypto | 00:09 |
| lifeless | so foot, gun, trigger | 00:10 |
| dhellmann | fungi, lifeless : did we come up with a solution for our own issue, or do we have to have this pycparsing thing fixed to even tell? because if that's the case, I think clarkb's idea of turning off bandersnatch might be something to explore | 00:10 |
| fungi | especially when going from 2.10 to 2.11 but whatever | 00:10 |
| clarkb | I guess if its an ABC then nothing else will have refs to instances of that class? | 00:10 |
| lifeless | dhellmann: I don't know if sdague's patch, which was the one we had group come up with, worked or not or just hit this fail | 00:10 |
| clarkb | though if you just set __weakref__ in slots on the ABC would that not be sufficient? | 00:10 |
| clarkb | oh no because slots is ioverridden in children | 00:10 |
| *** isviridov_away is now known as isviridov | 00:11 | |
| Kennan | I also tried latest devstack, it also broke on tempest install with ' TypeError: cannot create weak reference to 'Struct' object' | 00:11 |
| lifeless | fungi: where does eliben say that? | 00:11 |
| clarkb | Kennan: yes see scrollback | 00:11 |
| fungi | Kennan: yeah, that's courtesy of latest pycparser | 00:11 |
| dhellmann | lifeless: yeah, it hit this fail, so we don't know. that's why I was saying maybe the bandersnatch thing would be a good idea, since we could explore the other fix we have to make | 00:11 |
| lifeless | clarkb: no, slots are concrete | 00:11 |
| fungi | lifeless: latest comment in https://github.com/eliben/pycparser/issues/76 | 00:11 |
| *** jpeeler has joined #openstack-infra | 00:11 | |
| fungi | actually just now no longer latest comment. he wants some testing and will then tag a new release asap | 00:12 |
| clarkb | I can test | 00:13 |
| clarkb | if I can figure out github | 00:13 |
| ianw | lifeless: yeah; playing with __slots__ is almost like fiddling with your ABI ... plenty of potential to mess up. but i also think this is probably a corner-case, as there was no unit test of weak references. who thinks of that, really | 00:13 |
| *** luqas has quit IRC | 00:13 | |
| fungi | after the last couple days of reading debian-devel i've realized that i fail at github | 00:13 |
| greghaynes | cryptography says it fixes for them | 00:13 |
| *** sarob has joined #openstack-infra | 00:14 | |
| lifeless | ianw: my reaction was actually WTF are they taking weakrefs for | 00:14 |
| clarkb | ya works here | 00:15 |
| lifeless | fungi: oh? | 00:15 |
| fungi | i'm not even close to being a github apologist, but there were more than a few detractors unwilling to even interact with its ui | 00:16 |
| fungi | just mildly amused, but it's debian-devel so why am i surprised really | 00:17 |
| *** tsg has quit IRC | 00:17 | |
| clarkb | ianw: a simple (and msotly reasonably) set of functional tests since this project exists to parse C for python is to run other projects C compiles | 00:17 |
| clarkb | ianw: while that isn't a specific weakref test it would likely catch all kinds of fun issues and travis will run it for free | 00:18 |
| clarkb | we could even run it here on a periodic basis ( we don't have github triggers so hard to trigger per commit) | 00:18 |
| *** bhuvan has quit IRC | 00:18 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/zuul: Adding doc reference of ZUUL_CHANGES https://review.openstack.org/176140 | 00:19 |
| *** mtanino has quit IRC | 00:19 | |
| *** davideagnello has quit IRC | 00:20 | |
| fungi | TheJulia: fyi hashar just today said he also has some documentation updates on the way for zuul, so those might be a good idea to keep an eye out for and review | 00:22 |
| *** yamamoto has quit IRC | 00:22 | |
| fungi | we'll probably shoot for making sure its documentation is as polished as we can reasonably get it before tagging the impending releas3e | 00:23 |
| *** achanda has joined #openstack-infra | 00:23 | |
| *** david-lyle has quit IRC | 00:24 | |
| *** tqtran has quit IRC | 00:24 | |
| lifeless | fungi: I was rather amused at being (AFAICT) lumped in with the apologists | 00:26 |
| *** davideagnello has joined #openstack-infra | 00:27 | |
| fungi | you proprietary software monger you | 00:27 |
| lifeless | indeed | 00:27 |
| *** david-lyle has joined #openstack-infra | 00:27 | |
| *** sputnik13 has quit IRC | 00:27 | |
| *** achanda has quit IRC | 00:28 | |
| jogo | and pycparser is fixed https://github.com/eliben/pycparser/issues/76 | 00:29 |
| jogo | back to one wedge | 00:29 |
| *** tsg has joined #openstack-infra | 00:29 | |
| *** ddieterly has joined #openstack-infra | 00:31 | |
| swat30 | hey all | 00:32 |
| swat30 | pushed my first review and am seeing quite a few tests fail, some seem unrelated to my changes | 00:32 |
| fungi | jogo: one wedge... for now! ;) | 00:32 |
| swat30 | can someone take a look and let me know if I'm doing anything wrong? | 00:32 |
| swat30 | https://review.openstack.org/#/c/176019/ | 00:32 |
| clarkb | swat30: its likely related to the recent pycparser explosion but I will look | 00:32 |
| swat30 | clarkb, thanks! it does seem to be something odd going on | 00:33 |
| fungi | swat30: i find it interesting that your first change is to the stable/icehouse branch of neutron | 00:33 |
| clarkb | swat30: looks like the stevedore issue hit you instead which is the other wedge fungi and jogo are talking about | 00:33 |
| *** davideagnello has quit IRC | 00:33 | |
| swat30 | fungi, yea, working on an upgrade on an old quantum install | 00:33 |
| swat30 | ran into an issue w/ DB migrations | 00:34 |
| swat30 | saw a bug already filed, but unfixed | 00:34 |
| fungi | swat30: so bug 1241577 is icehouse-only? | 00:34 |
| openstack | bug 1241577 in neutron "sqlalchemy.exc.OperationalError: (OperationalError) Cannot add a NOT NULL column with default value NULL" [Low,In progress] https://launchpad.net/bugs/1241577 - Assigned to Jacob Godin (jacobgodin) | 00:34 |
| *** gmurphy has joined #openstack-infra | 00:35 | |
| jogo | dhellmann: now that the crpyto issue is fixed, you have a fix for the other issue? | 00:35 |
| swat30 | fungi, hm, good Q. the DB migration from grizz. is gone in Juno | 00:35 |
| *** ddieterly has quit IRC | 00:35 | |
| fungi | swat30: then sounds like icehouse is probably the only supported branch where it still exists, so that sounds right | 00:35 |
| swat30 | that was my thought fungi | 00:36 |
| swat30 | I'll look into it | 00:36 |
| *** emagana has quit IRC | 00:36 | |
| dhellmann | jogo: we have a couple of things in play. sdague submitted https://review.openstack.org/#/c/176113/ and I am working it from the other side with https://review.openstack.org/#/c/176089/1 | 00:36 |
| fungi | swat30: anyway, what clarkb says is true, we're running into some (hopefully very temporary) testing issues while trying to transition to a new requirements tracking workflow for the upcoming kilo release | 00:36 |
| jogo | dhellmann: btw you may find the discussion at the end of https://github.com/pyca/cryptography/issues/1858 funny | 00:37 |
| jogo | re: python dependency hell | 00:37 |
| *** jaypipes has quit IRC | 00:37 | |
| fungi | s/funny/depressing/ | 00:37 |
| dhellmann | jogo, fungi, sdague, ttx, lifeless: I started an etherpad for notes https://etherpad.openstack.org/p/april-21-gate-wedge | 00:38 |
| jogo | fungi: yes | 00:38 |
| swat30 | fungi, appreciate it. I saw some global req updates to a bunch of projects. I assume that's related? | 00:38 |
| dhellmann | jogo, fungi, sdague, ttx, lifeless: since this is only sort of related to the requirements plan, I didn't want to add all of the details there | 00:39 |
| fungi | swat30: lots of library releases too, yes | 00:39 |
| *** zhiwei has joined #openstack-infra | 00:39 | |
| swat30 | fungi, cool. I'll let it be for a bit and give it a recheck in a day or two. thanks for your help | 00:39 |
| *** zz_dimtruck is now known as dimtruck | 00:40 | |
| fungi | heh... "maybe this is a pip problem" | 00:40 |
| jogo | dhellmann: makes sense, looks like with a bit of luck we can be out of this mess in a few hours | 00:40 |
| clarkb | dhellmann: 176113 is the simple fix right? | 00:40 |
| jogo | fungi: :) | 00:40 |
| clarkb | dhellmann: eg if that one works then we can just bump that req? | 00:41 |
| * dhellmann goes to look up the change clarkb is referring to | 00:41 | |
| *** armax has quit IRC | 00:41 | |
| dhellmann | clarkb: maybe. We don't know if that will actually work, yet | 00:41 |
| fungi | dhellmann: "sean's patch" per the etherpad | 00:42 |
| clarkb | but we should know in about an hour? | 00:42 |
| dhellmann | clarkb: and it may not be sufficient, we may need to also merge that change into the other projects | 00:42 |
| dhellmann | clarkb: assuming nothing else happens, yeah | 00:42 |
| *** ddieterly has joined #openstack-infra | 00:42 | |
| fungi | clarkb: 27 minutes so sayeth the zuul | 00:42 |
| jogo | dhellmann: can we test this outside of the gate? | 00:42 |
| clarkb | jogo: if you have a tempest laying around ready to run | 00:43 |
| dhellmann | large-ops has already failed | 00:43 |
| fungi | also it's already failed gate-tempest-dsvm-large-ops though maybe that's unrelated? | 00:43 |
| jogo | dhellmann: link? | 00:43 |
| jogo | ahh found it | 00:43 |
| *** dannywilson has quit IRC | 00:43 | |
| clarkb | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062/ for those following along | 00:44 |
| dhellmann | the job we care about is https://jenkins05.openstack.org/job/check-grenade-dsvm/26295/console because that's the grenade test for the job that depends on the requirements change | 00:44 |
| *** dannywilson has joined #openstack-infra | 00:44 | |
| clarkb | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062/logs/devstacklog.txt.gz#_2015-04-22_00_42_31_668 | 00:44 |
| clarkb | I think this is just the next in line issue | 00:44 |
| jogo | clarkb: sigh | 00:45 |
| fungi | it's too bad they chain the mallet to this whack-a-mole machine | 00:45 |
| jogo | fungi: invest in bolt cutters | 00:45 |
| dhellmann | I wish we could get a traceback from that command, so we could see whether the issue is in the cli program or the plugin | 00:45 |
| fungi | it is my honor to present to the qa team... this crate of whack-a-mole mallets. may they serve you well | 00:45 |
| * jogo humbly accepts the crate on behalf of the qa team and hands them out | 00:47 | |
| * dhellmann bangs his head on his mallet | 00:47 | |
| jogo | dhellmann: the time stamp is interesting | 00:47 |
| jogo | morganfainberg: halp | 00:47 |
| jogo | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/apache/keystone.txt.gz | 00:47 |
| jogo | and | 00:48 |
| jogo | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/apache/keystone_access.txt.gz | 00:48 |
| jogo | 127.0.0.1 - - [22/Apr/2015:00:42:36 +0000] "POST /v2.0/tokens HTTP/1.1" 401 365 "-" "python-keystoneclient" | 00:48 |
| morganfainberg | looking | 00:48 |
| jogo | is the 401 error we are seeing at 42:36 | 00:48 |
| morganfainberg | but i need to jump out for food and an appointment very soon | 00:48 |
| jogo | morganfainberg: top level bug http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/devstacklog.txt.gz | 00:48 |
| morganfainberg | Authorization failed. Could not find user: admin (Disable debug mode to suppress these details.) (Disable debug mode to suppress these details.) from 127.0.0.1 | 00:49 |
| morganfainberg | is... that possible? | 00:49 |
| morganfainberg | i mean, the admin user should exist by this point | 00:49 |
| jogo | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/apache/keystone.txt.gz#_2015-04-22_00_42_36_981561 | 00:49 |
| jogo | I don't think keystone was ready yet | 00:50 |
| jogo | at 42:36 | 00:50 |
| morganfainberg | i mean, if the user legitimately isn't there this makes sense | 00:50 |
| morganfainberg | that would be exactly what i would expect to have occured | 00:50 |
| jogo | hmm http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/apache/keystone.txt.gz#_2015-04-22_00_42_37_283425 is the error | 00:50 |
| morganfainberg | yep | 00:51 |
| morganfainberg | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/devstacklog.txt.gz#_2015-04-22_00_41_43_871 | 00:51 |
| morganfainberg | dep resolution issues | 00:51 |
| morganfainberg | again | 00:51 |
| jogo | ahh http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/devstacklog.txt.gz#_2015-04-22_00_41_42_685 | 00:51 |
| jogo | yeah | 00:51 |
| jogo | doh | 00:51 |
| jogo | ughhh dhellmann | 00:52 |
| morganfainberg | yay! i mean... opposite of "yay" | 00:52 |
| dhellmann | jogo: ? | 00:52 |
| jogo | openstackclient.shell Exception raised: (oslo.utils 1.2.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.utils>=1.4.0'), set(['python-novaclient'])) | 00:52 |
| dhellmann | yeah | 00:52 |
| *** yamamoto has joined #openstack-infra | 00:53 | |
| *** prad has quit IRC | 00:53 | |
| * morganfainberg goes to get food and head off for a couple hourse. | 00:53 | |
| jroll | is it safe to assume nova jobs are failing right now? | 00:53 |
| anteaya | jroll: lots of failure right now | 00:54 |
| jroll | thanks for confirmation anteaya :) | 00:54 |
| anteaya | pyparser and a stevedore issue if my scorecard is correct | 00:55 |
| jogo | anteaya: pycparser is fixed | 00:55 |
| Kennan | the pip | 00:55 |
| anteaya | jroll: happy to do the least I can do | 00:55 |
| Kennan | http://logs.openstack.org/13/176113/1/check/gate-tempest-dsvm-large-ops/1a5c062//logs/pip-freeze.txt.gz | 00:55 |
| anteaya | jogo: yay | 00:55 |
| jogo | the issue is now oslo.utils too | 00:55 |
| anteaya | jogo: ah | 00:55 |
| * anteaya updates her scorecard | 00:55 | |
| fungi | (plus whatever problems are hiding behind that) | 00:55 |
| morganfainberg | anteaya, i'm playing bingo | 00:55 |
| jogo | fungi: :( | 00:56 |
| fungi | morganfainberg: it's all fun and games until someone loses an i | 00:56 |
| anteaya | morganfainberg: bingo whack-a-mole everyone has a game | 00:56 |
| morganfainberg | fung: hey, i think you lost one. | 00:56 |
| * morganfainberg sees himself out for the bad joke. | 00:56 | |
| jogo | how come devstack doesn't run with set -e | 00:57 |
| *** marun has quit IRC | 00:57 | |
| anteaya | morganfainberg: you might be greeted at the door with someone with a mallet insisting you stay | 00:57 |
| morganfainberg | anteaya, hehe | 00:57 |
| fungi | jogo: i think parts of it do, yeah? | 00:57 |
| dtroyer | jogo, it is supposed to, it appears to have been broken at some point | 00:57 |
| fungi | though it was not that long ago that we got most of devstack-gate running set -e | 00:58 |
| morganfainberg | it doesn't run set -e? | 00:58 |
| * morganfainberg thought it did | 00:58 | |
| Kennan | the issue is not common hit nowdays, we found in requests package, it not install with requirements.txt, but used old ubuntu default package version | 00:59 |
| dhellmann | jogo: I think this whack-a-mole approach is wrong. I'm working on https://review.openstack.org/#/c/176113 instead | 00:59 |
| jogo | dhellmann: is that the right link? that points to sdague's patch | 01:00 |
| *** tjones1 has joined #openstack-infra | 01:00 | |
| fungi | Kennan: yes, if you're pip installing into the system environment without -U then whatever version of requirements might have been installed from a distro package will get left in place, not replaced by any newer version from pypi | 01:00 |
| dhellmann | jogo: tab overload: https://review.openstack.org/#/c/176089 | 01:00 |
| *** dannywilson has quit IRC | 01:00 | |
| fungi | Kennan: same goes for any python library | 01:01 |
| *** johnsom has left #openstack-infra | 01:01 | |
| Kennan | yes, :fungi it was for oslo-utils issue, same issue, I think | 01:02 |
| *** ayoung has joined #openstack-infra | 01:02 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Revert "Turn off HP in nodepool"" https://review.openstack.org/176102 | 01:03 |
| *** annegentle has joined #openstack-infra | 01:03 | |
| *** agentle_ has joined #openstack-infra | 01:03 | |
| dhellmann | dtroyer: if you're still around, do you want to give a quick look to https://review.openstack.org/#/c/176089 ? I'll self-approve if the tests pass, otherwise | 01:05 |
| jogo | harlowja: makes sense, better then whack-a-mole | 01:06 |
| *** tjones1 has quit IRC | 01:06 | |
| jogo | derp | 01:06 |
| jogo | dhellmann: ^ | 01:06 |
| jogo | dtroyer: test patch https://review.openstack.org/176149 | 01:06 |
| dhellmann | jogo: right, though we'll have to see if it does actually fix the issue | 01:06 |
| jogo | dtroyer fungi: testing add set -e in devstack to see what happens | 01:06 |
| dtroyer | dhellmann: I'm in&out… I think that looks fine but I'll admit that's on the edges of my setuptools understanding | 01:06 |
| *** kaisers has quit IRC | 01:07 | |
| dhellmann | dtroyer: there is similar code in stevedore, but I don't have to accommodate the openstackclient test code that creates a fake, so it's simpler | 01:07 |
| dtroyer | jogo: it's in there, w need to figure out where it is getting turned off and not back on | 01:07 |
| *** agentle_ has quit IRC | 01:08 | |
| dhellmann | I'll fix the fake separately, but won't be able to land that fix until the juno issues are resolved | 01:08 |
| jogo | dtroyer: where is it a quick git grep didn't show anything | 01:08 |
| *** prad has joined #openstack-infra | 01:08 | |
| dtroyer | line 477 in my current branch - search for errexit | 01:08 |
| *** annegentle has quit IRC | 01:08 | |
| lifeless | is pycparser all good now ? | 01:09 |
| jogo | lifeless: yes | 01:09 |
| *** patrickeast has quit IRC | 01:11 | |
| *** gyee has quit IRC | 01:12 | |
| jogo | dtroyer: you may need If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases. | 01:13 |
| jogo | -E | 01:13 |
| dtroyer | I suspect that there's a function where errexit is disabled that isn't re-enabling it | 01:15 |
| *** sarob has quit IRC | 01:15 | |
| dtroyer | was trying to cut down on useless logging once upon a time | 01:15 |
| *** shashankhegde has quit IRC | 01:16 | |
| *** sarob has joined #openstack-infra | 01:16 | |
| *** baoli has joined #openstack-infra | 01:17 | |
| jogo | dtroyer: I think this should do it https://review.openstack.org/176149 | 01:18 |
| jogo | as that fits the symptoms some things work for the trap and some don't | 01:18 |
| * jogo heads out for some fresh air | 01:19 | |
| *** sarob has quit IRC | 01:21 | |
| *** baoli_ has joined #openstack-infra | 01:21 | |
| *** kaisers has joined #openstack-infra | 01:22 | |
| dtroyer | err…I confused errexit and xtrace above, it's xtrace we toggle on and off…jeez, time for air for me too? | 01:23 |
| *** baoli has quit IRC | 01:24 | |
| *** baoli_ has quit IRC | 01:26 | |
| *** asselin_ has joined #openstack-infra | 01:30 | |
| *** baoli has joined #openstack-infra | 01:34 | |
| HenryG | Will rechecks work now, or not yet? | 01:38 |
| *** ajmiller has joined #openstack-infra | 01:41 | |
| *** julim has joined #openstack-infra | 01:46 | |
| *** changbl has joined #openstack-infra | 01:48 | |
| *** otter768 has joined #openstack-infra | 01:56 | |
| *** stevemar has joined #openstack-infra | 01:56 | |
| *** tsg has quit IRC | 01:59 | |
| *** dboik has joined #openstack-infra | 02:00 | |
| *** agentle_ has joined #openstack-infra | 02:00 | |
| *** annegentle has joined #openstack-infra | 02:00 | |
| dtroyer | jogo: here's a simple brute-force test of errexit (-e) and errtrace (-E) in functions and subshells: https://gist.github.com/dtroyer/a065b77ebe0443659898 the script exits early for every combination except when both are off and the error is in a function. that's what we're seeing in the osc errors in these runs. errtrace only affects when the err_trap is executed. | 02:00 |
| *** otter768 has quit IRC | 02:01 | |
| *** unicell has quit IRC | 02:01 | |
| *** markvoelker has joined #openstack-infra | 02:01 | |
| *** harlowja is now known as harlowja_away | 02:03 | |
| *** dboik has quit IRC | 02:05 | |
| *** agentle_ has quit IRC | 02:09 | |
| *** annegentle has quit IRC | 02:09 | |
| *** xyang1 has quit IRC | 02:09 | |
| *** luqas has joined #openstack-infra | 02:10 | |
| *** shashankhegde has joined #openstack-infra | 02:13 | |
| *** luqas has quit IRC | 02:15 | |
| *** macjack has joined #openstack-infra | 02:20 | |
| *** kaisers has quit IRC | 02:24 | |
| *** agentle_ has joined #openstack-infra | 02:27 | |
| *** annegentle has joined #openstack-infra | 02:27 | |
| *** alexpilotti has quit IRC | 02:27 | |
| *** ChuckC has joined #openstack-infra | 02:29 | |
| *** a13x2121 has joined #openstack-infra | 02:34 | |
| *** Swami_ has joined #openstack-infra | 02:36 | |
| *** Swami has quit IRC | 02:37 | |
| *** esker has joined #openstack-infra | 02:38 | |
| *** jamielennox is now known as jamielennox|away | 02:39 | |
| *** sputnik13 has joined #openstack-infra | 02:39 | |
| *** kaisers has joined #openstack-infra | 02:39 | |
| *** sputnik13 has quit IRC | 02:44 | |
| *** sputnik13 has joined #openstack-infra | 02:44 | |
| Kennan | :lifeless | 02:47 |
| Kennan | did you limit the versionedobject becase 0.2.0 not work ? | 02:47 |
| *** sputnik13 has quit IRC | 02:47 | |
| Kennan | I see it is lower than >0.1.1 <0.2.0 | 02:48 |
| lifeless | Kennan: what? | 02:48 |
| Kennan | >=0.1.1 < 0.20 | 02:48 |
| lifeless | see what is | 02:48 |
| *** ibiris is now known as ibiris_away | 02:48 | |
| Kennan | because 0.2.0 the latest version breaks on openstack projects | 02:49 |
| Kennan | right ? | 02:49 |
| lifeless | Kennan: I have *no* idea what you're talking about | 02:50 |
| *** hichihara has joined #openstack-infra | 02:50 | |
| Kennan | https://github.com/openstack/oslo.versionedobjects/commit/4c0b0cb3676d24454e3dbbad9d80548cd3150326 | 02:50 |
| *** ivar-laz_ has joined #openstack-infra | 02:50 | |
| Kennan | the 0.2.0 has such change | 02:50 |
| *** adreznec has quit IRC | 02:50 | |
| Kennan | so we not use 0.2.0, because it breaks on some code | 02:51 |
| lifeless | HenryG: we're still wedged AIUI | 02:51 |
| Kennan | right ? | 02:51 |
| *** mfink has quit IRC | 02:51 | |
| *** ivar-lazzaro has quit IRC | 02:51 | |
| jogo | dtroyer: so your saying errtrace doesn't do anything without errexit? | 02:51 |
| lifeless | Kennan: no | 02:52 |
| lifeless | Kennan: we have upper caps on many things because thats the current strategy being used to avoid situations like we're in now | 02:52 |
| lifeless | Kennan: where whats installed doesn't work. | 02:52 |
| *** ivar-laz_ has quit IRC | 02:53 | |
| Kennan | ok. I am ok with that. Just curious about that. | 02:53 |
| Kennan | did you have any comments about https://review.openstack.org/#/c/173290/ | 02:53 |
| *** davideagnello has joined #openstack-infra | 02:53 | |
| Kennan | :lifeless I am not sure if we have clear of issue we talked before, ubutun system install conflict with pip install | 02:54 |
| Kennan | ? | 02:54 |
| Kennan | is it pip someting ? or devstack issue ? | 02:55 |
| lifeless | nothing to do with system install | 02:55 |
| lifeless | its because of pip issue 988 | 02:55 |
| lifeless | combined with pip issue 2687 | 02:56 |
| *** mfink has joined #openstack-infra | 02:58 | |
| Kennan | ok :lifeless so how to deal with that pip issue ? if pip not fix that | 03:00 |
| Kennan | I think its better to raise some version | 03:00 |
| lifeless | Kennan: there is ongoing debate about that | 03:00 |
| lifeless | Kennan: please see the thread(s) on openstack-dev for context | 03:00 |
| lifeless | Kennan: requirements is part of our mechanism for dealing with the issue | 03:00 |
| Kennan | yes. :lifeless. thanks. not subscribe openstack-dev threads recently | 03:01 |
| *** adreznec has joined #openstack-infra | 03:01 | |
| ianw | lifeless: so, as the testing guru, what's wrong with https://github.com/ianw/pycparser/commit/7f4fadaf4b20598843e802e3e9e796a1e268b6f4 ? :) | 03:06 |
| *** asselin__ has joined #openstack-infra | 03:09 | |
| lifeless | ianw: commented (twice) | 03:09 |
| *** agentle_ has quit IRC | 03:10 | |
| *** annegentle has quit IRC | 03:10 | |
| *** asselin_ has quit IRC | 03:11 | |
| ianw | lifeless: ok, thanks; i couldn't find many examples of projects sort of using themselves as functional tests for other projects. i think it's a good smoke test, at least | 03:12 |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 03:12 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,In progress] https://launchpad.net/bugs/1444806 - Assigned to Bharat Kumar Kobagana (bharat-kobagana) | 03:12 |
| dtroyer | jogo: I'm saying its a no-op WRT failing, it only changes how the err trap is called | 03:13 |
| *** agentle_ has joined #openstack-infra | 03:13 | |
| *** annegentle has joined #openstack-infra | 03:13 | |
| *** sputnik13 has joined #openstack-infra | 03:14 | |
| *** sputnik13 has quit IRC | 03:14 | |
| *** asalkeld has left #openstack-infra | 03:17 | |
| *** agentle_ has quit IRC | 03:20 | |
| *** annegentle has quit IRC | 03:20 | |
| *** annegentle has joined #openstack-infra | 03:21 | |
| *** agentle_ has joined #openstack-infra | 03:21 | |
| *** davideagnello has quit IRC | 03:25 | |
| *** agentle_ has quit IRC | 03:26 | |
| *** annegentle has quit IRC | 03:26 | |
| *** mescanef has quit IRC | 03:28 | |
| *** mescanef has joined #openstack-infra | 03:28 | |
| openstackgerrit | lifeless proposed openstack/requirements: raise stevedore requirements https://review.openstack.org/176113 | 03:30 |
| *** jerryz has joined #openstack-infra | 03:32 | |
| jogo | dtroyer: ahh, my bashfoo is very weak | 03:33 |
| *** baoli has quit IRC | 03:34 | |
| *** hichihara has quit IRC | 03:35 | |
| *** dboik has joined #openstack-infra | 03:36 | |
| *** dboik_ has joined #openstack-infra | 03:37 | |
| *** hichihara has joined #openstack-infra | 03:38 | |
| *** hichihara has quit IRC | 03:38 | |
| *** dboik has quit IRC | 03:40 | |
| *** Swami_ has quit IRC | 03:48 | |
| jogo | dtroyer: so what is causing devstack to not exit on failures? | 03:49 |
| *** asselin__ has quit IRC | 03:50 | |
| dtroyer | I don't know | 03:50 |
| *** shashankhegde has quit IRC | 03:50 | |
| jogo | how do $() and pipes change your test? | 03:51 |
| dtroyer | not sure I follow on $0, in a pipeline only the last component causes an error exit with non-zero return unless PIPEFAIL is in play, which we do only in a couple of places. | 03:53 |
| *** dboik_ has quit IRC | 03:54 | |
| jogo | dtroyer: I was thinking https://github.com/openstack-dev/devstack/blob/master/functions-common#L697 | 03:54 |
| jogo | that fails | 03:54 |
| jogo | http://logs.openstack.org/49/176149/2/check/check-tempest-dsvm-full/4e03070/logs/devstacklog.txt.gz#_2015-04-22_01_30_20_117 | 03:54 |
| jogo | but doesn't cause devstack to exit | 03:55 |
| *** ddieterly has quit IRC | 03:56 | |
| *** ajmiller has quit IRC | 03:57 | |
| *** otter768 has joined #openstack-infra | 03:57 | |
| jogo | perhaps get_field is messing things up | 03:58 |
| dtroyer | when that is still used, possibly. and that is new enough we can't use it in juno | 03:59 |
| dtroyer | er, we haven't backported the elimination of the pipelines in those function | 03:59 |
| jogo | dtroyer: so this is one of the errors we see in successful runs http://logs.openstack.org/49/176149/2/check/check-tempest-dsvm-full/4e03070/logs/devstacklog.txt.gz#_2015-04-22_01_30_20_117 | 03:59 |
| jogo | ahh, so we want to eliminate pipelines in those that makes sense | 04:00 |
| jogo | anyway its getting late here o/ | 04:00 |
| *** asselin_ has joined #openstack-infra | 04:00 | |
| dtroyer | I'm actually ready to try to stop using our tools in jobs where they may also be under test | 04:00 |
| dtroyer | ie, OSC isn't appropriate for utility use in DevStack, as much as I want it to be | 04:00 |
| *** dannywilson has joined #openstack-infra | 04:01 | |
| dtroyer | and I'm not kidding anymore about rewriting it in Go…spent >20 hours on that over the weekend... | 04:01 |
| *** otter768 has quit IRC | 04:02 | |
| jerryz | fungi: ping | 04:02 |
| *** fawadkhaliq has joined #openstack-infra | 04:02 | |
| *** juggler_ has joined #openstack-infra | 04:05 | |
| *** juggler has quit IRC | 04:06 | |
| *** krtaylor has quit IRC | 04:06 | |
| *** dannywilson has quit IRC | 04:06 | |
| *** sslypushenko has quit IRC | 04:09 | |
| *** krtaylor has joined #openstack-infra | 04:10 | |
| *** luqas has joined #openstack-infra | 04:11 | |
| *** mahito has joined #openstack-infra | 04:15 | |
| *** patrickeast has joined #openstack-infra | 04:15 | |
| *** luqas has quit IRC | 04:16 | |
| *** BharatK has quit IRC | 04:20 | |
| *** enikanorov__ has quit IRC | 04:21 | |
| *** agentle_ has joined #openstack-infra | 04:22 | |
| *** annegentle has joined #openstack-infra | 04:22 | |
| *** agentle_ has quit IRC | 04:27 | |
| *** annegentle has quit IRC | 04:28 | |
| *** sdake has joined #openstack-infra | 04:29 | |
| *** sdake_ has joined #openstack-infra | 04:32 | |
| *** sdake has quit IRC | 04:33 | |
| *** unicell has joined #openstack-infra | 04:36 | |
| lifeless | dhellmann: sdague: https://review.openstack.org/#/c/176113/ | 04:37 |
| lifeless | dhellmann: that seems promising, only failure was grenade | 04:37 |
| *** mrmartin has joined #openstack-infra | 04:42 | |
| *** fawadkhaliq has quit IRC | 04:43 | |
| *** ibiris_away is now known as ibiris | 04:43 | |
| *** prad has quit IRC | 04:45 | |
| dtroyer | same ole thing, uncapped clients (neutronclient this time) getting stevedore 1.4.0 via cliff 1.7.0 and swift choking on it as expected | 04:50 |
| lifeless | swift chokes on stevedore 1.4? | 04:52 |
| lifeless | dtroyer: in 176113? | 04:52 |
| dtroyer | juno swift does | 04:52 |
| dtroyer | http://logs.openstack.org/13/176113/2/check/check-grenade-dsvm/2a33d6a/logs/grenade.sh.txt.gz#_2015-04-22_03_44_32_983 | 04:52 |
| dtroyer | maybe it's the celio middleware doing it? | 04:53 |
| dtroyer | but that's not where it gets pulled in incorectly | 04:54 |
| dtroyer | just the first thing to notice | 04:54 |
| lifeless | loadwsgi.py", line 640, in find_egg_entry_point | 04:54 |
| lifeless | 2015-04-22 03:44:32.983 | pkg_resources.require(self.spec) | 04:54 |
| lifeless | thats the thing triggering the whinge | 04:54 |
| lifeless | so paste-deploy | 04:54 |
| lifeless | from File "/opt/stack/old/swift/swift/common/wsgi.py", line 316, in loadapp | 04:55 |
| lifeless | 2015-04-22 03:44:32.983 | ctx = loadcontext(loadwsgi.APP, conf_file, global_conf=global_conf) | 04:55 |
| dtroyer | http://logs.openstack.org/13/176113/2/check/check-grenade-dsvm/2a33d6a/logs/grenade.sh.txt.gz#_2015-04-22_03_43_36_029 is where it is being updated past the juno cap | 04:55 |
| lifeless | I wonder why it doesn't barf in regular juno tempest | 04:55 |
| lifeless | since we have 1.4 there too | 04:56 |
| *** asselin_ has quit IRC | 04:56 | |
| *** sarob has joined #openstack-infra | 04:56 | |
| dtroyer | no idea. I haven't duplicated this locally yet in plain devstack runs | 04:56 |
| lifeless | http://logs.openstack.org/13/176113/2/check/gate-tempest-dsvm-neutron-large-ops/5d3b0f8/logs/devstacklog.txt.gz#_2015-04-22_03_37_06_139 | 04:57 |
| *** deepakcs has joined #openstack-infra | 04:57 | |
| *** ddieterly has joined #openstack-infra | 04:57 | |
| dtroyer | but further down it is downgraded to 1.3.0 | 04:58 |
| dtroyer | http://logs.openstack.org/13/176113/2/check/gate-tempest-dsvm-neutron-large-ops/5d3b0f8/logs/devstacklog.txt.gz#_2015-04-22_03_37_38_739 | 04:58 |
| dtroyer | it's all a crapshoot | 04:58 |
| lifeless | yes | 04:59 |
| lifeless | anything that installs a git tree with a modified requirements | 04:59 |
| lifeless | will force stuff into alignment | 04:59 |
| lifeless | as long as it has a direct reference to $stuff | 04:59 |
| lifeless | anything with no direct reference will only be aligned if there is something intermediate with a versioned cap that is on pypi | 05:00 |
| lifeless | this is issue 2687 | 05:00 |
| lifeless | I might poke at that tomorrow | 05:00 |
| dtroyer | I don't know by number | 05:00 |
| lifeless | its conceptually straight forward to at least make it AND things together | 05:01 |
| dtroyer | I just need to get the venv-for-everyone bit finished and let the distros sort it out since they have a depsolver to work with ;) | 05:01 |
| *** sarob has quit IRC | 05:01 | |
| lifeless | dtroyer: https://github.com/pypa/pip/issues/2687 | 05:01 |
| dtroyer | yup | 05:01 |
| lifeless | dtroyer: I hope to have a solver in pip before vancouver | 05:01 |
| *** patrickeast has quit IRC | 05:01 | |
| *** ddieterly has quit IRC | 05:02 | |
| dtroyer | then you'll be showered in beer I'm sure | 05:02 |
| lifeless | sheese, I can't drink beer. | 05:02 |
| lifeless | Cider or bubbly wine plox :) | 05:02 |
| dtroyer | $COLD_BEVERAGE_OF_CHOICE | 05:02 |
| dtroyer | ok, midnight just slid by my place now too on its journey west | 05:04 |
| *** prad has joined #openstack-infra | 05:04 | |
| *** ibiris is now known as ibiris_away | 05:06 | |
| *** shashankhegde has joined #openstack-infra | 05:09 | |
| *** BharatK has joined #openstack-infra | 05:09 | |
| *** teran has quit IRC | 05:11 | |
| *** sks has joined #openstack-infra | 05:15 | |
| *** fawadkhaliq has joined #openstack-infra | 05:16 | |
| *** Hal has joined #openstack-infra | 05:19 | |
| *** Hal is now known as Guest24396 | 05:20 | |
| *** emagana has joined #openstack-infra | 05:20 | |
| *** annegentle has joined #openstack-infra | 05:24 | |
| *** agentle_ has joined #openstack-infra | 05:24 | |
| *** agentle_ has quit IRC | 05:28 | |
| *** annegentle has quit IRC | 05:29 | |
| *** yamahata has joined #openstack-infra | 05:30 | |
| *** coolsvap|afk is now known as coolsvap | 05:31 | |
| *** sdake_ has quit IRC | 05:42 | |
| *** tsg has joined #openstack-infra | 05:45 | |
| *** achanda has joined #openstack-infra | 05:49 | |
| *** Krinkle is now known as Krinkle|detached | 05:49 | |
| *** mrmartin has quit IRC | 05:49 | |
| *** dannywilson has joined #openstack-infra | 05:50 | |
| *** hdd has joined #openstack-infra | 05:51 | |
| *** otter768 has joined #openstack-infra | 05:58 | |
| *** tsg has quit IRC | 05:58 | |
| *** ddieterly has joined #openstack-infra | 05:59 | |
| *** tsg has joined #openstack-infra | 05:59 | |
| *** sushilkm has joined #openstack-infra | 05:59 | |
| *** sushilkm has left #openstack-infra | 05:59 | |
| *** otter768 has quit IRC | 06:03 | |
| *** ddieterly has quit IRC | 06:03 | |
| *** mrunge_ has joined #openstack-infra | 06:03 | |
| *** mrunge_ has quit IRC | 06:04 | |
| *** yolanda has joined #openstack-infra | 06:08 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** mrunge has joined #openstack-infra | 06:11 | |
| *** tnovacik has joined #openstack-infra | 06:11 | |
| *** tsg has quit IRC | 06:14 | |
| *** ildikov has quit IRC | 06:15 | |
| *** Ala has joined #openstack-infra | 06:17 | |
| *** achanda has quit IRC | 06:20 | |
| *** yolanda has quit IRC | 06:20 | |
| *** yolanda has joined #openstack-infra | 06:20 | |
| *** achanda has joined #openstack-infra | 06:21 | |
| *** agentle_ has joined #openstack-infra | 06:24 | |
| *** annegentle has joined #openstack-infra | 06:25 | |
| *** davideagnello has joined #openstack-infra | 06:26 | |
| *** fedexo has joined #openstack-infra | 06:26 | |
| *** agentle_ has quit IRC | 06:29 | |
| *** isviridov is now known as isviridov_away | 06:29 | |
| *** annegentle has quit IRC | 06:29 | |
| *** woodster_ has quit IRC | 06:30 | |
| *** HeOS has quit IRC | 06:31 | |
| *** davideagnello has quit IRC | 06:31 | |
| *** stevemar has quit IRC | 06:31 | |
| *** sdake has joined #openstack-infra | 06:34 | |
| *** mahito has quit IRC | 06:38 | |
| *** vlaza has joined #openstack-infra | 06:39 | |
| *** Guest24396 has quit IRC | 06:40 | |
| *** markvoelker has joined #openstack-infra | 06:40 | |
| *** _nadya_ has joined #openstack-infra | 06:43 | |
| *** markvoelker has quit IRC | 06:45 | |
| *** sarob has joined #openstack-infra | 06:45 | |
| *** _nadya_ has quit IRC | 06:48 | |
| *** sarob has quit IRC | 06:49 | |
| *** kaisers has quit IRC | 06:50 | |
| *** shashankhegde has quit IRC | 06:51 | |
| *** ptoohill has quit IRC | 06:53 | |
| *** emagana has quit IRC | 06:53 | |
| *** MaxV has joined #openstack-infra | 06:56 | |
| *** dtantsur|afk is now known as dtantsur | 06:56 | |
| *** ptoohill has joined #openstack-infra | 06:57 | |
| *** luqas has joined #openstack-infra | 06:57 | |
| *** luqas has quit IRC | 06:59 | |
| *** ddieterly has joined #openstack-infra | 06:59 | |
| yamamoto | zuul gurus around? | 07:00 |
| *** mrmartin has joined #openstack-infra | 07:02 | |
| *** kaisers has joined #openstack-infra | 07:03 | |
| *** ddieterly has quit IRC | 07:04 | |
| *** shardy has joined #openstack-infra | 07:04 | |
| *** mrodden has quit IRC | 07:06 | |
| *** mrodden has joined #openstack-infra | 07:07 | |
| *** funzo has joined #openstack-infra | 07:09 | |
| *** weshay has joined #openstack-infra | 07:10 | |
| *** kaisers has quit IRC | 07:11 | |
| *** jyuso1 has joined #openstack-infra | 07:12 | |
| *** scheuran has joined #openstack-infra | 07:12 | |
| *** doude has joined #openstack-infra | 07:12 | |
| *** coolsvap is now known as coolsvap|afk | 07:13 | |
| *** luqas has joined #openstack-infra | 07:14 | |
| *** luqas has quit IRC | 07:15 | |
| *** luqas has joined #openstack-infra | 07:16 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Neutron QoS Meeting https://review.openstack.org/176193 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Monasca Meeting details https://review.openstack.org/176194 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add OpenStack Community Infrastructure Meeting https://review.openstack.org/176195 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add translation test meeting https://review.openstack.org/176196 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Install guide meeting https://review.openstack.org/176197 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Networking Gudie update hangout https://review.openstack.org/176198 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add the Win the Enterprise meetings https://review.openstack.org/176199 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Akanda Team Meeting https://review.openstack.org/176200 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Neutron Service Chaining Meeting https://review.openstack.org/176201 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add log WG meeting https://review.openstack.org/176202 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add OpenStackClient Team Meeting https://review.openstack.org/176203 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Cue team meeting https://review.openstack.org/176204 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Chef Cookbook meeting. https://review.openstack.org/176205 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add magnetoDB Team Meeting https://review.openstack.org/176206 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add DefCore Progress Update Meeting. https://review.openstack.org/176207 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Add Community Office hours. https://review.openstack.org/176208 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Alternate Sahara meeting https://review.openstack.org/176209 | 07:19 |
| openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: Update Neutron VPNaaS Meeting https://review.openstack.org/176210 | 07:19 |
| *** mrodden has quit IRC | 07:20 | |
| *** mrodden has joined #openstack-infra | 07:21 | |
| openstackgerrit | Merged openstack-infra/shade: Invalidate user cache on user create https://review.openstack.org/173589 | 07:21 |
| *** MaxV has quit IRC | 07:23 | |
| *** sergsh has joined #openstack-infra | 07:25 | |
| *** agentle_ has joined #openstack-infra | 07:25 | |
| *** annegentle has joined #openstack-infra | 07:25 | |
| *** kaisers has joined #openstack-infra | 07:26 | |
| *** achanda has quit IRC | 07:29 | |
| *** agentle_ has quit IRC | 07:29 | |
| *** liusheng has joined #openstack-infra | 07:30 | |
| *** annegentle has quit IRC | 07:31 | |
| *** achanda_ has joined #openstack-infra | 07:32 | |
| *** Longgeek has joined #openstack-infra | 07:35 | |
| *** sdake has quit IRC | 07:35 | |
| *** luqas has quit IRC | 07:36 | |
| *** hashar has joined #openstack-infra | 07:36 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/puppet-openstackci: Add generic zuul manifests https://review.openstack.org/175970 | 07:38 |
| *** kaisers has quit IRC | 07:39 | |
| *** markvoelker has joined #openstack-infra | 07:41 | |
| *** arxcruz has joined #openstack-infra | 07:41 | |
| *** luqas has joined #openstack-infra | 07:42 | |
| *** yolanda has quit IRC | 07:43 | |
| *** jcoufal has joined #openstack-infra | 07:43 | |
| nibalizer | fungi: anteaya good ramp up projects for an opw intern might be: Run stackalytics in infra: https://review.openstack.org/#/c/98656/ and Run livegrep in infra: https://review.openstack.org/#/c/96877/ https://review.openstack.org/#/c/129726/ | 07:43 |
| *** btully has quit IRC | 07:44 | |
| *** andreykurilin__ has joined #openstack-infra | 07:44 | |
| *** fedexo has quit IRC | 07:44 | |
| *** achanda_ has quit IRC | 07:45 | |
| *** yolanda has joined #openstack-infra | 07:46 | |
| *** markvoelker has quit IRC | 07:46 | |
| jeblair | pleia2, clarkb: ^ | 07:46 |
| jeblair | nibalizer: great ideas! | 07:46 |
| *** sdake has joined #openstack-infra | 07:47 | |
| *** jistr has joined #openstack-infra | 07:50 | |
| openstackgerrit | Michal Vanco proposed openstack-infra/jenkins-job-builder: Support for Xvfb plugin https://review.openstack.org/175758 | 07:51 |
| *** ala_ has joined #openstack-infra | 07:53 | |
| *** Ala has quit IRC | 07:54 | |
| *** kaisers has joined #openstack-infra | 07:55 | |
| *** yolanda has quit IRC | 07:55 | |
| *** MaxV has joined #openstack-infra | 07:56 | |
| *** otter768 has joined #openstack-infra | 07:59 | |
| *** sdake has quit IRC | 08:00 | |
| *** ddieterly has joined #openstack-infra | 08:00 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/project-config: Move tripleo-puppet-elements to tripleo-core https://review.openstack.org/176225 | 08:01 |
| openstackgerrit | Jiri Stransky proposed openstack-infra/project-config: Add tripleo-core to puppet-tripleo https://review.openstack.org/176226 | 08:01 |
| *** yfried__ is now known as yfried|afk | 08:02 | |
| *** otter768 has quit IRC | 08:04 | |
| *** yamahata has quit IRC | 08:04 | |
| *** alexpilotti has joined #openstack-infra | 08:04 | |
| *** ddieterly has quit IRC | 08:05 | |
| *** isviridov_away is now known as isviridov | 08:09 | |
| *** ildikov has joined #openstack-infra | 08:10 | |
| *** ibiris_away is now known as ibiris | 08:10 | |
| *** Hal has joined #openstack-infra | 08:13 | |
| *** teran has joined #openstack-infra | 08:13 | |
| *** ildikov has quit IRC | 08:13 | |
| *** Hal is now known as Guest96116 | 08:14 | |
| *** Guest96116 has quit IRC | 08:18 | |
| *** ajo has joined #openstack-infra | 08:18 | |
| openstackgerrit | Angus Salkeld proposed openstack/requirements: Exclude novaclient 2.24.0 as it breaks heat's gate https://review.openstack.org/176232 | 08:19 |
| *** ildikov has joined #openstack-infra | 08:19 | |
| *** sandywalsh has quit IRC | 08:24 | |
| *** jamespage has quit IRC | 08:24 | |
| *** jamespage has joined #openstack-infra | 08:24 | |
| *** sandywalsh has joined #openstack-infra | 08:24 | |
| *** jamespage has quit IRC | 08:24 | |
| *** jamespage has joined #openstack-infra | 08:24 | |
| *** derekh has joined #openstack-infra | 08:25 | |
| *** yfried|afk has quit IRC | 08:26 | |
| *** yfried|afk has joined #openstack-infra | 08:26 | |
| *** __mimir has joined #openstack-infra | 08:26 | |
| *** annegentle has joined #openstack-infra | 08:26 | |
| *** _nadya_ has joined #openstack-infra | 08:26 | |
| jyuso1 | Hi infra.Intel PCI/NGFW/SRIOV/Hardware-Meters CI will be offline this week. | 08:28 |
| jyuso1 | We have to tidy the server room.Our CI servers and log-server will be shutdown until next week.We¡¯ll resume those servers ASAP.Thanks:) | 08:29 |
| *** krtaylor has quit IRC | 08:29 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag https://review.openstack.org/175849 | 08:31 |
| *** annegentle has quit IRC | 08:31 | |
| nibalizer | jeblair: so this is the thing that puppetlabs has 'vmpooler' kind of like nodepool, especially nodepool post zuulv3 I think | 08:32 |
| *** smyers has quit IRC | 08:32 | |
| nibalizer | http://loserfish.org/~scott/talks/2015-04-vmpooler-pdxdevops.pdf | 08:32 |
| *** prad has quit IRC | 08:32 | |
| *** med_ has quit IRC | 08:32 | |
| *** hamzy has quit IRC | 08:32 | |
| *** hamzy has joined #openstack-infra | 08:34 | |
| *** prad has joined #openstack-infra | 08:34 | |
| *** krtaylor has joined #openstack-infra | 08:34 | |
| *** juggler has joined #openstack-infra | 08:35 | |
| *** med_ has joined #openstack-infra | 08:35 | |
| *** med_ has quit IRC | 08:36 | |
| *** med_ has joined #openstack-infra | 08:36 | |
| *** fawadkhaliq has quit IRC | 08:36 | |
| *** HeOS has joined #openstack-infra | 08:36 | |
| *** smyers has joined #openstack-infra | 08:37 | |
| *** juggler_ has quit IRC | 08:37 | |
| jeblair | nibalizer: yay nodepool in ruby :) | 08:38 |
| *** coolsvap|afk is now known as coolsvap | 08:40 | |
| *** juggler has quit IRC | 08:41 | |
| jeblair | hrm, i get the following from etherpad: | 08:42 |
| jeblair | Error: Attempt to load undefined module. in https://etherpad.openstack.org/static/js/require-kernel.js (line 1) | 08:42 |
| *** markvoelker has joined #openstack-infra | 08:42 | |
| *** yolanda has joined #openstack-infra | 08:42 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/project-config: Add support for backport-potential commit flag https://review.openstack.org/175849 | 08:44 |
| jeblair | that might have been a local error. weird. i don't know if it was a network problem or if restarting my browser fixed it | 08:44 |
| *** dannywilson has quit IRC | 08:45 | |
| *** dannywilson has joined #openstack-infra | 08:46 | |
| *** ssam2 has joined #openstack-infra | 08:46 | |
| *** markvoelker has quit IRC | 08:47 | |
| ttx | jeblair: you up ? | 08:48 |
| *** andreykurilin__ has quit IRC | 08:49 | |
| ttx | jeblair: We could use some ninja-merging on https://review.openstack.org/#/c/176113/ | 08:49 |
| ttx | sounds preferable to disabling grenade on stable/juno temporarily. | 08:50 |
| *** dannywilson has quit IRC | 08:50 | |
| ttx | (though the latter might prove necessary if the house of cards continues to fall) | 08:50 |
| *** andreykurilin__ has joined #openstack-infra | 08:50 | |
| *** coolsvap is now known as coolsvap|afk | 08:51 | |
| *** devvesa has joined #openstack-infra | 08:52 | |
| *** prad has quit IRC | 08:53 | |
| *** jklare has quit IRC | 08:53 | |
| *** jklare has joined #openstack-infra | 08:55 | |
| *** _nadya_ has quit IRC | 08:55 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:55 | |
| *** devvesa has quit IRC | 08:57 | |
| *** GnomeKing has joined #openstack-infra | 08:57 | |
| *** devvesa has joined #openstack-infra | 08:57 | |
| *** sputnik13 has joined #openstack-infra | 08:58 | |
| *** GnomeKing is now known as Guest28882 | 08:58 | |
| *** teran has quit IRC | 08:59 | |
| *** prad has joined #openstack-infra | 08:59 | |
| *** pelix has joined #openstack-infra | 09:01 | |
| *** ihrachyshka has joined #openstack-infra | 09:01 | |
| *** sputnik13 has quit IRC | 09:02 | |
| *** juggler has joined #openstack-infra | 09:05 | |
| *** fawadkhaliq has joined #openstack-infra | 09:07 | |
| *** jlanoux has joined #openstack-infra | 09:11 | |
| *** Hal has joined #openstack-infra | 09:11 | |
| *** __mimir has quit IRC | 09:11 | |
| *** Hal is now known as Guest5363 | 09:11 | |
| *** ala_ has quit IRC | 09:12 | |
| *** dimtruck is now known as zz_dimtruck | 09:14 | |
| *** Kennan2 has joined #openstack-infra | 09:19 | |
| *** Kennan has quit IRC | 09:19 | |
| *** afazekas_ has joined #openstack-infra | 09:21 | |
| *** adiantum has joined #openstack-infra | 09:22 | |
| *** __mimir has joined #openstack-infra | 09:25 | |
| *** agentle_ has joined #openstack-infra | 09:27 | |
| *** annegentle has joined #openstack-infra | 09:27 | |
| *** dtantsur is now known as dtantsur|brb | 09:29 | |
| *** e0ne has joined #openstack-infra | 09:30 | |
| *** tnovacik has quit IRC | 09:30 | |
| *** annegentle has quit IRC | 09:31 | |
| *** agentle_ has quit IRC | 09:32 | |
| *** ala_ has joined #openstack-infra | 09:34 | |
| jeblair | ttx: what does that error mean? | 09:34 |
| *** yamamoto has quit IRC | 09:34 | |
| jeblair | ttx: (or on other words, why would force-merging that change fix the error reported in that change?) | 09:36 |
| *** kaisers has quit IRC | 09:40 | |
| *** ajo has quit IRC | 09:41 | |
| *** markvoelker has joined #openstack-infra | 09:43 | |
| *** kaisers has joined #openstack-infra | 09:43 | |
| *** dguitarbite has joined #openstack-infra | 09:45 | |
| *** ajo has joined #openstack-infra | 09:46 | |
| *** shardy_ has joined #openstack-infra | 09:46 | |
| *** markvoelker has quit IRC | 09:47 | |
| *** shardy has quit IRC | 09:48 | |
| *** shardy_ has quit IRC | 09:52 | |
| *** shardy has joined #openstack-infra | 09:52 | |
| *** hashar is now known as hasharLunch | 09:54 | |
| *** ajo has quit IRC | 09:58 | |
| *** jogo has quit IRC | 10:01 | |
| *** palar has joined #openstack-infra | 10:01 | |
| *** __mimir has quit IRC | 10:02 | |
| *** ddieterly has joined #openstack-infra | 10:02 | |
| *** andreykurilin__ has quit IRC | 10:04 | |
| *** ssam2 has quit IRC | 10:04 | |
| *** jogo has joined #openstack-infra | 10:06 | |
| *** ddieterly has quit IRC | 10:07 | |
| openstackgerrit | Sean Dague proposed openstack/requirements: uncap stevedore entirely https://review.openstack.org/176243 | 10:10 |
| *** teran has joined #openstack-infra | 10:12 | |
| *** teran_ has joined #openstack-infra | 10:14 | |
| *** ajo has joined #openstack-infra | 10:15 | |
| *** teran__ has joined #openstack-infra | 10:15 | |
| *** teran__ has quit IRC | 10:16 | |
| *** teran__ has joined #openstack-infra | 10:16 | |
| *** teran has quit IRC | 10:16 | |
| *** teran_ has quit IRC | 10:18 | |
| *** ssam2 has joined #openstack-infra | 10:19 | |
| openstackgerrit | Sean Dague proposed openstack/requirements: raise stevedore requirements https://review.openstack.org/176113 | 10:20 |
| sdague | ttx: I think if both of those get approved, things unwedge | 10:20 |
| jeblair | sdague: can they go through normal approval? | 10:22 |
| sdague | I *think* so? | 10:22 |
| sdague | honestly, there is a lot of cascade fail here | 10:22 |
| *** sarob has joined #openstack-infra | 10:23 | |
| sdague | and it takes an hour to get back results, so no idea | 10:23 |
| sdague | also, I've been awake for 20 minutes, so take all statements with "not fully awake yet" | 10:23 |
| jeblair | *nod* | 10:23 |
| *** yfried|afk is now known as yfried__ | 10:24 | |
| sdague | so, in the list of things we did not anticipate in the requirements cap path | 10:25 |
| sdague | 1) it would require a giant flush like we had yesterday, which mean a bunch of bugs | 10:25 |
| sdague | 2) we're actually still in a 2 release coupling for upgrade from juno -> liberty, because the grenade juno -> kilo, kilo -> liberty config is not merged | 10:26 |
| *** ssam2_ has joined #openstack-infra | 10:26 | |
| sdague | which code wise isn't a problem | 10:26 |
| sdague | but with narrow requirements, the odds of getting deadlocked there is a high | 10:27 |
| *** Guest28882 is now known as BobBall | 10:27 | |
| sdague | I also fully expect icehouse to just completely fail on https://review.openstack.org/176243 for unrelated reasons | 10:27 |
| sdague | given that I don't know if anyone has tried an icehouse patch since the flush | 10:27 |
| *** nfedotov has joined #openstack-infra | 10:27 | |
| *** deepakcs has quit IRC | 10:28 | |
| *** annegentle has joined #openstack-infra | 10:28 | |
| *** ssam2 has quit IRC | 10:28 | |
| *** sarob has quit IRC | 10:28 | |
| sdague | so... the problem is we can't land liberty patches right now until we fix icehouse | 10:28 |
| sdague | which, does not seem to be the kind of situation we want to be in | 10:28 |
| sdague | and that a force merge of https://review.openstack.org/176113 will fix | 10:28 |
| *** annegentle has quit IRC | 10:33 | |
| *** yfried__ is now known as yfried|afk | 10:34 | |
| *** jlanoux has quit IRC | 10:35 | |
| *** yamamoto has joined #openstack-infra | 10:35 | |
| *** _nadya_ has joined #openstack-infra | 10:40 | |
| *** yamamoto has quit IRC | 10:40 | |
| ttx | jeblair: the fix in itself fixes stable/juno, and therefore by extension stable/kilo and master. I'd rather unblock stuff now rather than lose a few more hours fixing stable/icehouse first | 10:41 |
| ttx | the grenade fail is not an upgarde fail, it's a "stable/icehouse is wedged" fail | 10:41 |
| ttx | so +1 to force merging to break the cascade and at least get kilo & master unblocked | 10:43 |
| ttx | Then we can unblock juno by fixing icehouse. | 10:43 |
| *** markvoelker has joined #openstack-infra | 10:43 | |
| ttx | alternative would be to remove grenade tests from juno, which sounds (1) worse and (2) farther away | 10:45 |
| *** hyakuhei has joined #openstack-infra | 10:46 | |
| ttx | sdague: or we can get lucky and have 176243 miraculously fix icehouse. | 10:46 |
| sdague | ttx: yeh, no idea, are there other icehouse recent runs somewhere? | 10:46 |
| ttx | but I fear it won't be enough, like you said | 10:46 |
| *** yamamoto has joined #openstack-infra | 10:46 | |
| sdague | it fixes the tactical issue that I saw on a failed juno run | 10:46 |
| ttx | sdague: let's wait for the test results on that one, looks like we won't get anyone to ninja-merge it in the next hour anyway | 10:47 |
| hyakuhei | Hey ttx, can you take a quick look at https://review.openstack.org/176251 please - just a typo fix. | 10:48 |
| *** markvoelker has quit IRC | 10:48 | |
| ttx | hyakuhei: approved | 10:49 |
| *** ssam2_ has quit IRC | 10:49 | |
| hyakuhei | Thanks :) | 10:49 |
| *** e0ne is now known as e0ne_ | 10:51 | |
| ttx | fungi: you around ? | 10:52 |
| ttx | bah / me lunches. | 10:52 |
| *** yfried|afk is now known as yfried__ | 10:53 | |
| *** ssam2 has joined #openstack-infra | 10:53 | |
| *** isviridov is now known as isviridov_away | 10:53 | |
| *** juggler_ has joined #openstack-infra | 10:54 | |
| *** e0ne_ is now known as e0ne | 10:55 | |
| *** zhiwei has quit IRC | 10:56 | |
| *** dtantsur|brb is now known as dtantsur | 10:58 | |
| *** juggler has quit IRC | 10:58 | |
| *** hyakuhei has quit IRC | 10:58 | |
| *** erlon has joined #openstack-infra | 10:59 | |
| *** deepakcs has joined #openstack-infra | 11:01 | |
| *** ildikov has quit IRC | 11:01 | |
| openstackgerrit | Sean Dague proposed openstack/requirements: add pycparser tracking to what-broke https://review.openstack.org/176120 | 11:02 |
| *** yfried__ is now known as yfried|afk | 11:04 | |
| *** marcusvrn has joined #openstack-infra | 11:04 | |
| *** _nadya_ has quit IRC | 11:05 | |
| *** Kennan2 has quit IRC | 11:08 | |
| BharatK | sdague: ping | 11:08 |
| BharatK | sdague: As you suggested, created a new for glusterfs, please review the patch https://review.openstack.org/#/c/173408/ | 11:09 |
| *** ildikov has joined #openstack-infra | 11:16 | |
| *** jlanoux has joined #openstack-infra | 11:18 | |
| *** afazekas_ has quit IRC | 11:18 | |
| *** Kennan has joined #openstack-infra | 11:23 | |
| *** doug-fish has joined #openstack-infra | 11:23 | |
| *** yfried|afk is now known as yfried__ | 11:24 | |
| *** hasharLunch is now known as hashar | 11:26 | |
| fungi | ttx: around now--what's up? | 11:26 |
| *** e0ne is now known as e0ne_ | 11:27 | |
| *** agentle_ has joined #openstack-infra | 11:28 | |
| *** shardy has quit IRC | 11:28 | |
| *** annegentle has joined #openstack-infra | 11:28 | |
| *** agentle_ has quit IRC | 11:32 | |
| *** afazekas_ has joined #openstack-infra | 11:34 | |
| *** annegentle has quit IRC | 11:34 | |
| ttx | fungi: we could unblock gate using a ninjafix | 11:34 |
| ttx | well, hmm, might not be necessary after all | 11:34 |
| ttx | catching up | 11:34 |
| fungi | looks like 176113 passed tests | 11:35 |
| * fungi too is catching up | 11:35 | |
| ttx | yeah, so by miracle it may fix things | 11:35 |
| fungi | not yet caffeinated | 11:35 |
| ttx | let's see how it goes | 11:35 |
| fungi | estimated half an hour to merge | 11:36 |
| *** woodster_ has joined #openstack-infra | 11:37 | |
| *** yfried__ is now known as yfried|afk | 11:37 | |
| *** e0ne_ has quit IRC | 11:38 | |
| *** smarcet has joined #openstack-infra | 11:38 | |
| *** esker has quit IRC | 11:38 | |
| *** bknudson has quit IRC | 11:40 | |
| *** ldnunes has joined #openstack-infra | 11:43 | |
| deepakcs | sdague, would appreciate if you can help review https://review.openstack.org/#/c/173408/ - will help cinder-glusterfs ci to turn back green :) | 11:43 |
| *** wuhg has joined #openstack-infra | 11:43 | |
| *** mpaolino has joined #openstack-infra | 11:44 | |
| sdague | deepakcs: please stop pinging me, I've been unwinding gate wedges, and this is very far down my priority list | 11:44 |
| *** markvoelker has joined #openstack-infra | 11:44 | |
| *** sslypushenko has joined #openstack-infra | 11:45 | |
| deepakcs | sdague, sure, i had asked others, but they said you need to look at it! | 11:45 |
| *** markvoelker has quit IRC | 11:45 | |
| *** markvoelker has joined #openstack-infra | 11:45 | |
| *** mugsie has quit IRC | 11:46 | |
| *** mugsie has joined #openstack-infra | 11:46 | |
| *** yfried|afk is now known as yfried__ | 11:46 | |
| *** baoli has joined #openstack-infra | 11:53 | |
| *** yolanda has quit IRC | 11:55 | |
| fungi | deepakcs: it will probably need to wait until the kilo release. official project teams are all very busy trying to get the release completed for the next couple weeks | 11:55 |
| *** ildikov has quit IRC | 11:56 | |
| *** yfried__ is now known as yfried|afk | 11:56 | |
| deepakcs | fungi, thanks for the gracious answer, that helps :) | 11:57 |
| *** e0ne has joined #openstack-infra | 11:58 | |
| *** otter768 has joined #openstack-infra | 12:00 | |
| *** ddieterly has joined #openstack-infra | 12:03 | |
| *** ayoung has quit IRC | 12:03 | |
| *** mmedvede has quit IRC | 12:04 | |
| *** otter768 has quit IRC | 12:05 | |
| sdague | fungi: and neutron rando fail | 12:05 |
| *** smarcet has quit IRC | 12:05 | |
| fungi | sdague: which bug? i missed it in the scrollback i think | 12:06 |
| sdague | so please ninja merge https://review.openstack.org/#/c/176243/ and https://review.openstack.org/#/c/176113/ | 12:06 |
| sdague | the requirements fixes | 12:06 |
| fungi | looking | 12:06 |
| fungi | uncap stevedore in icehouse, and raise the stevedore cap in juno | 12:07 |
| sdague | yep | 12:07 |
| fungi | looks like 176243 passes tests but 176113 does not | 12:07 |
| *** luqas has quit IRC | 12:07 | |
| sdague | 176113 passed tests | 12:07 |
| *** ddieterly has quit IRC | 12:08 | |
| fungi | ahh, yep. so it did | 12:08 |
| *** luqas has joined #openstack-infra | 12:08 | |
| fungi | i was looking at a partly loaded page. need to finish getting up and get to my normal computer | 12:08 |
| *** ildikov has joined #openstack-infra | 12:09 | |
| fungi | too many channels full of fire. proof i should not look at irc before getting out of bed ;) | 12:09 |
| *** deepakcs has quit IRC | 12:09 | |
| *** yolanda has joined #openstack-infra | 12:09 | |
| sdague | well, if requirements-core had force push on requirements, we wouldn't have to wake you :) | 12:10 |
| *** BharatK has quit IRC | 12:10 | |
| *** vladiskuz has quit IRC | 12:11 | |
| *** sarob has joined #openstack-infra | 12:12 | |
| *** dprince has joined #openstack-infra | 12:13 | |
| *** fawadkhaliq has quit IRC | 12:13 | |
| fungi | i don't think anyone should ever force-push a chance which can pass tests as long as the systems to run those jobs are up and working. they're enqueued at the head of the gate pipeline now | 12:13 |
| fungi | s/chance/change/ | 12:14 |
| sdague | fungi: so they can die on another rando fail and we can keep doing this for hours? | 12:14 |
| sdague | those are provably correct changes at this point | 12:14 |
| *** mmedvede has joined #openstack-infra | 12:14 | |
| *** tnovacik has joined #openstack-infra | 12:14 | |
| *** jaypipes has joined #openstack-infra | 12:15 | |
| sdague | and this continues to block a most projects in OpenStack from getting stuff done | 12:16 |
| fungi | sounds like we should probably stop running integration tests in the gate for now | 12:16 |
| fungi | if none of them can complete reliably in a reasonable amount of time | 12:16 |
| fungi | i'll get the job change ready | 12:16 |
| *** ociuhandu has joined #openstack-infra | 12:17 | |
| fungi | we can switch to run them in check only. should get things moving a whole lot faster and we can get this back on track sooner | 12:17 |
| sdague | I'm confused. | 12:17 |
| sdague | We've seen these changes pass the relevant tests | 12:17 |
| sdague | when they failed it was unrelated | 12:18 |
| fungi | they passed in check, right? we run them in check | 12:18 |
| *** tsg has joined #openstack-infra | 12:18 | |
| fungi | it sounds like most of the delay is getting integration jobs through the gate since they're unreliable | 12:18 |
| *** sarob has quit IRC | 12:18 | |
| *** tiswanso has joined #openstack-infra | 12:18 | |
| *** cdent has joined #openstack-infra | 12:18 | |
| fungi | s/unreliable/fragile/ | 12:18 |
| *** rfolco has joined #openstack-infra | 12:19 | |
| fungi | just trying to figure out how to speed this up so we don't need to bypass gating completely instead | 12:19 |
| fungi | some testing is better than none, after all | 12:19 |
| sdague | we do it by relying on the fact that there are smart humans | 12:19 |
| sdague | that know better than machines some times | 12:19 |
| ekarlso | what's the recommended way for testing / mocking clients atm ? | 12:20 |
| *** baoli has quit IRC | 12:21 | |
| fungi | okay, so we do still want to run integration testing on other changes in the gate pipeline, just not these changes because we're pretty sure they shouldn't have adverse interactions with each other beyond maybe grenade, since they're patches to icehouse and juno? | 12:21 |
| sdague | that's been my frustration with this argument that we get into for stuff like this, there are times when smart humans actually have figured out the fix, and have data to support it, but are powerless to address that fix without fixing everything else first | 12:21 |
| jeblair | sdague, fungi: all i care about is that we stop wedging things. is someone working on figuring out how we got into this state and preventing it in the future? if so, i'm okay force-merging changes that unwedge us. | 12:21 |
| ttx | well, we basically know that those two fixes get us closer to a world of good, and have to wait 6 hours to get them in | 12:22 |
| sdague | fungi: not only are we pretty sure, we tested master patches Depends-On these changes, and saw them pass | 12:22 |
| sdague | that's about as solid data as I can give you | 12:23 |
| ttx | 6 hours we can't really afford to lose at this point. I know they directly translate into 6 hours less of weekend time at this point | 12:23 |
| sdague | https://review.openstack.org/#/c/176012/ | 12:23 |
| fungi | i think those in favor of capping requirements did not realize it was going to result in a more time-consuming puzzle to solve than the weekly random breakages we could run down in an hour or two. but undoing all this now would probably be even more work | 12:23 |
| *** dhritishikhar has joined #openstack-infra | 12:23 | |
| sdague | fungi: yes, there were many dragons | 12:23 |
| fungi | so i agree unwedging things at this point probably means pressing forward | 12:24 |
| fungi | and hoping we're not making it worse | 12:24 |
| ttx | pushing those at this point just means winning back 1+ hour | 12:24 |
| ttx | since we know they will pass at some point | 12:24 |
| ttx | and we keep on pushing them | 12:25 |
| ttx | If everything else was not blocked we could afford losing 1+ hour | 12:25 |
| fungi | i'm fine manually merging them now if it gets everyone closer to having everything working again sooner | 12:25 |
| sdague | jeblair: in honesty, no one has figured out a system that can prevent wedging given that python doesn't have a real package manager | 12:25 |
| *** dhritishikhar has quit IRC | 12:26 | |
| ttx | seriously, we can't lose another day at this point. This is the most fucked-up release period ever | 12:26 |
| *** dhritishikhar has joined #openstack-infra | 12:27 | |
| ttx | it will directly translate in pain and lack of quality, so every hour count | 12:27 |
| fungi | so anyway, it sounds like jeblair agrees that getting us un-wedged sooner even if it means officially bypassing gating is an acceptable solution | 12:27 |
| fungi | i'll go ahead and do that for these two requirements changes | 12:28 |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** annegentle has joined #openstack-infra | 12:29 | |
| *** arielb has joined #openstack-infra | 12:29 | |
| *** mwagner_lap has quit IRC | 12:31 | |
| openstackgerrit | Merged openstack/requirements: uncap stevedore entirely https://review.openstack.org/176243 | 12:31 |
| *** agentle_ has joined #openstack-infra | 12:31 | |
| jeblair | yeah. my two criteria are that someone believes that it will really fix the problem and any current failing tests don't look like new regressions. i'm happy with sdague's evaluation there | 12:31 |
| openstackgerrit | Merged openstack/requirements: raise stevedore requirements https://review.openstack.org/176113 | 12:31 |
| jeblair | the other part is that we figure out how to prevent something similar in the future. i expect we may need to do that at the summit | 12:31 |
| sdague | jeblair: honestly, given all the shortcomings of pip, I think that even in the medium term the solution includes human override by requirements core | 12:32 |
| jeblair | because clearly this is a really complex issue and i doubt we can do that right now | 12:32 |
| sdague | because I think it's probably a year before we have a system robust enough to address this, even if we start now | 12:32 |
| jeblair | sdague: that's not really acceptable. we need to understand what's causing the wedging. this is a new behavior -- what's causing it? is it because we're doing grenade testing now? | 12:33 |
| sdague | it's because we've got 200 pip trees | 12:33 |
| ttx | jeblair: I think the new element in the equation is the capping. | 12:33 |
| sdague | it's not new behavior, it's the same behavior we've always seen | 12:33 |
| ttx | And its ability to leak | 12:34 |
| ttx | which grenade has the ability to magnify | 12:34 |
| * dhellmann catches up | 12:34 | |
| jeblair | okay, so it sounds like our capping strategy isn't sufficient. perhaps it needs complete tranitive dependencies? | 12:34 |
| fungi | well, a big chunk of it is that pip install multiple times in the same environment produces deterministic but uncontrolled behavior, and that is how devstack operates right now | 12:35 |
| sdague | right, so things that made this worse | 12:35 |
| jeblair | fungi: right, but complete fixed transitive deps works around that, right? | 12:35 |
| *** dhritishikhar_ has joined #openstack-infra | 12:36 | |
| sdague | 1) installing more things from pypi instead of git trees | 12:36 |
| dhellmann | fungi: right, so we talked yesterday about making devstack build a list of requirements for the things it is going to install, and then install them all at once | 12:36 |
| fungi | jeblair: yes | 12:36 |
| *** dhritishikhar has quit IRC | 12:36 | |
| sdague | so the fact that we make projects use installed software, means we gave up control of the dependencies of that installed software | 12:36 |
| sdague | we no longer control python-barbicanclient dependencies | 12:37 |
| sdague | at least not point in time | 12:37 |
| sdague | we have eventual consistency, which is not good enough | 12:37 |
| *** Prophidys1 has left #openstack-infra | 12:37 | |
| jeblair | sdague: ah, right because we aren't overwriting its requirements.txt file? | 12:37 |
| sdague | correct | 12:37 |
| dhellmann | sdague: true, but if we feed pip 2 requirements that conflicting dependencies *at the same time* it will correctly report an error, right? | 12:37 |
| dhellmann | ^that^that have | 12:38 |
| sdague | dhellmann: sure, but the problem here isn't that, it's the transitive unwind | 12:38 |
| EmilienM | good morning, can we get a chance to have a review on project-config/tripleo/puppet? https://review.openstack.org/#/c/176070/ (small patch) | 12:38 |
| dhellmann | sdague: right, my point was that a change to devstack might help with that | 12:38 |
| sdague | jeblair: and the reality is, every time anyone thinks they've addressed 1 problem here, we uncover a couple of other dragons | 12:38 |
| sdague | and then the energy to solve the long game gets lost in tactical unwind firefights | 12:39 |
| sdague | which is why I believe human override is an important part of the medium term game | 12:39 |
| jeblair | sdague: right. so have we learned enough to have a useful conversation at summit? | 12:39 |
| sdague | define "useful conversation" | 12:40 |
| sdague | we've learned a lot | 12:40 |
| *** claudiub has joined #openstack-infra | 12:40 | |
| *** dhritishikhar_ has quit IRC | 12:40 | |
| sdague | I don't think anyone has a solution sorted | 12:40 |
| *** e0ne is now known as e0ne_ | 12:41 | |
| jeblair | sdague: i think we need to revsit the things that got us here -- what are we trying to do by installing from git, installing from pypi, capping requirements, etc. i think we need to step back and consider all of those things and design a solution that takes all those into account rather than patching things one at a time. | 12:41 |
| *** gordc has joined #openstack-infra | 12:41 | |
| sdague | jeblair: sure. | 12:41 |
| sdague | there are some attempted ML threads on that right now that lifeless started | 12:42 |
| *** _nadya_ has joined #openstack-infra | 12:42 | |
| jeblair | sdague: i'm supportive of human overrides to get us to the release and to a conversation about that. i would really like to have that conversation before we start agreeing on things like "human overrides for a year" | 12:42 |
| sdague | honestly, it's probably a better ML conversation than face to face one, because there are *so* many details | 12:42 |
| sdague | and if you lose one detail somewhere, the whole thing crumbles | 12:42 |
| *** bknudson has joined #openstack-infra | 12:43 | |
| sdague | and we've convinced ourselves a few times that we had a new plan, and a detail was missed, and.... we get *yesterday* | 12:43 |
| fungi | and now i have a beatles song stuck in my head | 12:44 |
| jeblair | sdague: i think perhaps we didn't record that plan very well | 12:44 |
| jeblair | sdague: in particular, i was suprised that the requirements were capped _before_ the release | 12:44 |
| sdague | jeblair: could be, I'll be honest, I mostly took a pass on this last go around because I was too burnt out on the problem space and the fire fighting | 12:44 |
| ttx | jeblair: agree that capping oslo libs on master was not such a great idea | 12:45 |
| fungi | agreed. the earlier discussions were that we'd cut release branches from master, then immediately post-release introduce requirements caps to protect stable | 12:45 |
| ttx | It's because the general thinking was... stable/kilo requirements should only be cut when all RCs are in | 12:46 |
| *** dannywilson has joined #openstack-infra | 12:46 | |
| ttx | But maybe it's the other way around | 12:46 |
| *** kgiusti has joined #openstack-infra | 12:46 | |
| *** e0ne_ is now known as e0ne | 12:46 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job https://review.openstack.org/176286 | 12:47 |
| fungi | which would have allowed keeping momentum up through the release and then dealing with any fallout when there was less of a time crunch | 12:47 |
| ttx | fungi: a bit uncertain on the zuul status right now, why is it testing 176113 again | 12:47 |
| *** yfried|afk is now known as yfried__ | 12:48 | |
| fungi | ttx: yes, i'm performing surgery to get those excised from the gate and requeue the tempest fix which depended on them | 12:48 |
| ttx | ok | 12:48 |
| * ttx holds on approving RC2 changes until surgery is complete | 12:49 | |
| *** radez_g0n3 is now known as radez | 12:50 | |
| *** dannywilson has quit IRC | 12:50 | |
| *** tiswanso has quit IRC | 12:50 | |
| dhellmann | we capped on master to avoid having the feature freeze in oslo last for an excessive period, since we freeze before the other projects and can only really thaw after their stable branches are cut | 12:50 |
| fungi | right, the pushback on waiting for the release to cap was that clients and libraries wanted to go ahead and start developing liberty features rather than being unable to introduce breaking changes prior to kilo release | 12:52 |
| *** bswartz has quit IRC | 12:53 | |
| fungi | though in retrospect, we could have added branches to all of them and capped them on stable/kilo nee proposed/kilo | 12:53 |
| fungi | but that's where the mixed proposed vs stable branch fallback logic issue came up | 12:54 |
| *** amitgandhinz has joined #openstack-infra | 12:54 | |
| *** AJaeger has joined #openstack-infra | 12:55 | |
| *** dhritishikhar_ has joined #openstack-infra | 12:56 | |
| *** pc_m has joined #openstack-infra | 12:56 | |
| *** spzala has joined #openstack-infra | 12:56 | |
| *** baoli has joined #openstack-infra | 12:57 | |
| sdague | so, realistically, I think caps in libraries are part of the problems | 12:57 |
| sdague | global-requirements works ok for a set of things that release all at once | 12:57 |
| *** yfried__ is now known as yfried|afk | 12:58 | |
| sdague | but it's having really weird behavior on things that don't, like these libraries | 12:58 |
| *** eharney has joined #openstack-infra | 12:58 | |
| sdague | especially once you think about what needs to be released for everything to have the same oslo.config definition | 12:59 |
| *** yfried|afk is now known as yfried__ | 12:59 | |
| fungi | ttx: okay, the tempest fix is at the top of the gate now and zuul seems to be wrapping up processing its events/results queues now | 13:02 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Add a ceph job https://review.openstack.org/176292 | 13:02 |
| *** sergsh has quit IRC | 13:03 | |
| dhellmann | yeah, I'm ok with not capping the dependencies within the libraries, but we still want the apps to have theirs capped at a point that lets us create stable branches of the libs and then move on with liberty work | 13:03 |
| jeblair | sdague, dhellmann: well regardless of whether we have a session at the summit about it, i look forward to talking to the two of you about it. i think i need that in order to get on the same page. you have learned a lot that i need to learn from you. :) | 13:03 |
| *** ociuhandu has quit IRC | 13:03 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job https://review.openstack.org/176286 | 13:03 |
| jeblair | sdague, ttx: if grenade is exacerbating the situation, can we make grenade non-voting on requirements stable branches? | 13:03 |
| fungi | sdague: "libraries" is a fuzzy concept though... if you have library A which depends on B and it's incompatible with B>=2 then you need to record that somewhere such that you can run unit tests for A | 13:04 |
| *** dhritishikhar_ has quit IRC | 13:04 | |
| sdague | jeblair: so, in this case, grenade wasn't really causing any issues directly | 13:04 |
| sdague | it was just upleveling fails | 13:04 |
| *** ildikov has quit IRC | 13:04 | |
| sdague | so stable/juno being broken, broke master patches | 13:04 |
| sdague | the real issue was ./stack.sh on stable/juno didn't work | 13:05 |
| sdague | and the fix for that was blocked on ./stack.sh on stable/icehouse not working | 13:05 |
| jeblair | sdague: yeah, that's why i'm thinking that in the short/medium term that making it non-voting might be a reasonable compromise -- because we know it's a wedgeable situation so don't have the computer block human judgement there | 13:06 |
| sdague | jeblair: yeh, or make grenade non-voting on non master in general | 13:06 |
| fungi | anyway, i've come to realize that pip's current design assumes your requirements lists are not package-specific and that without modifying pip heavily to change that, the closest we can get to its intended behavior is to have job-specific (perhaps tox env specific) requirements lists rather than tying them to projects | 13:07 |
| fungi | er, rather than tying them to packages i mean | 13:07 |
| *** dkranz has joined #openstack-infra | 13:07 | |
| *** mattfarina has joined #openstack-infra | 13:07 | |
| sdague | fungi: yeh, that would be an option | 13:07 |
| sdague | again, if libraries were pretty wide in what they allowed, we could do all that upfront | 13:08 |
| *** mrmartin has quit IRC | 13:10 | |
| sdague | jeblair: so if you think a summit session is the right format, I'm happy to participate | 13:12 |
| *** afazekas_ has quit IRC | 13:13 | |
| *** yfried__ is now known as yfried|afk | 13:14 | |
| jeblair | sdague: all i'm certain about at the moment is i have some things i need to learn from you and dhellmann :) | 13:14 |
| *** dhritishikhar has joined #openstack-infra | 13:15 | |
| sdague | honestly, this might make for a reasonable sprint in the next cycle, because I feel like there is a lot to keep in ones head, but a few folks, and a few days, and a white board might go far | 13:15 |
| jeblair | sdague: yeah, that might be a good idea | 13:17 |
| *** samueldmq has joined #openstack-infra | 13:17 | |
| *** ociuhandu has joined #openstack-infra | 13:18 | |
| *** yamahata has joined #openstack-infra | 13:19 | |
| *** ildikov has joined #openstack-infra | 13:19 | |
| *** agentle_ has quit IRC | 13:20 | |
| *** annegentle has quit IRC | 13:20 | |
| *** ddieterly has joined #openstack-infra | 13:21 | |
| EmilienM | how to you generally proceed to move some repos from X to Y (ie, from stackforge to openstack) ? My question is more about users. | 13:21 |
| fungi | users move themselves? | 13:22 |
| *** mriedem_away is now known as mriedem | 13:22 | |
| EmilienM | fungi: yeah... but I'm sure we will have a lot of feedback from people having CI broken or I don't know | 13:22 |
| fungi | oh, your users are consuming source code via git | 13:23 |
| EmilienM | I'm just wondering actually | 13:23 |
| fungi | not packages | 13:23 |
| EmilienM | fungi: yes | 13:23 |
| EmilienM | fungi: RH does, but not sure about other users. Most of them are using git | 13:23 |
| *** dhritishikhar has quit IRC | 13:23 | |
| EmilienM | before moving everything to github/openstack I would like to figure out that one | 13:23 |
| *** dustins has joined #openstack-infra | 13:23 | |
| fungi | well, on the github end it's pretty simple. we transfer the repos if possible and github creates redirects | 13:24 |
| sdague | EmilienM: you probably should just send out a heads up with a 'get remote set-url' | 13:24 |
| *** esker has joined #openstack-infra | 13:24 | |
| *** peristeri has joined #openstack-infra | 13:24 | |
| fungi | EmilienM: but on the git.openstack.org end we don't have a similar redirect service | 13:24 |
| fungi | so anyone consuming from there would need to change their remote urls (or we would need to design a server-side solution) | 13:25 |
| *** bswartz has joined #openstack-infra | 13:25 | |
| EmilienM | sdague, fungi: wfm | 13:25 |
| marcusvrn | hi guys, can some jenkins expert help me? I'm trying to trigger a job using zuul, but when jenkins tries to run the job on master, it stays as pending "There are no node with label '' ". If I run manually that job, it works... | 13:27 |
| *** arielb has quit IRC | 13:27 | |
| EmilienM | fungi: more a technical question - once I've communicated the information, how should I proceed to do the actual change ? An email to openstack-dev with [infra] tag as an official request? | 13:27 |
| *** _nadya_ has quit IRC | 13:29 | |
| fungi | EmilienM: a patch to openstack-infra/project-config and an entry at https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames with the change url | 13:29 |
| *** yfried|afk is now known as yfried__ | 13:29 | |
| EmilienM | fungi: awesome. Thx | 13:29 |
| *** agentle_ has joined #openstack-infra | 13:29 | |
| *** annegentle has joined #openstack-infra | 13:29 | |
| *** stevemar has joined #openstack-infra | 13:29 | |
| fungi | EmilienM: then we discuss project rename scheduling since it involves a nontrivial gerrit outage to make changes in its database and filesystem and rebuild search indexes | 13:29 |
| fungi | EmilienM: also we're postponing further rename maintenances until after the kilo release, just as a heads up | 13:30 |
| EmilienM | fungi: it makes sense | 13:30 |
| jeblair | fungi: this look good as a starting point? https://etherpad.openstack.org/p/infra-liberty-summit-planning | 13:30 |
| fungi | jeblair: yep! that should do great | 13:31 |
| fungi | thanks! | 13:31 |
| *** agentle_ has quit IRC | 13:32 | |
| *** annegentle has quit IRC | 13:32 | |
| *** jpeeler has quit IRC | 13:32 | |
| *** annegentle has joined #openstack-infra | 13:32 | |
| *** ayoung has joined #openstack-infra | 13:33 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: add support for stable/kilo https://review.openstack.org/175391 | 13:33 |
| *** andreykurilin__ has joined #openstack-infra | 13:33 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: add support for stable/kilo https://review.openstack.org/175391 | 13:34 |
| sdague | jeblair: is it intentional that you can't have any other text on the Depends-On line? | 13:35 |
| jeblair | sdague: yes -- i think part of the idea was to try to be compatible with the syntax in the previous gerrit work | 13:37 |
| *** stevemar has quit IRC | 13:37 | |
| jeblair | sdague: but i mean if there's a compelling need to extend, i think we can talk about it | 13:38 |
| sdague | jeblair: ok, that's fine. I mistakenly thought I could put a comment after it | 13:38 |
| sdague | with multiple Depends-On, it's nice to list what they do | 13:38 |
| sdague | https://review.openstack.org/#/c/175391/ | 13:38 |
| *** wenlock has joined #openstack-infra | 13:38 | |
| *** BharatK has joined #openstack-infra | 13:39 | |
| sdague | I think the current behavior is fine, I just wanted to confirm it was intentional | 13:39 |
| *** palar has quit IRC | 13:39 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks https://review.openstack.org/134052 | 13:41 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks https://review.openstack.org/134052 | 13:42 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-dev/hacking: Add support for flake8 off_by_default for optional checks https://review.openstack.org/134052 | 13:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Decrease PO files for manual imports https://review.openstack.org/176313 | 13:44 |
| *** jpeeler has joined #openstack-infra | 13:44 | |
| *** jpeeler has quit IRC | 13:44 | |
| *** jpeeler has joined #openstack-infra | 13:44 | |
| *** ociuhandu has quit IRC | 13:44 | |
| *** sarob has joined #openstack-infra | 13:45 | |
| *** xyang1 has joined #openstack-infra | 13:45 | |
| *** ociuhandu has joined #openstack-infra | 13:46 | |
| *** e0ne is now known as e0ne_ | 13:46 | |
| *** mwagner_lap has joined #openstack-infra | 13:48 | |
| *** mgedmin_ has joined #openstack-infra | 13:48 | |
| *** mgedmin_ has quit IRC | 13:48 | |
| *** radez is now known as radez_g0n3 | 13:49 | |
| *** sarob has quit IRC | 13:49 | |
| *** rlandy has quit IRC | 13:49 | |
| *** amotoki has joined #openstack-infra | 13:49 | |
| *** rlandy has joined #openstack-infra | 13:50 | |
| *** BobH has joined #openstack-infra | 13:55 | |
| *** tonytan4ever has joined #openstack-infra | 13:55 | |
| BobH | Good morning all. I'm back to trying to get my 3rd party CI working after a bit of delay. | 13:56 |
| *** e0ne_ has quit IRC | 13:56 | |
| BobH | First issue following the instructions is that hiera-puppet is apparently already installed with puppet. | 13:57 |
| *** annegentle has quit IRC | 13:57 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 13:57 | |
| BobH | however, i do not see a hiera.yaml in /etc/puppet. Can I just simply copy the one in /etc? | 13:57 |
| *** ddieterly has quit IRC | 13:58 | |
| *** jcoufal has quit IRC | 13:59 | |
| *** e0ne has joined #openstack-infra | 13:59 | |
| *** annegentle has joined #openstack-infra | 13:59 | |
| *** apmelton_away is now known as apmelton | 13:59 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Decrease PO files for Doc team imports https://review.openstack.org/176313 | 14:00 |
| *** dhritishikhar has joined #openstack-infra | 14:01 | |
| *** dboik has joined #openstack-infra | 14:01 | |
| *** otter768 has joined #openstack-infra | 14:01 | |
| *** tqtran has joined #openstack-infra | 14:02 | |
| *** ala_ has quit IRC | 14:03 | |
| BobBall | Any ideas why my zuul runs jobs against stable branches? http://paste.openstack.org/show/205103/ I thought the "branch: ^master.*$" restriction in the jobs stansa should ensure it only runs against master? | 14:04 |
| *** ddieterly has joined #openstack-infra | 14:04 | |
| *** zhiwei has joined #openstack-infra | 14:05 | |
| fungi | BobH: what are "the instructions" you're following? | 14:05 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:05 | |
| *** dhritishikhar has quit IRC | 14:05 | |
| fungi | BobBall: is that zuul running dsvm-tempest.* jobs against stable branches? do you have an example change it reported on? | 14:06 |
| *** dhritishikhar has joined #openstack-infra | 14:06 | |
| *** otter768 has quit IRC | 14:06 | |
| BobBall | https://review.openstack.org/#/q/reviewer:openstack-ci%2540xenproject.org+status:open,n,z so https://review.openstack.org/#/c/173125/ | 14:06 |
| *** btully has joined #openstack-infra | 14:06 | |
| BobBall | Oh | 14:06 |
| BobBall | It's not running that job on it, no, but it's commenting | 14:06 |
| BobBall | which I don't understand. | 14:07 |
| BobH | http://ci.openstack.org/puppet.html#id2 | 14:07 |
| *** smarcet has joined #openstack-infra | 14:07 | |
| fungi | BobBall: you told it not to run those jobs, so it didn't. sounds like you need to tell it to run in a pipeline which onlt triggers on changes to the branch(es) you want instead | 14:07 |
| *** stevemar has joined #openstack-infra | 14:07 | |
| BobBall | OK - so it's the pipeline definition I need to look at; thanks. | 14:07 |
| smarcet | fungi: hi, just want to continue discussion about openstackid resource server | 14:08 |
| fungi | the pipeline you're using is still triggering on all changes, but then if they're stable branch changes it has nothing to run for them, so successfully reports that it did so | 14:08 |
| fungi | smarcet: you bet. just a sec and i'll finally have my coffee | 14:08 |
| smarcet | fungi: np :) | 14:08 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Set root disk == node disk for puppet jobs https://review.openstack.org/172210 | 14:09 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Add a ceph job https://review.openstack.org/176292 | 14:09 |
| *** erikmwilson is now known as Guest95658 | 14:11 | |
| *** erikmwil_ has joined #openstack-infra | 14:11 | |
| *** Guest95658 has quit IRC | 14:11 | |
| *** erikmwil_ is now known as erikmwilson | 14:11 | |
| *** erikmwilson_ has joined #openstack-infra | 14:11 | |
| fungi | ahh, now i feel human | 14:12 |
| fungi | smarcet: so you're writing a laravel-based resource api service from the ground up, with a similar design to openstackid but new codebase | 14:12 |
| *** sks has quit IRC | 14:12 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Puppet: move from Stackforge to OpenStack https://review.openstack.org/176326 | 14:13 |
| fungi | smarcet: and we're going to want to deploy that on a new server (or pair of servers, one for dev like we have for openstackid) | 14:13 |
| smarcet | fungi: rite, we will reusing existing code | 14:13 |
| smarcet | but on a diferent code base | 14:13 |
| smarcet | fungi : so i think that on first place, we would need a new repo here https://github.com/openstack-infra | 14:14 |
| fungi | do you want to start with an empty project and then commit it all through code review from the start, or do you already have a separate repo you've started committing into that we'll need to import? | 14:14 |
| fungi | also, most importantly, does this repo already have a name ;) | 14:14 |
| *** zz_dimtruck is now known as dimtruck | 14:15 | |
| smarcet | fungi: i would rather prefer start with an empty project, and do all commit through code review workflow, btw, this workflow is still tied to ttps://blueprints.launchpad.net/openstack-ci/? | 14:15 |
| smarcet | fungi: yeah we need a name | 14:15 |
| *** dhritishikhar has quit IRC | 14:16 | |
| *** hdd has quit IRC | 14:16 | |
| smarcet | fungi: what about openstackid-resource-server? | 14:16 |
| *** ssam2_ has joined #openstack-infra | 14:16 | |
| *** ssam2 has quit IRC | 14:16 | |
| *** nelsnelson has joined #openstack-infra | 14:16 | |
| fungi | smarcet: creating the project just involves a very trivial change following the instructions at http://docs.openstack.org/infra/manual/creators.html | 14:17 |
| *** dhritishikhar has joined #openstack-infra | 14:17 | |
| fungi | smarcet: as for the name, that's relatively long, not necessarily a problem but we should see if we can shorten it | 14:17 |
| *** salv-orlando has quit IRC | 14:18 | |
| fungi | is it completely tied to openstackid or is it something that could also be used independent of openstackid in some situations? | 14:18 |
| smarcet | fungi: re: name len, ok np, re: git repo i will take look at that link | 14:18 |
| *** tonytan4ever has quit IRC | 14:18 | |
| * fungi just realized that step 1 in that guide is "pick a good name" | 14:18 | |
| smarcet | fungi: is tied to openstackid, bc its need to query openstackid authorization server to check the token | 14:18 |
| *** ddieterly has quit IRC | 14:19 | |
| fungi | but for example, other software might also query the openstackid authorization server to check tokens as well. wondering if something else might make a more descriptive name | 14:19 |
| *** ChuckC has quit IRC | 14:19 | |
| fungi | this is the server where the marketplace api and other similar resources are going to be hosted, right? | 14:20 |
| smarcet | fungi: this guide states that https://launchpad.net is used for bugtracking | 14:20 |
| fungi | er, service | 14:20 |
| smarcet | fungi: rite | 14:20 |
| smarcet | fungi: so what is the role of storyboard.openstack.org? | 14:21 |
| fungi | smarcet: right, that guide is not 100% applicable to infra projects. it's written more from the standpoint of adding new repos to openstack overall | 14:21 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Drop postgres jobs and add multinode to tempest check https://review.openstack.org/176328 | 14:21 |
| mtreinish | sdague: ^^^ | 14:21 |
| mtreinish | jogo: you'll like that too | 14:21 |
| *** mtanino has joined #openstack-infra | 14:21 | |
| *** isviridov_away is now known as isviridov | 14:22 | |
| smarcet | fungi: i am asking this, bc when i want to commit to gerrit , i need to refer the commit to a blue print id | 14:22 |
| fungi | smarcet: there's an optional line we can add to openstack-infra/project-config:gerrit/projects.yaml to indicate that storyboard should set itself up to host issue tracking | 14:22 |
| smarcet | fungi: but i tried to do the same with storyboard and didnt work at all | 14:22 |
| fungi | the infrastructure team hasn't really been using blueprints. what specifically is the goal there? | 14:23 |
| smarcet | fungi: so not sure which app to use ( launchpad or storyboard) | 14:23 |
| smarcet | fungi: well, i think that was a requirement given by reed | 14:23 |
| *** smccully has quit IRC | 14:24 | |
| fungi | we've generally used specifications like you see at http://specs.openstack.org/openstack-infra/infra-specs/ for coordinating extremely complex activities, otherwise either linking to bug reports or just writing a good descriptive commit message with each change seems sufficient. but let's get up with reed and find out if that was a requirement for a reason | 14:24 |
| *** smccully has joined #openstack-infra | 14:24 | |
| fungi | i expect he'll wander by any moment | 14:25 |
| fungi | anyway, putting the blueprint discussion aside for the moment, how about a name like foundation-resources? | 14:25 |
| fungi | or openstackid-resources if it's really tightly coupled to openstackid itself | 14:26 |
| smarcet | fungi: ok, meanwhile could i create thie project on launchpad? | 14:26 |
| *** stevemar has quit IRC | 14:26 | |
| BobH | fungi: here: http://ci.openstack.org/puppet.html#id2 | 14:27 |
| mtreinish | sdague: :( way to rain on my parade | 14:27 |
| smarcet | fungi: re:name, i like openstackid-resources | 14:27 |
| fungi | smarcet: there's no need to do the launchpad part of it for now. that can always be done later if it ends up being necessary | 14:27 |
| smarcet | fine with me | 14:27 |
| smarcet | fungi : roger | 14:27 |
| fungi | smarcet: okay, i'll whip up a quick project-config change for this and you can see how that maps to the process in that guide | 14:28 |
| mtreinish | sdague: although with the schema input validation won't most of the negative stuff be caught earlier now? | 14:28 |
| smarcet | fungi: ok tnx :) | 14:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Decrease PO files for Doc team imports https://review.openstack.org/176313 | 14:29 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul: Add support for layout configuration split https://review.openstack.org/152290 | 14:29 |
| *** wuhg has quit IRC | 14:29 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update unregister_machine to use tasks https://review.openstack.org/171325 | 14:30 |
| smarcet | fungi: btw, i was planning to use latest laravel version ( 5.x), but its needs php >= 5.4 | 14:30 |
| NobodyCam | morning INFRA folk, off the wall question: if someone was setting up third party testing, where would the best place to post the logs be? | 14:30 |
| *** radez_g0n3 is now known as radez | 14:31 | |
| fungi | smarcet: we can deploy it on ubuntu trusty and that should have new enough php to meet that requirement i think | 14:31 |
| smarcet | fungi: does infra server supports that? i am asking bc we cant upgrade openstackid to use latest laravel so far | 14:31 |
| fungi | smarcet: yes, at the time when the openstackid server was initially deployed ubuntu trusty hadn't been released yet. we should still look into an upgrade maintenance to be able to move it onto a newer server soon | 14:32 |
| smarcet | fungi: roger that tnx | 14:32 |
| fungi | smarcet: looks like php 5.5.9 so should work? | 14:33 |
| fungi | http://packages.ubuntu.com/search?keywords=php5 | 14:33 |
| smarcet | fungi: yup should do it | 14:33 |
| *** isviridov is now known as isviridov_away | 14:34 | |
| *** dhritishikhar has quit IRC | 14:35 | |
| *** dizquierdo has joined #openstack-infra | 14:35 | |
| *** jerryz has quit IRC | 14:36 | |
| *** unicell1 has joined #openstack-infra | 14:36 | |
| *** unicell has quit IRC | 14:37 | |
| *** yfried__ is now known as yfried|afk | 14:38 | |
| *** dangers_away is now known as dangers | 14:38 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add the openstackid-resources repo https://review.openstack.org/176334 | 14:39 |
| fungi | smarcet: that ^ is what will create the new project in our systems | 14:39 |
| smarcet | fungi: ok tnx | 14:40 |
| fungi | i'm writing a governance change now (order of change dependencies is reversed based on a tc resolution yesterday, and the guide is about to get updated to reflect that with https://review.openstack.org/173485 ) | 14:40 |
| jeblair | mordred: why was https://review.openstack.org/#/c/165815/ abandoned? | 14:40 |
| *** asselin_ has joined #openstack-infra | 14:42 | |
| fungi | smarcet: the corresponding governance bookkeeping addition is https://review.openstack.org/176338 | 14:43 |
| *** sdake has joined #openstack-infra | 14:43 | |
| jeblair | mordred: is it because master is shaping up for release? | 14:43 |
| jeblair | and the 0.10 branch is being abandoned? | 14:43 |
| *** yamahata has quit IRC | 14:43 | |
| *** sdake_ has joined #openstack-infra | 14:44 | |
| *** yamahata has joined #openstack-infra | 14:44 | |
| fungi | jeblair: do you mind if i update your 173485 infra manual change based on AJaeger's feedback before i +2 it? | 14:44 |
| *** yfried|afk is now known as yfried__ | 14:44 | |
| smarcet | fungi: who is the one in charge to approve those changes? reed? | 14:44 |
| *** tsg has quit IRC | 14:45 | |
| *** tqtran has quit IRC | 14:45 | |
| openstackgerrit | Jerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching https://review.openstack.org/156178 | 14:46 |
| AJaeger | jeblair, btw. could you review this infra-manual and approve it, please? https://review.openstack.org/#/c/138595/ It has two +2s but some wanted to give you a chance to review. | 14:47 |
| *** MaxV has quit IRC | 14:47 | |
| *** sdake has quit IRC | 14:48 | |
| *** MaxV has joined #openstack-infra | 14:48 | |
| fungi | smarcet: these people... https://review.openstack.org/#/admin/groups/459,members and https://review.openstack.org/#/admin/groups/306,members | 14:48 |
| smarcet | fungi: ok tnx | 14:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib) https://review.openstack.org/176343 | 14:48 |
| *** jerryz has joined #openstack-infra | 14:48 | |
| fungi | smarcet: in irc that's AJaeger, anteaya, sdague, clarkb, pleia2, jeblair, me, jhesketh, mordred and SergeyLukjanov | 14:49 |
| *** tonytan4ever has joined #openstack-infra | 14:49 | |
| smarcet | fungi: cool tnx for the info :) | 14:49 |
| fungi | as long as a couple of them look it over and are okay with it, that's all we need | 14:49 |
| *** dkranz has quit IRC | 14:49 | |
| jerryz | fungi: when i rebased on the change to switch to pymysql you added recently, unit test would fail due to mysql connection denied. | 14:50 |
| jerryz | fungi: pymysql somehow used my fqdn to connect to mysql server instead of localhost | 14:50 |
| fungi | jerryz: which repo? nodepool? | 14:51 |
| jerryz | fungi: yes | 14:51 |
| fungi | jerryz: i think the main behavior difference is that pymysql only supports connection via tcp socket, not unix socket (which was mysql-python's default behavior) | 14:51 |
| fungi | jerryz: in our case it seems to connect to the loopback interface. let me double-check our config | 14:52 |
| *** mrunge has quit IRC | 14:52 | |
| clarkb | jerryz could it be a permissions issue? mysql treats the connection endpoints independently for access | 14:53 |
| fungi | jerryz: right, so the dburi in our nodepool.yaml looks like mysql+pymysql://username:password@localhost/dbname | 14:53 |
| fungi | "localhost" being the key part there | 14:53 |
| clarkb | so you can have permission to connect via unix socket but not localhost tcp | 14:53 |
| AJaeger | is zuul/gerrit stuck? https://review.openstack.org/#/c/175324/ passed the check queue 35 mins ago and is not yet in the gate queue | 14:53 |
| fungi | AJaeger: the changes it depends on aren't in the gate | 14:54 |
| fungi | its parent commit has a -1 from jenkins still | 14:54 |
| AJaeger | fungi, ARgh - somebody rebased my patch... | 14:54 |
| *** davideagnello has joined #openstack-infra | 14:54 | |
| AJaeger | Thanks, that explains it... | 14:54 |
| *** tonytan4ever has quit IRC | 14:55 | |
| jerryz | fungi, clarkb: this is the error i am seeing http://paste.openstack.org/show/205117/ | 14:55 |
| fungi | jerryz: checking in our nodepool database, our users are just identified by "localhost" in the host column and their username in the user column (of the mysql.user table) | 14:56 |
| clarkb | NobodyCam: someplace accessible by the users of our gerrit server | 14:56 |
| jerryz | fungi: i did that as well | 14:56 |
| clarkb | NobodyCam: we are currently moving to storing logs in swift with a wsgi app in front to do the serving (so we can modify logs on the fly and so we don't have to deal with crazy cdn stuff at least not yet) | 14:57 |
| jerryz | fungi: but unit test still use fqdn to connect the database | 14:57 |
| clarkb | jerryz: yup that is a permissions issue | 14:57 |
| *** yfried__ is now known as yfried|afk | 14:57 | |
| fungi | jerryz: oh! unit tests specifically, not your production server | 14:57 |
| NobodyCam | clarkb: ah so no real formal location | 14:57 |
| fungi | jerryz: what patch are you seeing this from? | 14:58 |
| clarkb | its saing this is the user connecting to this location and trying to access this db and that isn't allowed | 14:58 |
| jerryz | fungi: the patch can pass on infra jenkins | 14:58 |
| clarkb | NobodyCam: no we don't provide hosting for you nor do we enforce a specific type of file hosting | 14:58 |
| *** hdd has joined #openstack-infra | 14:58 | |
| fungi | jerryz: got it, so it's a database setup issue on the worker where that job runs | 14:58 |
| jerryz | fungi: but i can't make tox -epy27 to pass locally | 14:58 |
| NobodyCam | clarkb: :) awesome TY | 14:58 |
| *** hdd has quit IRC | 14:59 | |
| fungi | jerryz: so you probably need your local mysql permissions set similar to how we do it on our job workers | 14:59 |
| *** signed8bit has joined #openstack-infra | 14:59 | |
| jerryz | fungi: i have granted the privileges to openstack_citest@localhost | 15:00 |
| *** russellb has quit IRC | 15:00 | |
| fungi | jerryz: a summary of the commands needed can be found at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n454 | 15:01 |
| fungi | jerryz: so probably needs to be openstack_citest@% instead | 15:01 |
| jerryz | fungi: yes that is the key | 15:02 |
| jerryz | '$DB_USER'@'%' | 15:02 |
| jerryz | fungi: readme need to be updated | 15:02 |
| fungi | jerryz: awesome--thanks for finding that | 15:03 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 15:03 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 15:03 | |
| fungi | jerryz: one of the things i want to get to next once bindep has been beaten into shape is some way to provide easy to consume database config scripts for projects that need some sort of setup like this | 15:04 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 15:04 | |
| fungi | something they can carry or easily access from their individual git repositories and be able to document consistently | 15:04 |
| *** _nadya_ has joined #openstack-infra | 15:04 | |
| *** dkranz has joined #openstack-infra | 15:04 | |
| *** russellb has joined #openstack-infra | 15:05 | |
| fungi | but those (mysql|pgsql)-(db|prep) builder macros will suffice initially | 15:05 |
| *** annegentle has quit IRC | 15:06 | |
| *** agentle_ has joined #openstack-infra | 15:06 | |
| *** annegentle has joined #openstack-infra | 15:06 | |
| jerryz | fungi: cool. | 15:07 |
| jerryz | fungi: yes but people need some knowledge of jjb and where to find the template. | 15:08 |
| fungi | ultimately i think we need some sort of structured framework we can standardize on where a particular repo can say "to run these tests, first do this as root" | 15:09 |
| fungi | right, keeping that in jjb configuration is not the end goal | 15:09 |
| fungi | it's just a stepping stone | 15:09 |
| fungi | right now we're still actually doing that prep in our nodepool images when they're created | 15:10 |
| fungi | so step 1 is to do it at the start of each job instead | 15:10 |
| gordc | ttx: do you have any concerns with making a ceilometerclient 1.0.14 release for kilo (with no requirements changes). we have a user who is using kilo in an upcoming product release | 15:10 |
| *** yfried|afk is now known as yfried__ | 15:10 | |
| fungi | jerryz: then step 2 is to make it so it can be overridden in each repo | 15:10 |
| *** jamesmcarthur has joined #openstack-infra | 15:10 | |
| ttx | gordc: yes we have a concern, would like to delay that a few more hours | 15:11 |
| gordc | ttx: just for reference, 1.0.14 would have same changes as 1.1.0 (minus all the requirements changes) | 15:11 |
| ttx | gordc: we just unbroke the world and would like it to stay calm for a bit. But yeah worst case, we'll do them tomorrow | 15:11 |
| gordc | ttx: sure thing. understandable given other gate issues... i'll hold off until you guys have it figured out | 15:11 |
| gordc | ttx: sounds good. thanks! | 15:12 |
| russellb | hrm, governance.openstack.org seems to be down | 15:12 |
| fungi | checking | 15:12 |
| fungi | nice. that's our static.openstack.org server, so all sorts of things are down | 15:13 |
| clarkb | :/ | 15:13 |
| russellb | :\ | 15:13 |
| ttx | that cloud thing sure is flaky | 15:13 |
| russellb | ttx: +1 | 15:13 |
| *** agentle_ has quit IRC | 15:13 | |
| *** annegentle has quit IRC | 15:13 | |
| fungi | apache seems to be running, not restarted recently | 15:13 |
| fungi | nothing recent in dmesg | 15:14 |
| fungi | now it's finally returned content to me | 15:14 |
| clarkb | load is reasonable too so its not like it just stopped serving requests because there were too many | 15:14 |
| fungi | updatedb.mlocate running there since 06:38 utc | 15:14 |
| fungi | might have been brief i/o starvation | 15:15 |
| fungi | that will eventually get better when the last of our logs expire from there | 15:15 |
| russellb | ah, there it goes, yeah loaded eventually | 15:15 |
| clarkb | speaking of anyone want to review the topic:enable_swift changes? | 15:15 |
| fungi | want to? yes | 15:15 |
| fungi | here we go! http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 15:16 |
| *** sabeen has joined #openstack-infra | 15:16 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 15:16 |
| clarkb | ya | 15:16 |
| fungi | someone decided to download a ton of comething | 15:16 |
| jogo | mtreinish: already removed postgres from neutron | 15:16 |
| fungi | something | 15:16 |
| *** dboik_ has joined #openstack-infra | 15:16 | |
| clarkb | fungi: its image builds | 15:16 |
| *** fawadkhaliq has joined #openstack-infra | 15:16 | |
| clarkb | fungi: grabbing images to cache for devstack | 15:16 |
| clarkb | we should put those in swift | 15:17 |
| *** dboik_ has quit IRC | 15:17 | |
| fungi | oh... the ironic agent images? | 15:17 |
| *** dboik_ has joined #openstack-infra | 15:17 | |
| fungi | yep, timing is right for that | 15:17 |
| clarkb | and trove image | 15:17 |
| fungi | doesn't help that mlocate is reindexing the world at the same time | 15:18 |
| fungi | so i/o load and network load | 15:18 |
| *** andreykurilin__ has quit IRC | 15:18 | |
| openstackgerrit | Victor Sergeyev proposed openstack-infra/project-config: Make oslo.db py3 job voting https://review.openstack.org/176356 | 15:18 |
| fungi | i have to say, our log volume is looking emptier and emptier. we trimmed it back a lot around the end of last year http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 15:19 |
| *** dboik has quit IRC | 15:20 | |
| russellb | fungi: clarkb thanks for looking, and sorry to distract :) | 15:20 |
| mtreinish | jogo: well only partially right? there were 2 sets of those jobs iirc | 15:20 |
| clarkb | russellb: no problem at all, I would much prefer to know that we should use swift more than irritate our users :) | 15:20 |
| *** rlandy has quit IRC | 15:21 | |
| *** tqtran has joined #openstack-infra | 15:22 | |
| *** rlandy has joined #openstack-infra | 15:23 | |
| jogo | mtreinish: correct, I235f6fe1c0f37c8f3acaf92562d2d6320a86483c | 15:23 |
| fungi | clarkb: so i guess this ratchets up the priority a bit for getting tarballs.o.o into swift, and in such a way that downloads happen via http redirect or something rather than getting proxied through static.o.o | 15:25 |
| *** arxcruz has quit IRC | 15:25 | |
| fungi | unlike logs where we want to mangle them on the way through | 15:25 |
| fungi | for tarballs there is zero benefit to proxying | 15:25 |
| *** yolanda has quit IRC | 15:25 | |
| *** tqtran has quit IRC | 15:26 | |
| fungi | (tarballs, images, other binary blob artifacts) | 15:26 |
| *** armax has joined #openstack-infra | 15:26 | |
| fungi | ties in pretty nicely to http://specs.openstack.org/openstack-infra/infra-specs/specs/nodepool-workers.html too | 15:27 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Search methods for networks, subnets and routers https://review.openstack.org/175027 | 15:27 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Update volume API for new getters and dict retval https://review.openstack.org/176359 | 15:27 |
| fungi | er, well even without that i guess, we could in theory start pushing dib nodepool images up to swift independent of having separate dib workers | 15:29 |
| fungi | (aside from where we have to for glance v2 api i mean) | 15:29 |
| *** sabeen3 has joined #openstack-infra | 15:29 | |
| clarkb | fungi: I think step one is getting them into swift, it will remove the disk IO and make it purely network which I think is cheaper for us | 15:30 |
| clarkb | fungi: then we can worry about the CDNing | 15:30 |
| fungi | but be able to serve them alongside tarballs and other non-log artifacts | 15:30 |
| *** sabeen has quit IRC | 15:30 | |
| fungi | yep, agreed | 15:30 |
| *** dizquierdo has quit IRC | 15:30 | |
| *** vlaza has quit IRC | 15:31 | |
| clarkb | the other thing that helps this specific case is using dib to build those images since dib will cache them and only download updates when necessary | 15:32 |
| *** claudiub has quit IRC | 15:32 | |
| clarkb | so all the things are tied together in a web of fixit :) | 15:32 |
| *** zhiwei has left #openstack-infra | 15:33 | |
| hogepodge | Is is just me, or is anyone else experiencing connection problems with etherpads? | 15:33 |
| *** annegentle has joined #openstack-infra | 15:33 | |
| *** marun has joined #openstack-infra | 15:33 | |
| clarkb | hogepodge: anteaya had trouble but no one else was able to reproduce | 15:33 |
| hogepodge | I'm having to force reconnect and am losing edits every few minutes. | 15:34 |
| clarkb | ya that was what anteaya saw, what browser and version are you using? | 15:34 |
| *** dkranz has quit IRC | 15:34 | |
| *** dhritishikhar has joined #openstack-infra | 15:34 | |
| hogepodge | Safari 8.0.5 | 15:34 |
| *** rwsu has joined #openstack-infra | 15:34 | |
| fungi | at the moment i'm having trouble getting an etherpad to load | 15:34 |
| *** ihrachyshka has quit IRC | 15:35 | |
| fungi | so not similar to what anteaya described | 15:35 |
| hogepodge | I've been seeing slow loads also, but I thought maybe it was a caching issue from the update. | 15:35 |
| clarkb | https://etherpad.openstack.org/p/repo-renaming-2015-04-17 loads for me | 15:35 |
| *** e0ne has quit IRC | 15:36 | |
| fungi | it took a couple minutes for https://etherpad.openstack.org/p/infra-liberty-summit-planning to finish loading for me, for some reason | 15:36 |
| hogepodge | It's not a matter of not loading, it's just taking a long time on older docs and when I'm editing the connection times out and I lose the edits | 15:36 |
| hogepodge | I've taken to editing outside of the etherpad and pasting in to make sure I don't lose anything. | 15:36 |
| clarkb | fungi: that one loads straight up for me too | 15:37 |
| *** e0ne has joined #openstack-infra | 15:37 | |
| fungi | hogepodge: yeah, that does sound similar to what anteaya ran into | 15:37 |
| clarkb | hogepodge: what etherpad are you editing? | 15:37 |
| hogepodge | clarkb https://etherpad.openstack.org/p/DefCoreScale.13 | 15:37 |
| clarkb | I can confirm the slowconnection disconnect in the logs | 15:37 |
| * clarkb goes to look | 15:37 | |
| *** asselin_ has quit IRC | 15:38 | |
| fungi | there's a chance it was a browser extension. she disabled all her extensions and restarted the browser and it went away, then she added back her extensions one by one and it kept working for a while, but eventually she got another disconnect | 15:38 |
| clarkb | DIAGNOSTIC-INFO: {"disconnectedMessage":"reconnect_timeout","padId":"DefCoreScale.13","socket":{}} | 15:38 |
| clarkb | that is a different error | 15:38 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support to CFP in builders sections https://review.openstack.org/164504 | 15:38 |
| hogepodge | for this morning, and last night this one https://etherpad.openstack.org/p/havana-capabilities-test-procedure | 15:38 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Use new getters in update_subnet and update_router https://review.openstack.org/175837 | 15:38 |
| *** dhritishikhar has quit IRC | 15:38 | |
| clarkb | https://github.com/ether/etherpad-lite/issues/2523 could you have it open in multiple tabs? | 15:39 |
| hogepodge | fungi: I'll take a look into that. I'm only running an ad blocker. | 15:39 |
| *** yfried__ is now known as yfried|afk | 15:39 | |
| *** annegentle has quit IRC | 15:39 | |
| clarkb | hogepodge: I do see two clients against that particular pad, hard to figure out if the have the same id becaus apahce is proxying | 15:40 |
| hogepodge | clarkb: no, I have multiple eterpads open, but to different documents | 15:40 |
| *** annegentle has joined #openstack-infra | 15:40 | |
| fungi | depending on how the ad blocker is implemented, if it disconnects or blocks an http request for an ajax api call then it might cause this sort of symptom | 15:40 |
| *** smoser has left #openstack-infra | 15:40 | |
| *** Sukhdev has joined #openstack-infra | 15:40 | |
| hogepodge | fungi: I disabled it for all etherpad sites, so we'll see what happens. | 15:40 |
| clarkb | hogepodge: who is the dark green unknown user on that etherpad? | 15:40 |
| *** yolanda has joined #openstack-infra | 15:40 | |
| anteaya | hogepodge: are you on ubuntu? | 15:41 |
| fungi | probably not unless someone ported safari to linux | 15:41 |
| anteaya | hogepodge: ubuntu has a default of 4 addons | 15:41 |
| anteaya | ah sorry I haven't read all the backscroll | 15:41 |
| hogepodge | clarkb heh, it could be me. It looks like I have a couple of users at the same time. I'll restart my browswer. | 15:41 |
| *** dhritishikhar has joined #openstack-infra | 15:41 | |
| clarkb | hogepodge: I bet that is your problem | 15:42 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Initial work for supporting downstream-ext https://review.openstack.org/158117 | 15:42 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add FindBugs support to Publishers https://review.openstack.org/166028 | 15:42 |
| clarkb | its annoying that the error message isn't better from etherpad-lite though | 15:42 |
| clarkb | anteaya: you might also want to double check you aren't double connected to the etherpad(s) you had trouble with | 15:43 |
| anteaya | hogepodge: I only had disconnects when I used firefox, I never disconnected when I used chromium | 15:43 |
| *** yamahata has quit IRC | 15:43 | |
| anteaya | I was double connected with chromium and had no disconnects | 15:43 |
| anteaya | I had problems with firefox when I was single connected | 15:43 |
| hogepodge | clarkb: for some reason it's still bringing me in as two users. Cleared my cache and cookies. | 15:43 |
| clarkb | hogepodge: probably a second tab | 15:43 |
| clarkb | you would need to close it | 15:44 |
| hogepodge | nope, no tab open | 15:44 |
| *** gyee has joined #openstack-infra | 15:44 | |
| mtreinish | hogepodge: looking at that etherpad, you know you need to add a include regex list file option to os-testr then you can just use the the idempotent ids to do the selection and not worry about moves | 15:44 |
| mtreinish | instead of having to worry about passing the whole test_id everytime | 15:44 |
| hogepodge | mtreinish: if I use just the idempotent id won't I oversubscribe to the tests though? it's not a 1:1 mapping | 15:46 |
| jerryz | fungi: https://github.com/openstack-infra/nodepool/blob/master/nodepool/tests/__init__.py#L164 this is still causing issue for me. | 15:47 |
| *** dhritishikhar has quit IRC | 15:47 | |
| mtreinish | hogepodge: there is to a test method, which means given a config it'll completely test that feature | 15:47 |
| *** dkranz has joined #openstack-infra | 15:47 | |
| *** dhritishikhar has joined #openstack-infra | 15:47 | |
| *** weshay has quit IRC | 15:47 | |
| mtreinish | hogepodge: like if you tell tempest your cloud supports ipv4 and ipv6 would you want defcore validation to pass if one failed? | 15:47 |
| mtreinish | for a particular test | 15:47 |
| *** sdake_ has quit IRC | 15:48 | |
| hogepodge | mtreinish: but we may not require all of the tests associated with the test method, so there could be false positive/negatives. The intent of that list is to only run the tests that are required and nothing more | 15:48 |
| *** yfried|afk is now known as yfried__ | 15:48 | |
| zaro | morning | 15:49 |
| *** spredzy is now known as spredzy|afk | 15:49 | |
| fungi | jerryz: so you're getting an exception raised from the grant call, but not from the create call before it? | 15:49 |
| mtreinish | hogepodge: give me an example, because the only cases I can think of where subclassing is used are things things you shouldn't differentiate on in defcore (like ip version) | 15:49 |
| mtreinish | of if you did it should be a strong opinion, which you should force in the config file (like service api version) | 15:50 |
| mtreinish | err, service api major version (ie cinder v1 or v2) | 15:50 |
| jerryz | fungi: no, grant privilege to localhost won't suffice. because pymysql attempts to connect to fqdn instead of localhost each time. | 15:50 |
| mtreinish | hogepodge: and in either case I don't think a failure would be a false result | 15:51 |
| fungi | jerryz: i wonder if the difference is our /etc/hosts file | 15:51 |
| jerryz | fungi: probably | 15:51 |
| hogepodge | mtreinish: I'll speak with catherine about it. She's the expert on it, and initially raised the flag after analyzing the collisions. | 15:52 |
| hogepodge | mtreinish: it may not be an issue right now, though. Let me look and see if it is. I'll compare output from using the full names to the output from just using the ids | 15:53 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add support for Sonatype CLM plugin https://review.openstack.org/164446 | 15:53 |
| fungi | jerryz: here's an example of one of ours http://paste.openstack.org/show/205123/ | 15:53 |
| hogepodge | mtreinish: One of the things I'd like to do is add a non-voting gate job to defcore so that if tempest moves change anything we'll know. | 15:53 |
| fungi | jerryz: having localhost mapped to 127.0.0.1 is one thing to double-check you've got in yours | 15:53 |
| *** smarcet has quit IRC | 15:54 | |
| *** ashleighfarnham has joined #openstack-infra | 15:55 | |
| mtreinish | hogepodge: well tbh that's what the ids being added was meant to prevent the need for | 15:55 |
| mtreinish | hogepodge: I just feel like the concern on this stuff is coming from looking at the mechanics of listing and counting instead of looking at what the tests are actually doing | 15:56 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Remove OpenStack library caps for liberty https://review.openstack.org/173924 | 15:56 |
| anteaya | asselin: when you are about can you please read this email and propose your work to the etherpad: http://lists.openstack.org/pipermail/openstack-infra/2015-April/002700.html | 15:57 |
| *** funzo has quit IRC | 15:57 | |
| *** deepakcs has joined #openstack-infra | 15:57 | |
| anteaya | asselin: that way we can get you a table as was discussed in the meeting yesterday | 15:57 |
| hogepodge | mtreinish: There's not a 1:1 mapping of test names to ids, so there's always the possibility of collision between required and non-required tests, regardless of the functionality. | 15:57 |
| *** funzo has joined #openstack-infra | 15:58 | |
| *** dboik_ has quit IRC | 15:59 | |
| hogepodge | mtreinish: you may disagree with that philosophically, but it's the reality of the process. My mistake was in not realizing the issue and pushing for the solution that sergei has proposed earlier. | 15:59 |
| mtreinish | hogepodge: well the overload only happens in a subclass, which will only happen if there is a config selector between tests. It's testing the same functionality over all the permuations you tell tempest it can run | 15:59 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/devstack-gate: add support for stable/kilo https://review.openstack.org/175391 | 15:59 |
| *** ashleighfarnham has quit IRC | 15:59 | |
| *** ashleighfarnham has joined #openstack-infra | 16:00 | |
| *** dhritishikhar has quit IRC | 16:00 | |
| mtreinish | hogepodge: I'm saying if defcore wants to say cloud X supports network feature Y. Then unless the feature Y test for all subclasses enabled via config passes you can't say that | 16:00 |
| *** dhritishikhar has joined #openstack-infra | 16:00 | |
| *** dboik has joined #openstack-infra | 16:00 | |
| mtreinish | hogepodge: because the runner of the test is saying my clouds supports ipv6 and ipv4 but the test for the feature doesn't work on both | 16:01 |
| *** flip214 has joined #openstack-infra | 16:01 | |
| *** otter768 has joined #openstack-infra | 16:02 | |
| jerryz | fungi: just checked it is similar to yours. i even went ahead to delete the line 127.0.0.1 mapping to fqdn | 16:02 |
| hogepodge | mtreinish: what do you mean? It would be reasonable to say ipv4 is required, but ipv6 is advisory. We would need to distinguish between the two | 16:02 |
| *** funzo has quit IRC | 16:02 | |
| jerryz | fungi: still got the same error | 16:03 |
| fungi | jerryz: is it the same permission error you pasted at http://paste.openstack.org/show/205117/ or a new one? | 16:04 |
| mtreinish | hogepodge: no you don't. So you say ipv4 is required that means you force everyone to set that config option in tempest. But ipv6 is float, it can either be true or false it doesn't matter the burden is on the end user | 16:04 |
| *** yfried__ is now known as yfried|afk | 16:04 | |
| mtreinish | hogepodge: but if they set it true and the ipv6 tests fail then their cloud fails for that functionality because they're saying things should support ipv6 and they don't | 16:05 |
| hogepodge | but by id only it's possible to run both tests when only one is required, and it's reasonable to say that one would pass and one would fail. | 16:05 |
| asselin | anteaya, reading | 16:05 |
| *** hashar is now known as hasharAway | 16:05 | |
| mtreinish | hogepodge: I'm saying its not reasonable, because it's all based on the config file. If you want that you just need to enforce the value of the option used to switch the tests | 16:06 |
| mtreinish | hogepodge: so if you want to say only ipv4 tell people they have to set ipv4 true and ipv6 false | 16:06 |
| *** _nadya_ has quit IRC | 16:06 | |
| jerryz | fungi: new one since i changed openstack_citest@‘%’ . http://paste.openstack.org/show/205125/ | 16:06 |
| *** teran has joined #openstack-infra | 16:07 | |
| *** dprince has quit IRC | 16:07 | |
| *** scheuran has quit IRC | 16:07 | |
| hogepodge | mtreinish: It does matter in the larger picture though, because we're going to be asking people to run many api tests (non required) to help guide future capabilities. From a management standpoint it means a lot more work to index and keep things straight | 16:07 |
| *** otter768 has quit IRC | 16:07 | |
| *** shashankhegde has joined #openstack-infra | 16:07 | |
| mtreinish | hogepodge: but if you leave it up to the user, then you can't say some functionality works when it doesn't for the configuration they say their cloud supports | 16:08 |
| zaro | clarkb, fungi : looks we will probably see this issue in our Gerrit openstack/2.10.2 branch, https://groups.google.com/d/msg/repo-discuss/Kv4bWLESbQ4/-oSNbuTQwkUJ | 16:08 |
| zaro | our branch contains sshd 0.14.0 | 16:08 |
| mtreinish | hogepodge: ok, maybe. I'm not exactly sure what you mean there. | 16:09 |
| *** yfried|afk has quit IRC | 16:09 | |
| *** teran__ has quit IRC | 16:09 | |
| hogepodge | mtreinish: They'll send passed results only, but we still have to sort the results. If we just use id we can't know if they pass ipv4 or ipv6 | 16:09 |
| *** marun has quit IRC | 16:09 | |
| *** Guest5363 has quit IRC | 16:10 | |
| zaro | clarkb, fungi : upgrade is scheduled in about two weeks so not sure what we want to do if upstream doesn't revert sshd back to a working version of SSHD | 16:10 |
| hogepodge | I'm on a call right now, so can't respond for about an hour | 16:11 |
| *** erikmwilson has quit IRC | 16:11 | |
| *** devvesa has quit IRC | 16:11 | |
| *** MaxV has quit IRC | 16:11 | |
| *** erikmwilson has joined #openstack-infra | 16:11 | |
| mtreinish | hogepodge: well I'm not sure that matters if you're saying you don't care about ipv6. But you can do that because skips are still in the test results | 16:12 |
| mtreinish | so you have all the necessary information | 16:12 |
| mtreinish | hogepodge: ok sure | 16:12 |
| *** jistr has quit IRC | 16:12 | |
| *** erikmwilson has quit IRC | 16:13 | |
| *** erikmwilson has joined #openstack-infra | 16:13 | |
| *** ddieterly has joined #openstack-infra | 16:14 | |
| *** dboik has quit IRC | 16:15 | |
| anteaya | I will further note that I can attempt to open up an etherpad in firefox and while it is trying to load, open up the same etherpad in chromium and chromium is well finished while I close the tab on firefox because it is still trying to load | 16:16 |
| *** unicell1 has quit IRC | 16:18 | |
| *** jlanoux has quit IRC | 16:20 | |
| openstackgerrit | Ryan Petrello proposed openstack-infra/project-config: Set the proper homepage for akanda repositories. https://review.openstack.org/176386 | 16:21 |
| *** radez is now known as radez_afk | 16:21 | |
| *** coolsvap|afk has quit IRC | 16:21 | |
| *** dboik has joined #openstack-infra | 16:24 | |
| *** coolsvap|afk has joined #openstack-infra | 16:24 | |
| *** dprince has joined #openstack-infra | 16:25 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser from jenkins_jobs.builder https://review.openstack.org/176389 | 16:25 |
| *** aduarte has joined #openstack-infra | 16:25 | |
| *** nfedotov has quit IRC | 16:26 | |
| *** ajmiller has joined #openstack-infra | 16:26 | |
| waynr | pelix, zaro, clarkb (and anyone else familiar with JJB) I'm hoping to move quickly on https://review.openstack.org/176389 since it is primarily a code organization change | 16:27 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/infra-specs: Specification proposal about system-config testing using containers https://review.openstack.org/172833 | 16:28 |
| *** Swami has joined #openstack-infra | 16:28 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/infra-specs: Specification proposal about system-config testing using containers https://review.openstack.org/172833 | 16:28 |
| fungi | zaro: got it. so we might want to only upgrade to 10.2 while they sort out whether there are sshd issues in 10.3? | 16:32 |
| fungi | er, 2.10.2 as opposed to 2.10.3 | 16:32 |
| sslypushenko | mtreinish: I updated accouts-generator tool according to your comments | 16:32 |
| *** ddieterly has quit IRC | 16:32 | |
| asselin | what's the difference b/w fishbowl and workroom? | 16:32 |
| jerryz | fungi: granting the privilege to '%' nodepool/tests/__init__.py works for me. however, it is still unknown why the connection was resolved to my fqdn. | 16:33 |
| jeblair | fungi, pleia2: http://packages.ubuntu.com/vivid/cgit | 16:33 |
| mordred | jeblair: zomg | 16:33 |
| mordred | fungi, jerryz: I have not read the whole scrollback - but it soudns like a case of having not deleted the anonymous user | 16:34 |
| *** dannywilson has joined #openstack-infra | 16:34 | |
| jeblair | notnownikki just told me about that | 16:34 |
| mordred | jeblair: I would like to say that I'm a fan of that | 16:35 |
| zaro | fungi: yep, what would you suggest we do with current openstack/2.10.2 branch we have setup? | 16:35 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for create_image with glance v1 https://review.openstack.org/175699 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Explicitly request cloud name in test_caching https://review.openstack.org/175698 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add tests and invalidation for glance v2 upload https://review.openstack.org/176024 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for caching in list_images https://review.openstack.org/174717 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Test flavor cache and add invalidation https://review.openstack.org/174707 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Test that deleting user invalidates user cache https://review.openstack.org/173596 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Fix major update_user issues https://review.openstack.org/174704 | 16:36 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: create_user should return the user created https://review.openstack.org/174642 | 16:36 |
| SpamapS | mordred: ^^ rebases. :-P | 16:36 |
| *** odyssey4me has left #openstack-infra | 16:36 | |
| zaro | waynr: i like that change, will try to take a look today. | 16:36 |
| mordred | SpamapS: woot! | 16:36 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Make default from address mandatory https://review.openstack.org/176393 | 16:36 |
| *** andreykurilin__ has joined #openstack-infra | 16:37 | |
| jerryz | mordred: could you elaborate? | 16:37 |
| *** luqas has quit IRC | 16:37 | |
| zaro | waynr: may want to ping jhesketh on that jjb change as well. | 16:37 |
| mtreinish | anteaya, pleia2: so about that thread from a couple weeks ago about election tooling. I put together: https://github.com/mtreinish/electionbot as the basic idea behind on how we can use the ml | 16:37 |
| *** ddieterly has joined #openstack-infra | 16:38 | |
| mordred | jerryz: mysql ships with a user account with a name of '' | 16:38 |
| mtreinish | but it's just idea code, I haven't actually tried to run it or anything | 16:38 |
| mordred | jerryz: it's a terrible idea | 16:38 |
| mtreinish | but before I put any real effort into it I figured I should share | 16:38 |
| *** dtantsur is now known as dtantsur|afk | 16:38 | |
| *** davideagnello has quit IRC | 16:38 | |
| mordred | jerryz: and what's worse about it is that the username/hostname resolution rules in mysql do not produce the result they think you would if that user is present | 16:38 |
| *** dboik has quit IRC | 16:39 | |
| mordred | so often you'll wind up creating users and host matches that arent necessary, because the anonymous user was intercepting your login attempts | 16:39 |
| mordred | jerryz: but after lunch I'll read the whole scrollback | 16:39 |
| pleia2 | mtreinish: cool, I'll have a look | 16:40 |
| zaro | fungi: i guess we can leave it alone and wait for upstream to revert back to sshd 0.9 but i'm concerned that may not happen before we do upgrade so i'm thinking of a plan B | 16:40 |
| *** mpaolino has quit IRC | 16:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/devstack-gate: add support for stable/kilo https://review.openstack.org/175391 | 16:41 |
| *** ddieterly has quit IRC | 16:42 | |
| openstackgerrit | Stephanie Miller proposed openstack-infra/system-config: Make default email mandatory https://review.openstack.org/176394 | 16:42 |
| cinerama | pleia2: hi there | 16:42 |
| pleia2 | cinerama: hey :) | 16:43 |
| cinerama | pleia2: do you have some time today to talk through client stuff with me? | 16:44 |
| pleia2 | cinerama: sure, maybe in an hour or so? I've just been working through some administrative/paperworky stuff this morning, should be done in a bit | 16:45 |
| fungi | zaro: yeah, it seems like only upgrading as far as 2.10.2 until they work out the mina-sshd issues might be fine. then if it's solved in 2.10.4 or something, that's an easy upgrade to schedule whenever | 16:45 |
| cinerama | pleia2: cool. i also have a couple of teensy patches put up | 16:45 |
| pleia2 | jeblair, fungi, clarkb, anteaya: I added nibalizer's outreachy ideas here, so we can add more as needed https://etherpad.openstack.org/p/infra-liberty-outreachy | 16:46 |
| *** Krinkle|detached is now known as Krinkle | 16:48 | |
| *** unicell has joined #openstack-infra | 16:48 | |
| *** subscope_ has joined #openstack-infra | 16:48 | |
| *** dannywilson has quit IRC | 16:48 | |
| nibalizer | woot | 16:49 |
| pleia2 | nibalizer: and thanks :) good ideas | 16:50 |
| *** e0ne has quit IRC | 16:50 | |
| waynr | jhesketh: hoping to get quick review and merge on this since it is primarily organizational: https://review.openstack.org/176389 | 16:51 |
| *** tonytan4ever has joined #openstack-infra | 16:52 | |
| fungi | ooh... now i'm getting the pad reconnects too | 16:53 |
| *** viglesias has quit IRC | 16:54 | |
| *** dhritishikhar has quit IRC | 16:54 | |
| *** mpaolino has joined #openstack-infra | 16:54 | |
| *** melwitt has joined #openstack-infra | 16:55 | |
| *** dhritishikhar has joined #openstack-infra | 16:55 | |
| *** yolanda has quit IRC | 16:55 | |
| anteaya | fungi: oh, mind if I'm happy about that? | 16:56 |
| fungi | not at all | 16:56 |
| anteaya | in that you are far better at debugging than myself | 16:57 |
| fungi | it seems to have been short-lived | 16:57 |
| anteaya | what browser? | 16:57 |
| fungi | nope, there it went again | 16:57 |
| fungi | firefox 31 | 16:57 |
| *** bhuvan has joined #openstack-infra | 16:57 | |
| fungi | wasn't doing it at all yesterday and i've not updated anything | 16:57 |
| anteaya | are you typing when it disconnects? or is it just open | 16:58 |
| fungi | just open | 16:58 |
| fungi | other people are typing | 16:58 |
| anteaya | funny, mine was usually while I was typing, but a few times when it was just open | 16:58 |
| fungi | it's possible there are leaks in wherever it handles its javascript stuff and i've bogged it down with a ton of gerrit tabs and whatnot | 16:58 |
| anteaya | can you open the same tab in chromium and see what your experience is? | 16:59 |
| anteaya | or another browser? | 16:59 |
| fungi | because it's also been taking me 2+ minutes to open or force reload the pad | 16:59 |
| anteaya | ah | 16:59 |
| *** harlowja_away is now known as harlowja | 16:59 | |
| *** deepakcs has quit IRC | 16:59 | |
| fungi | i don't have chromium installed but may test it later if i get around to it. for now i'm just going to see if restarting my browser helps at all | 17:00 |
| *** dizquierdo has joined #openstack-infra | 17:00 | |
| anteaya | I don't have a ton of tabs open but I do have a number of other pads open when it took forever to load a new pad this morning, the one I gave up on and just worked on in chromium | 17:00 |
| anteaya | go you | 17:00 |
| *** viglesias has joined #openstack-infra | 17:00 | |
| *** ddieterly has joined #openstack-infra | 17:00 | |
| *** derekh has quit IRC | 17:00 | |
| *** ddieterly has quit IRC | 17:02 | |
| *** bhuvan has quit IRC | 17:03 | |
| pelix | waynr: just a few minor nits on https://review.openstack.org/176389 but looks real good, be nice to have the code base cleaned up a bit. | 17:03 |
| *** shashankhegde has quit IRC | 17:03 | |
| *** andreykurilin__ has quit IRC | 17:03 | |
| *** ildikov has quit IRC | 17:04 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** tqtran has joined #openstack-infra | 17:05 | |
| anteaya | mtreinish: thank you | 17:06 |
| *** subscope_ has quit IRC | 17:07 | |
| *** jamesmcarthur has joined #openstack-infra | 17:07 | |
| mtreinish | anteaya: sure, np. I figured I should try and back up my assertions on the ML | 17:07 |
| *** subscope_ has joined #openstack-infra | 17:07 | |
| *** salv-orlando has joined #openstack-infra | 17:07 | |
| anteaya | and you did so | 17:08 |
| anteaya | can we discuss the potential values for elec_type: https://github.com/mtreinish/electionbot/blob/master/electionbot/elections.py#L38 | 17:09 |
| anteaya | I'm thinking we might be able to find a better value here :) | 17:09 |
| jerryz | fungi: adding skip-name-resolve fixed my issue. this time when i use pymysql with localhost as host to connect to the database, the connection won't get resolved to fqdn | 17:09 |
| anteaya | mtreinish: acclaimed comes to mind | 17:09 |
| mtreinish | anteaya: hehe, well it's internal only. I doubt it would come up in openstack though | 17:10 |
| jerryz | fungi: i wonder if infra's slave has this mysqld option enabled | 17:10 |
| mtreinish | it's an election for a single spot without projects or a seat count | 17:10 |
| fungi | jerryz: is that a parameter to the database connection, or a mysqld config option? | 17:10 |
| *** ddieterly has joined #openstack-infra | 17:10 | |
| fungi | jerryz: ahh, i'll check | 17:10 |
| jerryz | fungi: in my.cnf | 17:10 |
| anteaya | mtreinish: I like to plan for the possibility that it would | 17:11 |
| jeblair | anteaya: i'm seeing a patch in your future :) | 17:11 |
| anteaya | ha ha ha | 17:11 |
| *** AJaeger_ has joined #openstack-infra | 17:11 | |
| mtreinish | anteaya: sure that's why I put it in there. We can change that value to be whatever | 17:12 |
| zaro | fungi: did you get the info i provided yesterday on my Gerrit testing with regards to bouncycastle error? | 17:12 |
| *** annegentle has quit IRC | 17:13 | |
| fungi | zaro: i may have missed it. last night is a bit of a blur | 17:13 |
| anteaya | mtreinish: an election for a single spot without projects or a seat count: leader, lead? | 17:14 |
| *** dizquierdo has quit IRC | 17:14 | |
| fungi | jerryz: i don't see skip-name-resolve in /etc/mysql/my.cnf on our workers | 17:14 |
| *** yfried|afk has joined #openstack-infra | 17:15 | |
| *** sushilkm has joined #openstack-infra | 17:15 | |
| *** sushilkm has left #openstack-infra | 17:15 | |
| mtreinish | anteaya: sure, either works. That value is just an internal keyword for when it does validation and eventually when it automates the election creation (I didn't look at the CIVS stuff yet) | 17:15 |
| anteaya | doyen? | 17:15 |
| anteaya | there is a word we haven't overloaded yet | 17:16 |
| *** unicell has quit IRC | 17:16 | |
| *** unicell1 has joined #openstack-infra | 17:16 | |
| mtreinish | anteaya: heh, yeah I like that | 17:16 |
| anteaya | oh personally I would be really hesitant to have it automate election creation | 17:16 |
| mtreinish | why? | 17:17 |
| *** russellb has quit IRC | 17:17 | |
| anteaya | if we can clean up the nomination process I think we will have reduced the burden | 17:17 |
| mordred | anteaya: we could write a bot which generates an election for every patch proposal | 17:17 |
| anteaya | mtreinish: have you created a poll with more than 1000 people on the CVIS? | 17:17 |
| anteaya | mordred: we could | 17:17 |
| *** ddieterly has quit IRC | 17:17 | |
| *** dizquierdo has joined #openstack-infra | 17:17 | |
| mtreinish | anteaya: no, I haven't had a chance to look at it yet | 17:17 |
| *** jamesmcarthur has quit IRC | 17:17 | |
| anteaya | please create youself a couple of test polls | 17:18 |
| anteaya | feel free to use my email to send me some ballots | 17:18 |
| anteaya | using the tools are really helpful, relying on the tools is somethign else entirely | 17:18 |
| anteaya | is really helpful | 17:18 |
| mtreinish | anteaya: ah, it has a 1000 person limit | 17:19 |
| anteaya | I'm all for using helpful tools, I personally am very much against relying on them for elections | 17:19 |
| anteaya | yes | 17:19 |
| mtreinish | is there an api interface? or is it all web based? | 17:19 |
| *** jamesmcarthur has joined #openstack-infra | 17:19 | |
| fungi | well, 1000 addresses per batch | 17:19 |
| anteaya | so you have to split the list and submit a second list | 17:19 |
| anteaya | mtreinish: I have never looked myself, you can email the creater/maintainer | 17:19 |
| *** tjones1 has joined #openstack-infra | 17:19 | |
| fungi | or download the source | 17:20 |
| anteaya | it is only one person and he is very busy, be very polite please, as you always are | 17:20 |
| mtreinish | anteaya: sure, and it's all on github too: https://github.com/andrewcmyers/civs | 17:20 |
| anteaya | great | 17:20 |
| zaro | did a recheck in gerrit but can't find change in zuul queue. https://review.openstack.org/#/c/173999/ | 17:20 |
| *** davideagnello has joined #openstack-infra | 17:21 | |
| *** russellb has joined #openstack-infra | 17:21 | |
| anteaya | he does reply to emails, I have offered to help him if he needed any help and he never took me up on it | 17:21 |
| anteaya | I figured it was the least I could do since we were using his service | 17:21 |
| *** teran has quit IRC | 17:22 | |
| fungi | hrm, latest change in the check pipeline is from 34 minutes ago | 17:22 |
| fungi | i bet stream-events tasks are hanging again | 17:22 |
| *** teran has joined #openstack-infra | 17:22 | |
| fungi | i'll get a few gerrit thread dumps spaced 10 seconds or so apart, then restart it again | 17:22 |
| *** lennyb has joined #openstack-infra | 17:23 | |
| anteaya | stream-events is hung for me | 17:23 |
| *** jamesmcarthur has quit IRC | 17:24 | |
| *** dhritishikhar_ has joined #openstack-infra | 17:24 | |
| *** sdake has joined #openstack-infra | 17:24 | |
| clarkb | get connection data too right? | 17:25 |
| clarkb | also I am not here (Drs office) | 17:25 |
| *** Shrews has quit IRC | 17:26 | |
| *** ChuckC has joined #openstack-infra | 17:26 | |
| *** Shrews has joined #openstack-infra | 17:26 | |
| zaro | fungi: so the info was that i tested Gerrit 2.8.4.17 (without jeblair's change). when i enabled contactstore in gerrit.config Gerrit doesn't startup completely. | 17:26 |
| fungi | zaro: yep, i saw that much | 17:27 |
| zaro | fungi: log with full debug: http://paste.openstack.org/show/205031/ | 17:27 |
| *** dhritishikhar has quit IRC | 17:27 | |
| zaro | fungi: ohh. so I'm not sure i can debug further on my VM. | 17:27 |
| *** dizquierdo has quit IRC | 17:28 | |
| zaro | fungi: options now are to revert review-dev back to 2.8 and test it there or you try and run it manually on review.o.o | 17:28 |
| *** dizquierdo has joined #openstack-infra | 17:28 | |
| *** dboik has joined #openstack-infra | 17:29 | |
| *** bhuvan has joined #openstack-infra | 17:29 | |
| *** sdake has quit IRC | 17:29 | |
| zaro | fungi: since you'll need to restart for stream-event hang maybe try to deploy 2.8.4.19? | 17:29 |
| *** sdake has joined #openstack-infra | 17:30 | |
| *** dboik has quit IRC | 17:30 | |
| *** dboik has joined #openstack-infra | 17:31 | |
| *** dboik has quit IRC | 17:31 | |
| *** ssam2_ has quit IRC | 17:31 | |
| fungi | #status notice gerrit is restarting to clear hung stream-events tasks. any review events between 16:48 and 17:32 utc will need to be rechecked or have their approval votes reapplied to trigger testing in zuul | 17:31 |
| openstackstatus | fungi: sending notice | 17:31 |
| *** dboik has joined #openstack-infra | 17:31 | |
| zaro | fungi: just remember to install the core plugins from 2.8.4.19 war as well: http://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/init.pp#n572 | 17:32 |
| -openstackstatus- NOTICE: gerrit is restarting to clear hung stream-events tasks. any review events between 16:48 and 17:32 utc will need to be rechecked or have their approval votes reapplied to trigger testing in zuul | 17:32 | |
| fungi | okay, gerrit is back online now | 17:32 |
| zaro | ok. i guess no go on 19 | 17:33 |
| openstackstatus | fungi: finished sending notice | 17:33 |
| fungi | i wonder if the content of jstack dumps is security sensitive? | 17:33 |
| *** Shrews has quit IRC | 17:34 | |
| fungi | i see some threads blocked in com.google.gerrit.sshd.commands.StreamEvents.writeEvents() | 17:35 |
| *** Shrews has joined #openstack-infra | 17:35 | |
| Kiall | Weird... patch merged, then got a merged failed ;) https://review.openstack.org/#/c/176318/ | 17:35 |
| Kiall | Bug? | 17:35 |
| *** russellb has quit IRC | 17:35 | |
| *** ddieterly has joined #openstack-infra | 17:35 | |
| fungi | Kiall: i think that may be a race we see when merge-check pipeline processing gets backed up | 17:36 |
| *** shashankhegde has joined #openstack-infra | 17:36 | |
| fungi | zaro: do you expect a jstack dump of the gerrit process to contain any passwords or keys? | 17:37 |
| *** gyee is now known as chinese_gyee | 17:37 | |
| fungi | i'm looking through these now to see if it's safe to publish them or not | 17:37 |
| *** chinese_gyee is now known as gyee | 17:38 | |
| *** luqas has joined #openstack-infra | 17:38 | |
| clarkb | I wonder if a well behaved proxy in front of gerrit would help | 17:38 |
| asselin | anteaya, so do you think a fishbowl or workroom is better? | 17:38 |
| *** pelix has quit IRC | 17:38 | |
| openstackgerrit | Jerry Zhao proposed openstack-infra/nodepool: add option to use ipv6 for image update and node launching https://review.openstack.org/156178 | 17:38 |
| clarkb | so only one persistent ssh connection then everything else connects to a publiaher of wvents | 17:38 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/system-config: Update wildfly puppet module version https://review.openstack.org/176409 | 17:39 |
| fungi | oh, got it, a many-to-one event stream ssh proxy | 17:39 |
| anteaya | asselin: which would allow you to get the most work done? I think wasting time with more discussion doesn't help you create patches, the spec is up and approved, more time for chat is time wasted to my reconing, what do you think? | 17:39 |
| clarkb | ya | 17:40 |
| *** ddieterly has quit IRC | 17:40 | |
| fungi | not sure how that would work on the ssh end of things | 17:40 |
| *** hdd has joined #openstack-infra | 17:40 | |
| *** russellb has joined #openstack-infra | 17:40 | |
| zaro | fungi: sorry i have not used that tool | 17:40 |
| anteaya | asselin: fishbowl is talk, workroom is work | 17:40 |
| fungi | connection session keys are negotiated between one client and one server | 17:40 |
| *** Sukhdev has quit IRC | 17:40 | |
| asselin | anteaya, ok thanks | 17:40 |
| anteaya | asselin: sure | 17:40 |
| zaro | fungi: same thing as this? https://groups.google.com/d/msg/repo-discuss/NuFti4SVNQM/AMHOGXk2zPsJ | 17:40 |
| clarkb | well it would need to be a smart app specific proxy then client side would be rabbit/zmq/protobufs/something | 17:40 |
| clarkb | fungi so only ssh between gerrit and proxy | 17:41 |
| clarkb | then use a proper message bus for publishing | 17:41 |
| Kiall | fungi: actually, I just noticed that every designate patch (not sure about others) just have a merge conflict reported on it.. even ones I'm 150% sure are OK.. I can recheck, but, might be affecting everyone if it's really an issue :) | 17:42 |
| fungi | zaro: sort of, except the threads don't have names, just thread numbers and states | 17:42 |
| *** david-lyle has quit IRC | 17:42 | |
| fungi | zaro: though i had to use jstack -F so not sure if that makes a difference | 17:42 |
| *** ivar-lazzaro has joined #openstack-infra | 17:43 | |
| anteaya | mtreinish: to submit a patch to your github repo, git add, git commit, git push? | 17:43 |
| fungi | Kiall: it's possible a zuul merger worker tried to update from gerrit right at the moment i was restarting it. that looks like about the right timing | 17:43 |
| mordred | as a middle step - an ssh-based one that was a single gerrit ssh connection and published an ssh stream on the other side (just a multi-plexer-passthrough) might even be possible to hack together with regular ssh - and then we could just deploy 5 or 6 of them as needed | 17:43 |
| *** luqas has quit IRC | 17:43 | |
| clarkb | mordred ya | 17:44 |
| *** lennyb has left #openstack-infra | 17:44 | |
| mordred | clarkb: like, brainddead version - have a stream-events consumer writing the output to a file, and write a command on the server called "gerrit stream-events" that is just tail -f on the file | 17:44 |
| Kiall | fungi: makes sense :) | 17:44 |
| * mordred waves hands | 17:44 | |
| mordred | clarkb: but it could reduce load on gerrit itself, and should take about 15 minutes to spin up a test version | 17:44 |
| pleia2 | cinerama: hey, so just pulled up the outstanding zanata topic reviews, I can review after we chat, what else do we have on our plate? | 17:44 |
| fungi | mordred: clarkb: proxying the auth would get interesting. the usernames and keys live in gerrit's database | 17:45 |
| * pleia2 opens up zanata-install etherpad too | 17:45 | |
| fungi | though i suppose we could just have one "stream" user and auth via password or something | 17:45 |
| mordred | fungi: we could make the account no-auth - it's tied to running the stream events command | 17:45 |
| *** radez_afk is now known as radez | 17:45 | |
| cinerama | pleia2: i've been updating the etherpad today and patching some bits and pieces. i think the main thing is getting the jenkins scripts doing the things correctly | 17:45 |
| mordred | fungi: why a password even? just passwordless - who cares if you consume the stream | 17:45 |
| anteaya | mtreinish: nope | 17:45 |
| *** mpaolino has quit IRC | 17:45 | |
| fungi | passwordless how? | 17:46 |
| mtreinish | anteaya: it's github, I always forget the workflow. I think you have to fork it to your account push the commit to that and then submit a pull request | 17:46 |
| anteaya | ah okay | 17:46 |
| mordred | fungi: dunno - haven't thought about it that much :) | 17:46 |
| pleia2 | cinerama: cool, reading through now, give me a nudge if there's something I should pay particular attention to | 17:46 |
| cinerama | pleia2: so - comparing the capabilities of the clients, deciding how we want to handle the account for it, etc | 17:46 |
| anteaya | I will attempt that | 17:46 |
| fungi | maybe there's some no-op auth method supported by openssh i'm unaware of | 17:46 |
| mtreinish | anteaya: do you want me to just submit the project-config change to add it to gerrit? | 17:46 |
| *** dkranz has quit IRC | 17:46 | |
| cinerama | pleia2: i got a bit stuck on the client parts so i thought we could work through that together | 17:46 |
| anteaya | after I go for a walk, I need to go for a walk | 17:46 |
| mordred | fungi: I've never had teh need to look for such a setting myself :) | 17:46 |
| cinerama | pleia2: i think i just need to get unstuck | 17:46 |
| fungi | nor i! | 17:46 |
| anteaya | mtreinish: I would like to get some opinions on what the elections officials would like | 17:47 |
| anteaya | I was just submitting a patch because I think language is important and jeblair encouraged me | 17:47 |
| pleia2 | cinerama: yeah, I got as far as building the client the other day from the puppet manifest, but I haven't actually done anything with it | 17:47 |
| anteaya | I still think that having a repo for election nominations is easier than using the mailing list at all | 17:48 |
| mtreinish | anteaya: sure that's why I didn't do it to start. I just thought the workflow would be easier if you're ready for a patch :) | 17:48 |
| fungi | zaro: aha, looks like to get thread names i need to run jstack as the gerrit2 user | 17:48 |
| anteaya | mtreinish: thanks and I understand, know that regardless of the outcome where ever it lives I'm supportive of doyen over dictator | 17:49 |
| cinerama | pleia2: do we have the zanata client on a dev proposal slave? | 17:49 |
| mordred | fungi: PermitEmptyPasswords | 17:49 |
| anteaya | but I personally would like the current election officials to have the time after they are finished to offer their thoughts on what they would like | 17:49 |
| mordred | When password authentication is allowed, it specifies whether the server allows login to accounts with empty password strings. The default is “no”. | 17:49 |
| mordred | fungi: I think that would be it | 17:49 |
| pleia2 | cinerama: yes, I believe clarkb tested it after the change merged a couple weeks back | 17:50 |
| anteaya | mtreinish: and I'm appreciative of the fact you are willing to back up your ideas with code, thank you | 17:50 |
| anteaya | mtreinish: and i need to get a walk in while it is not raining :) | 17:50 |
| *** yamahata has joined #openstack-infra | 17:51 | |
| pleia2 | cinerama: confirmed, zanata-cli is there | 17:51 |
| *** achanda has joined #openstack-infra | 17:51 | |
| cinerama | pleia2: so for translation, i also care about the propose_translation* scripts in project-config, and common_translation_update.sh. any more? clarkb? | 17:52 |
| *** ddieterly has joined #openstack-infra | 17:52 | |
| fungi | mordred: poking around a little, looks like that may do it but you need to be mindful of which of the authentication methods can effectively negate that | 17:52 |
| *** ddieterly has quit IRC | 17:52 | |
| fungi | mordred: it looks like it may not work with pam auth, or keyboard-interactive password auth, but testing will tell | 17:53 |
| *** jerryz has quit IRC | 17:53 | |
| pleia2 | cinerama: refreshing my memory real quick on the spec to confirm | 17:54 |
| mordred | fungi: so we'd need to basically configure the daemon to only allow password auth | 17:54 |
| mordred | with pam and keyboard-interactive turned off | 17:55 |
| fungi | possibly, yeah | 17:55 |
| cinerama | pleia2: oh the other weird thing about the client is that at times it expects you to provide data on the command line which is in the zanata.ini file. not sure if it's a bug or i'm just not using it right | 17:55 |
| cinerama | pleia2: the other thing is that the client is unhappy with self-signed certs - i think i imported the cert from translate-dev into the ssl keystore on my VM i was plaing with | 17:56 |
| pleia2 | cinerama: we'll have a real cert on the live production cert, we just don't typically do them with dev servers | 17:57 |
| *** sarob has joined #openstack-infra | 17:57 | |
| *** annegentle has joined #openstack-infra | 17:58 | |
| pleia2 | so if it's possible to work around it with the client, we can do that for now | 17:58 |
| cinerama | pleia2: yep. does impact our testing a bit | 17:58 |
| *** e0ne has joined #openstack-infra | 17:58 | |
| * pleia2 nods | 17:58 | |
| pleia2 | we should just document the "make it work" steps so everyone working on this knows how to do testing (particularly as we do reviews) | 17:58 |
| pleia2 | cinerama: re: client + zanata.ini file, the issue is that you need access to the data in the zanata.ini on the server, but don't have it? (I assume for user auth or something?) | 17:59 |
| *** sushilkm has joined #openstack-infra | 18:00 | |
| *** sushilkm has left #openstack-infra | 18:00 | |
| cinerama | pleia2: so the zanata.ini file is in place client side as it should be but the client whinges about not having the username, api key, and url, which are all in the ini | 18:00 |
| * SpamapS looks at epic zuul queue and is reminded to do some work on documenting design of infra cloud. | 18:01 | |
| *** annegentle has quit IRC | 18:01 | |
| pleia2 | cinerama: hm, yeah, I'm not seeing a --config option either to point it at the .ini | 18:01 |
| *** dkranz has joined #openstack-infra | 18:01 | |
| *** annegentle has joined #openstack-infra | 18:02 | |
| fungi | for future reference: sudo apt-get install --no-install-recommends openjdk-7-jdk ; sudo -u gerrit2 jstack `cat ~gerrit2/review_site/logs/gerrit.pid` ; sudo apt-get purge --auto-remove openjdk-7-jdk | 18:02 |
| fungi | redirect that middle command to files a few times 10 or so seconds apart (stdout and stderr) | 18:02 |
| fungi | that will get us some thread dumps with the thread names resolved correctly | 18:03 |
| *** yamahata has quit IRC | 18:03 | |
| *** otter768 has joined #openstack-infra | 18:03 | |
| fungi | clarkb: mordred: jeblair: pleia2: jhesketh: SergeyLukjanov: ^ | 18:03 |
| cinerama | pleia2: some of the subcommands seem to deal with finding that information & others don't. the ini file default location is where we put it in the client.pp | 18:04 |
| SergeyLukjanov | fungi, ack, interesting | 18:04 |
| pleia2 | cinerama: yeah, on proposal it's in /home/jenkins/.config/zanata.ini | 18:04 |
| cinerama | pleia2: so i think bug but... | 18:05 |
| pleia2 | cinerama: does your zanata.ini have anything in it? it doesn't actually have the values populated on the proposal slave | 18:06 |
| *** david-lyle has joined #openstack-infra | 18:07 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:07 | |
| *** dhritishikhar_ has quit IRC | 18:07 | |
| *** dhritishikhar_ has joined #openstack-infra | 18:07 | |
| *** otter768 has quit IRC | 18:08 | |
| *** isviridov_away is now known as isviridov | 18:11 | |
| *** eharney has quit IRC | 18:11 | |
| fungi | zaro: so https://issues.apache.org/jira/browse/SSHD-348 looks like 0.14 is supposed to solve the stream-events hangs? i can confirm these are blocked in Object.wait() as described in https://issues.apache.org/jira/browse/SSHD-348 | 18:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib) https://review.openstack.org/176343 | 18:14 |
| fungi | https://gerrit-review.googlesource.com/65696 seems to have been when they imported 0.14.0 | 18:14 |
| fungi | a couple weeks ago now | 18:14 |
| EmilienM | if some core-reviewers are around, we have two 'small' patches in infra that could help us to move forward with our CI: https://review.openstack.org/#/c/176070/ and https://review.openstack.org/#/c/176343/ | 18:14 |
| cinerama | pleia2: oops. i will look at the config for the proposal slave in a sec. we'll need to make an account that the client can use...how should we do that? | 18:15 |
| pleia2 | cinerama: in the long run for production, someone on infra should probably just create a jenkins user to interact with it in zanata, for now I think you can just add one on the dev server | 18:16 |
| cinerama | pleia2: what email should we use for the dev jenkins user? | 18:17 |
| annegentle | fungi: hey, AJaeger and I want to pick your brain for ideas on sharing the FTP credentials for docs.openstack when you have a minute. | 18:17 |
| pleia2 | cinerama: hm, does it need to do the auth process? | 18:18 |
| annegentle | fungi: sharing them so far has been me calling Jim. What can we do to expand to AJaeger and Lana incoming PTL? | 18:18 |
| *** dangers is now known as dangers_away | 18:18 | |
| fungi | annegentle: i can sort of multi-task. cloudsites lets you add multiple ftp accounts for one site, right? | 18:18 |
| pleia2 | cinerama: actually, we can always change the email address later, just use one of yours for now | 18:18 |
| cinerama | pleia2: it does. we can't currently use the local accounts at the same time as openid accounts | 18:18 |
| annegentle | fungi: oh! Hadn't thought of that but yes | 18:18 |
| pleia2 | oh, it needs an openid account, hm | 18:18 |
| annegentle | fungi: and it really can be "when you have a minute" :) | 18:19 |
| cinerama | pleia2: we would need to change zanata to do that. they did offer support if we wanted to contribute that iirc | 18:19 |
| pleia2 | cinerama: to support both? | 18:19 |
| fungi | annegentle: though i suppose you need rackspace dashboard access to manage ftp accounts... so there's still likely a bottleneck even if we hand out individual ftp accounts | 18:20 |
| cinerama | pleia2: yeah. they supply a login screen where you can choose from different openid providers but we'd need to whip up a login screen for the case where a specific openid provider is specified + using local accounts | 18:20 |
| fungi | annegentle: i'm _hopeful_ we'll be done with rackspace cloudsites hosting by the time the next ptl election rolls around, but that depends on how much time we get to put into implementation there | 18:20 |
| pleia2 | cinerama: hm, does the client do openid? | 18:21 |
| *** garyk has quit IRC | 18:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:21 | |
| *** tonytan_brb has joined #openstack-infra | 18:22 | |
| *** ianw has quit IRC | 18:22 | |
| cinerama | pleia2: the client uses an API key which you generate on the web for that account. hmm. so we might not actually need to be able to login on the website. let me have a play for a sec | 18:23 |
| annegentle | fungi: jeblair said hopefully at the start of liberty so I'd love that | 18:23 |
| *** tonytan4ever has quit IRC | 18:23 | |
| annegentle | fungi: but yes, we should share these creds in the meantime | 18:23 |
| annegentle | fungi: and true on the dashboard / control panel access | 18:24 |
| annegentle | fungi: I can always do phone calls | 18:24 |
| *** ayoung has quit IRC | 18:24 | |
| *** alexpilotti has quit IRC | 18:25 | |
| *** ianw has joined #openstack-infra | 18:25 | |
| cinerama | pleia2: heh. i don't think i can create a user on the web. i might be able to create a local user by messing with the database but i also can't manipulate other users' api keys from the web | 18:25 |
| fungi | annegentle: well, i mean phone calls are just one out-of-band communication mechanism. we could for example to openpgp-encrypted e-mails to hand off passwords if preferred | 18:26 |
| zaro | fungi: i'm not sure about that. IIRC that sshd fix happened, sshd released ver 0.14.0, gerrit updated to 0.14.0 but problem still occurs. i believe that's what happened, but would need to double check | 18:26 |
| annegentle | hm | 18:26 |
| fungi | zaro: oh, the bug you pointed me at about mina-sshd 0.14.0 said that gerrit itself froze | 18:26 |
| annegentle | fungi: how much would it matter if Lana, Andreas and I haven't signed each other's keys? No biggie? | 18:26 |
| *** ddieterly has joined #openstack-infra | 18:27 | |
| fungi | zaro: "the web-interface was dead" according to the initial message | 18:27 |
| AJaeger_ | annegentle: my key was signed by fungi and others | 18:27 |
| pleia2 | cinerama: hm, for some initial testing, can we use our own user accounts? | 18:27 |
| annegentle | AJaeger_: yeah | 18:27 |
| annegentle | to me, it's good enough | 18:27 |
| fungi | annegentle: no less secure than a phone call ;) | 18:27 |
| cinerama | pleia2: yes | 18:27 |
| annegentle | right :) | 18:27 |
| *** marun has joined #openstack-infra | 18:28 | |
| pleia2 | cinerama: I think we just document the issue we have now (what address+associated openstackid account to use, and who/how to generate the keys) and chat with the rest of the team when we need to go into prod | 18:28 |
| fungi | annegentle: "i think this is the right key and e-mail address" is probably at least as secure as "i think this is the right phone number and that sounds like her voice" | 18:28 |
| pleia2 | cinerama: or at least get closer to prod :) | 18:28 |
| annegentle | fungi: heh | 18:29 |
| cinerama | pleia2: we'll need to change system-config/modules/openstack_project/manifests/proposal_slave.pp to pass along the values | 18:29 |
| *** jogo has quit IRC | 18:29 | |
| pleia2 | cinerama: yeah, probably something we'll need to add to hiera | 18:29 |
| cinerama | pleia2: i will propose that in a sec | 18:30 |
| zaro | fungi: maybe a slightly different problem than just ssh hang, not sure but looks like user definately confirmed some stability issues with updated sshd lib. | 18:30 |
| pleia2 | cinerama: we just don't know what the values are yet until we discuss it with everyone :) | 18:30 |
| cinerama | pleia2: yep but for dev we can at least chuck in our temporary values | 18:30 |
| clarkb | ok I am back | 18:31 |
| *** BharatK has quit IRC | 18:31 | |
| zaro | fungi: still not sure which path you wanted to take with testing 2.8.4.19 on review-dev or review? | 18:31 |
| fungi | zaro: yep, i agree that's a new risk and probably a worse issue, but not necessarily the same issue | 18:31 |
| pleia2 | cinerama: well my hope is that once we get to actually using the proposal slave, we'll be closer to prod and have real values | 18:32 |
| pleia2 | not sure we should be adding dev server values to the proposal slave | 18:32 |
| cinerama | pleia2: so do you want to experiment with a local client for now to get things working, or just mod the zanata.ini on there by hand? | 18:32 |
| zaro | fungi: maybe just 2 weeks out until upgrade so maybe just cherry-pick jeblair's change onto our 2.10.2 branch and test on review-dev? | 18:33 |
| clarkb | pleia2: cinerama ya we should be able to run the scripts by hand from an arbitrary location against the dev server | 18:33 |
| *** ildikov has joined #openstack-infra | 18:33 | |
| pleia2 | cinerama: I think we leave the proposal slave alone until we're actually ready to go with most things | 18:33 |
| fungi | zaro: that sounds reasonable | 18:33 |
| openstackgerrit | Merged openstack-infra/shade: Search methods for networks, subnets and routers https://review.openstack.org/175027 | 18:33 |
| *** yamahata has joined #openstack-infra | 18:33 | |
| clarkb | pleia2: +1 | 18:33 |
| openstackgerrit | Merged openstack-infra/shade: Use new getters in update_subnet and update_router https://review.openstack.org/175837 | 18:33 |
| pleia2 | cinerama: there's nothing special about the proposal slave that we can't replicate in our local instances | 18:33 |
| fungi | zaro: want to see if it applies cleanly and put up a change? i'll go ahead and approve it once the buck tests pass | 18:33 |
| zaro | fungi: i'll do try that. | 18:33 |
| *** ddieterly has quit IRC | 18:33 | |
| * zaro needs to head out for errands, bbl. | 18:34 | |
| fungi | zaro: at this point i'm hesitant to take review.openstack.org offline long enough to replace the warfile between now and the kilo release | 18:34 |
| cinerama | pleia2: just thoguht it might be easier for clarkb to give us a hand, etc | 18:34 |
| fungi | so waiting to have more debug info in 2.10 makes sense | 18:34 |
| clarkb | cinerama: I thnk as long as we are pushing the code someplace (a gerrit chagne) so we can all grab it and run it that should be fine | 18:34 |
| *** sarob has quit IRC | 18:35 | |
| clarkb | fungi: zaro catching up, latest 2.10 may not fix ssh problems? | 18:35 |
| openstackgerrit | Merged openstack-infra/shade: Update volume API for new getters and dict retval https://review.openstack.org/176359 | 18:36 |
| openstackgerrit | Merged openstack-infra/shade: Add test of OperatorCloud auth_type=None https://review.openstack.org/172165 | 18:36 |
| clarkb | and we don't have jeblair's change deployed because gerrit won't start wiht that war? | 18:36 |
| fungi | clarkb: sounds like it's supposed to fix this problem but may introduce new and more exciting problems instead | 18:36 |
| openstackgerrit | Merged openstack-infra/shade: Update register_machine to use tasks https://review.openstack.org/171324 | 18:36 |
| openstackgerrit | Merged openstack-infra/shade: Update unregister_machine to use tasks https://review.openstack.org/171325 | 18:36 |
| cinerama | we need to be able to pass those values anyway so i'll at least put up the change that would be required | 18:36 |
| clarkb | the best kind of update | 18:36 |
| fungi | clarkb: right, i couldn't get gerrit to start on review.o.o with the updated warfile for that debugging patch | 18:36 |
| pleia2 | cinerama: ok, maybe just WIP it until we have the values to put into hiera? | 18:37 |
| fungi | clarkb: and if we upgrade via puppet we get a 10+ minute outage for reindexing to complete (which we could work around if we need to) | 18:37 |
| *** dkranz has quit IRC | 18:37 | |
| cinerama | pleia2: sure | 18:38 |
| *** dboik_ has joined #openstack-infra | 18:38 | |
| fungi | but more just thinking that the number of restarts for stream-events hangs between now and release day are likely preferable to taking gerrit offline for a while to try and reapply that updated warfile | 18:38 |
| fungi | given that it doesn't make these restarts unnecessary, just has a chance of allowing us to do some deeper debugging | 18:39 |
| *** ddieterly has joined #openstack-infra | 18:39 | |
| clarkb | fungi: ya | 18:39 |
| fungi | so in total, we'll most likely have more gerrit downtime between now and the release if we try that again, vs if we wait | 18:39 |
| clarkb | if it was a fix we should do it but its just debug info | 18:39 |
| fungi | stab-in-the-dark debug info at that | 18:40 |
| openstackgerrit | Samuel BARTEL proposed openstack-infra/project-config: add project fuel-plugin-nova-nfs https://review.openstack.org/175452 | 18:40 |
| *** jogo has joined #openstack-infra | 18:40 | |
| pleia2 | cinerama: so I'm going to get myself up to speed with the client stuff, be prepared for stupid questions this afternoon as I make my way through :) | 18:40 |
| *** yamahata has quit IRC | 18:40 | |
| *** samuelBartel has joined #openstack-infra | 18:41 | |
| *** dboik has quit IRC | 18:41 | |
| fungi | clarkb: also there's a _chance_ that this new problem with mina-sshd 0.14.0 in gerrit 2.10.3 will be solved before time for us to upgrade | 18:41 |
| cinerama | pleia2: np - in fact that will actually help a lot | 18:41 |
| clarkb | fungi: that would be nice | 18:41 |
| clarkb | fungi: zaro where is the link to the new gerrit ssh thing? | 18:41 |
| fungi | the discussion there is at least active and current | 18:41 |
| *** isviridov is now known as isviridov_away | 18:42 | |
| fungi | https://groups.google.com/forum/#!msg/repo-discuss/Kv4bWLESbQ4/-oSNbuTQwkUJ | 18:42 |
| *** yamamoto has quit IRC | 18:42 | |
| redrobot | fungi I had a question about the changes you've been doing to the gerrit ACLs. According to the Project Creator's Guide, we should have 2 gerrit groups barbican-core and barbican-release. I see a third gropu barbican-milestone in gerrit, and I was wondering what the purpose of it was? | 18:42 |
| clarkb | ty | 18:42 |
| AJaeger_ | redrobot: previously we used -milestone instead of -release | 18:43 |
| AJaeger_ | so, it's the same | 18:43 |
| fungi | redrobot: you have the option of a separate group in charge of approving fixes proposed to milestones (generally just the proposed branch immediately prior to release now) | 18:43 |
| *** Hal has joined #openstack-infra | 18:44 | |
| fungi | redrobot: the release group was previously called "ptl" instead but confused people because it was mostly used for the ptl and their release delegates to be able to tag releases of libraries | 18:44 |
| *** Hal is now known as Guest89237 | 18:44 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib) https://review.openstack.org/176343 | 18:44 |
| EmilienM | fungi: not sure I did correctly, let me know what you think | 18:45 |
| fungi | now those milestone and release groups are probably mostly redundant in newer projects | 18:45 |
| clarkb | anything pressing I should look at? otherwise I am going to go back to trying to catch up on code review | 18:45 |
| fungi | redrobot: the "milestone" groups in older projects were at one time called "drivers" instead and were sync'd in from the groups of the same name in launchpad | 18:45 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/shade: Use a watch that isn't affected by time going backwards https://review.openstack.org/176430 | 18:45 |
| *** ddieterly has quit IRC | 18:45 | |
| redrobot | fungi I see... so I can safely ignore xxx-milestone for now? It doesn't seem to have any privileges in stable/kilo | 18:45 |
| fungi | redrobot: yeah, sounds like you probably won't have any need for it | 18:46 |
| redrobot | fungi right on, thanks! | 18:46 |
| redrobot | and thank you too AJaeger_ | 18:46 |
| *** eharney has joined #openstack-infra | 18:46 | |
| *** tonytan4ever has joined #openstack-infra | 18:48 | |
| fungi | clarkb: i proposed the new repo creation change for the openstackid-resources api server split after discussing with smarcet: https://review.openstack.org/176334 | 18:48 |
| *** ZZelle has quit IRC | 18:48 | |
| fungi | that's a quickie review | 18:48 |
| *** ZZelle has joined #openstack-infra | 18:49 | |
| *** dkranz has joined #openstack-infra | 18:49 | |
| *** thedodd has joined #openstack-infra | 18:49 | |
| fungi | trying to encourage tipit to start new projects in gerrit from the beginning (he seems confident that he's going to write it from scratch rather than the other option we discussed of forking the openstackid codebase) | 18:49 |
| *** tonytan_brb has quit IRC | 18:50 | |
| *** stevemar has joined #openstack-infra | 18:50 | |
| clarkb | cool will review now | 18:51 |
| clarkb | iirc TC approved the change letting us all make projects without all the oversight in the world | 18:51 |
| fungi | yep. i'm about to do a couple cosmetic fixes to jeblair's corresponding infra-manual patch and then we can approve it as well i think | 18:52 |
| fungi | but i basically followed the new process from that | 18:52 |
| fungi | clarkb: also outreachy ideas brainstorming is happening in https://etherpad.openstack.org/p/infra-liberty-outreachy | 18:52 |
| clarkb | oh I had a couple ideas to add | 18:53 |
| *** hdd has quit IRC | 18:53 | |
| fungi | please do. i haven't had a sec to try to come up with any, but the sooner we get up with vkmc on this stuff the more likely we are to have an intern | 18:53 |
| *** ddieterly has joined #openstack-infra | 18:54 | |
| *** hdd has joined #openstack-infra | 18:54 | |
| clarkb | ok added, back to reviews | 18:56 |
| stevemar | fungi, clarkb qq for you guys, someone i know wants to do research on openstack reviews and such, is there any data available or would he have to skim everything? | 18:57 |
| *** a13x2121 has quit IRC | 18:57 | |
| samueldmq | stevemar, just the metrics for someone/some org ? | 18:57 |
| clarkb | stevemar: gerrit has a rest api | 18:57 |
| fungi | stevemar: there are some pretty comprehensive apis to gerrit if he's not adverse to scripting something to query and distill what he needs | 18:57 |
| fungi | stevemar: there's also stackalytics.com and activity.openstack.org with some already baked analysis | 18:58 |
| samueldmq | ^ I have used stackalytics api | 18:58 |
| *** ddieterly has quit IRC | 18:58 | |
| stevemar | will performing a bunch of actions on the REST apis be painful? | 18:58 |
| fungi | stevemar: nope. i have some scripts that query a ton of stuff via the gerrit rest api | 18:59 |
| *** dizquierdo has quit IRC | 18:59 | |
| stevemar | awesome, i let him know about the -infra channel | 18:59 |
| stevemar | cool cool | 18:59 |
| fungi | stevemar: also if you've ever used gertty, it operates _entirely_ via gerrit's rest api, which is why i say it's comprehensive | 18:59 |
| stevemar | thanks guys :) | 18:59 |
| stevemar | very preliminary discussion | 18:59 |
| *** dizquierdo has joined #openstack-infra | 19:00 | |
| fungi | anyone with an interest in our development process and tools is welcome | 19:00 |
| fungi | i love talking with people about this stuff | 19:00 |
| *** jamesmcarthur has joined #openstack-infra | 19:00 | |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Remove YamlParser from jenkins_jobs.builder https://review.openstack.org/176389 | 19:01 |
| clarkb | stevemar: also you can grab all the git repos to do munging of git info | 19:01 |
| clarkb | so lots of options here, likely depends on what perspective you are looking for | 19:01 |
| *** dizquierdo has quit IRC | 19:02 | |
| *** shashankhegde has quit IRC | 19:02 | |
| *** pblaho_ has joined #openstack-infra | 19:02 | |
| greghaynes | mordred: looks like the big nodepool + shade patch stack now passes after rechecks | 19:02 |
| greghaynes | FYI | 19:02 |
| *** dizquierdo has joined #openstack-infra | 19:02 | |
| greghaynes | (and FYI for any other potential reviewers https://review.openstack.org/#/c/168603/7 :)) | 19:03 |
| clarkb | greghaynes: you should recheck the children too? | 19:03 |
| greghaynes | I did | 19:03 |
| clarkb | oh nevermind gertty sync lag | 19:03 |
| greghaynes | much rechecking was had | 19:04 |
| mordred | greghaynes: woot | 19:04 |
| stevemar | clarkb, fungi just talking to a researcher/grad student about CI/CD, he did a paper on a smaller project, but something on openstack would be awesome | 19:04 |
| mordred | greghaynes: I'm very excited about that | 19:04 |
| * clarkb notes that none of the shade changes brought up yesterday are using the priority topic | 19:04 | |
| mordred | stevemar: make sure he's aware of our graphite server as well | 19:04 |
| *** dtantsur|afk has quit IRC | 19:04 | |
| clarkb | so if people think they are required for nodepool + shade then that should be updated | 19:05 |
| mordred | clarkb: I think they're not actually essential to dib-nodepool | 19:05 |
| fungi | stevemar: yes, i think we're probably the largest free/open ci/cd operation of all time, so an excellent research target | 19:05 |
| stevemar | mordred, whosa whatsa graphite server? | 19:05 |
| mordred | they're more essential to fleshing out shade and making the ansible playbooks work | 19:05 |
| clarkb | mordred: ok, thats what was being argued, I agree | 19:05 |
| fungi | stevemar: graphite.openstack.org (it's what we use to generate the graphs at the bottom of the zuul status page among other things) | 19:05 |
| openstackgerrit | Merged openstack-infra/shade: Test that deleting user invalidates user cache https://review.openstack.org/173596 | 19:05 |
| mordred | clarkb: they're important - but many of them are not directly related to nodepool consumption | 19:05 |
| clarkb | mordred: roger | 19:05 |
| stevemar | ah | 19:05 |
| *** pblaho has quit IRC | 19:05 | |
| fungi | stevemar: statsd target and introspection interface | 19:06 |
| clarkb | stevemar: we shove timing data and counters in there for our tests and things then graphite has an api to do maths on them | 19:06 |
| mordred | SpamapS, Shrews: ^^ that said - with the interface update we're perpetrating right now ... we should get that applied to images and servers before we land nodepool patches that use those apis | 19:06 |
| clarkb | mordred: does that mean 168603 should be WIP? | 19:06 |
| stevemar | that is pretty crazy | 19:06 |
| fungi | stevemar: right now it's mostly storing performance, timing and counts from zuul activities, and some from nodepool as well | 19:06 |
| mordred | clarkb: no - that one is good to go | 19:07 |
| mordred | clarkb: that just uses the client passthrough objects - no interface changes happening there | 19:07 |
| *** a13x212 has joined #openstack-infra | 19:07 | |
| *** BobH1 has joined #openstack-infra | 19:07 | |
| clarkb | also why are we changing interfaces after we made it stable for ansible? | 19:07 |
| mordred | clarkb: the ansible patches have not yet landed - and in working on those we found issues in the interface that we need to fix to provide sane ansible results | 19:08 |
| fungi | stevemar: there's also an anonymously accessible mysql database we're sticking subunit trending information in from individual tests (via mtreinish's subunit2sql project) which can be mined for all sorts of job performance data | 19:08 |
| *** Guest89237 has quit IRC | 19:08 | |
| mordred | clarkb: basically, we had inconsistent semantics around get, list and search | 19:08 |
| fungi | stevemar: and logstash.openstack.org where we have a kibana front-end with more recent job logs for high-detail investigation | 19:09 |
| clarkb | mordred: oh I thought ansible stuff had landed | 19:09 |
| Shrews | mordred: they're on the roadmap | 19:09 |
| mordred | not yet | 19:09 |
| openstackgerrit | Merged openstack-infra/shade: create_user should return the user created https://review.openstack.org/174642 | 19:09 |
| mordred | Shrews, clarkb: and actually - I just checked - we have no nodepool patches up yet that would be affected | 19:09 |
| *** zz_zz_ja is now known as zz_zz_zz_ja | 19:09 | |
| *** pblaho__ has joined #openstack-infra | 19:10 | |
| mordred | Shrews: this: https://review.openstack.org/#/c/168634 is the only thing we have up that might want thought - I'm goign to WIP it real quick | 19:10 |
| openstackgerrit | Merged openstack-infra/shade: Fix major update_user issues https://review.openstack.org/174704 | 19:10 |
| stevemar | mordred, fungi, this is all wonderful info. thanks guys | 19:10 |
| *** pblaho_ has quit IRC | 19:10 | |
| *** dtantsur has joined #openstack-infra | 19:10 | |
| mtreinish | fungi: there's a subunit2sql command for graphing test run_time over time :) | 19:11 |
| * Shrews looks | 19:11 | |
| *** sarob has joined #openstack-infra | 19:11 | |
| mtreinish | fungi: I actually just pushed some patches up to make it a bit more useful: https://review.openstack.org/#/q/status:open+project:openstack-infra/subunit2sql+branch:master+topic:176114,n,z | 19:11 |
| mordred | Shrews: get_flavor_by_ram() seems to be a good candidate for the new search interface ... EXCEPT - it's a min ram number that gets passed in, so it still may want to be special in some ways | 19:12 |
| fungi | oh, neat | 19:12 |
| *** Hal has joined #openstack-infra | 19:12 | |
| *** sarob has quit IRC | 19:12 | |
| *** Hal is now known as Guest53534 | 19:12 | |
| mtreinish | well really only the first 2 in that series are for improving the run time graphing | 19:13 |
| mtreinish | the rest is for graphing/tracking test failure rates over time | 19:13 |
| Shrews | mordred: i think we can keep get_flavor_by_ram() as it can self filter after calling list. a get_flavor() method should be made available, though, if we do not have it | 19:13 |
| *** ArxCruz has joined #openstack-infra | 19:13 | |
| *** Guest53534 has quit IRC | 19:13 | |
| mordred | fungi, clarkb, jhesketh: thoughts/opinions on i18n support in shade? since it's intended to be a friendly end user library - it might perhaps want to support translated strings for error messages and such? | 19:14 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/system-config: WIP: Pass through zanata client parameters https://review.openstack.org/176440 | 19:14 |
| *** annegentle has quit IRC | 19:14 | |
| mordred | fungi, clarkb, jhesketh: the question arose as I was looking in to what gets dragged in by oslo.utils - because harlowja submitted a patch to change how elapsed time was calculated that uses timeutils in oslo.utils | 19:14 |
| *** claudiub has joined #openstack-infra | 19:15 | |
| harlowja | afaik oslo.utils already getting dragged in by all the clients :-P | 19:15 |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Generalize fingerprint for 1446847 https://review.openstack.org/176441 | 19:15 |
| clarkb | mordred: these opinions are my own. I think it is a mistake to drag in half of oslo*. stdlib works great | 19:15 |
| SpamapS | It's possible the whole infra toolchain would benefit from i18n to ease adoption for 3rd-party CI vendors who are not primarily english speaking. | 19:15 |
| SpamapS | harlowja: the clients are not going to be depended on in the mid-term future. | 19:15 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/system-config: WIP: Pass through zanata client parameters https://review.openstack.org/176440 | 19:15 |
| clarkb | mordred: but i8ln is likely a good idea since shade is making its own error messages | 19:15 |
| *** stevemar has quit IRC | 19:15 | |
| mordred | clarkb: well, in this case, apparently there is a real issue in the stdlib that timeutils addresses | 19:15 |
| Shrews | mordred: a bigger issue is that we still return image and server objects without making them dicts. that will break many things, i imagine | 19:15 |
| greghaynes | clarkb: there is a specific issue here that stdlib has | 19:15 |
| mordred | clarkb: which is also addressed by a pep targetted at python3.3 and later | 19:16 |
| harlowja | SpamapS ok, so oslo.utils is trying to stay mostly minimalish | 19:16 |
| clarkb | greghaynes: mordred can somone educate me then? | 19:16 |
| mordred | clarkb: https://www.python.org/dev/peps/pep-0418/ | 19:16 |
| harlowja | https://github.com/openstack/oslo.utils/blob/master/requirements.txt#L5 | 19:16 |
| harlowja | see ^ | 19:16 |
| SpamapS | harlowja: I think it definitely has value in the waiting loops. | 19:16 |
| clarkb | harlowja: thats not very minimal | 19:16 |
| greghaynes | https://review.openstack.org/#/c/176430/ is the patch | 19:16 |
| harlowja | clarkb ya, can't do much about that :-P | 19:16 |
| harlowja | SpamapS https://review.openstack.org/#/c/176439/ is probably also useful for shade stuff | 19:17 |
| clarkb | mordred: that pep doesn't help me | 19:17 |
| harlowja | since it looks like it has its own little task manager + future like thing | 19:17 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Reverse dependency relationship in project creation https://review.openstack.org/173485 | 19:17 |
| clarkb | under what circumstances does time.time go backwards | 19:17 |
| greghaynes | ntp | 19:17 |
| SpamapS | ugh, netifaces | 19:17 |
| clarkb | and under what circumstances is that actually a problem | 19:17 |
| clarkb | SpamapS: yup | 19:17 |
| mordred | clarkb: if you're calculating timeouts | 19:17 |
| greghaynes | time change of any kind, basically | 19:17 |
| mordred | clarkb: as we do | 19:17 |
| mordred | and the time changes for some reason | 19:17 |
| clarkb | mordred: ya if you do the math right it should still just work | 19:17 |
| mordred | clarkb: apparently there are cases when not | 19:18 |
| *** david-lyle has quit IRC | 19:18 | |
| harlowja | the stop watch class from oslo.utils that i sucked over from taskflow should handle the cases for u | 19:18 |
| clarkb | mordred: no there are cases time.time can go backwards | 19:18 |
| SpamapS | ntpd, btw, is not allowed to make the clock go backwards. | 19:18 |
| harlowja | but up to u guys :-P | 19:18 |
| SpamapS | it _is_ allowed to make it stop. | 19:18 |
| clarkb | mordred: but that doesn't mean your timeout will fail | 19:18 |
| *** annegentle has joined #openstack-infra | 19:18 | |
| harlowja | https://news.ycombinator.com/item?id=4128208 (read at your own will, lol) | 19:18 |
| clarkb | mordred: your timeout may take more or less time | 19:18 |
| clarkb | mordred: but it should still timeout | 19:19 |
| SpamapS | So you can see the same second for a few milliseconds, but ntpd won't adjust time backward fast _at all_. | 19:19 |
| lifeless | SpamapS: uhm, what ? ntp disagrees with you | 19:19 |
| SpamapS | lifeless: NTP, or NTPD? | 19:19 |
| lifeless | ntpd | 19:19 |
| lifeless | 'This may on occasion cause the clock to be set backwards if the local clock time is more than 128 s in the future relative to the server. In some applications, this behavior may be unacceptable. If the -x option is included on the command line, the clock will never be stepped and only slew corrections will be used.' | 19:19 |
| clarkb | harlowja: right so my argument is that time going backwards isn't an issue | 19:19 |
| harlowja | up to u guys and how much u care | 19:19 |
| openstackgerrit | James Slagle proposed openstack-infra/reviewstats: Sync TripleO projects. https://review.openstack.org/176444 | 19:20 |
| SpamapS | lifeless: it's been a while since I slogged through that man page. Ugh. | 19:20 |
| clarkb | harlowja: usually you say time_now = time.time() ; then later if time_now + timeout > time.time() then raise timeout | 19:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add the openstackid-resources repo https://review.openstack.org/176334 | 19:20 |
| greghaynes | sure, so even more generally, the time is only used here in an output message | 19:20 |
| lifeless | SpamapS: I'm sorry :) | 19:20 |
| clarkb | harlowja: ^ will function if time goes back in time | 19:20 |
| greghaynes | so it doesnt matter in that sense | 19:20 |
| SpamapS | lifeless: assumption broken. Thanks. :) | 19:20 |
| greghaynes | even if its negative | 19:20 |
| freerunner | Hi there! Folks, review please https://review.openstack.org/#/c/176398/ . | 19:20 |
| clarkb | harlowja: that is not worth a super heavyweight dep imo | 19:20 |
| harlowja | ok, feel free to move that stopwatch stuff to another pypi library if u want :-P | 19:21 |
| harlowja | i just put it there, but it could be anywhere | 19:21 |
| clarkb | harlowja: no I am saying just don't worry about it | 19:21 |
| clarkb | you don't need a lib | 19:21 |
| clarkb | just need smart code | 19:21 |
| greghaynes | well, thats the point of libs in general | 19:21 |
| harlowja | i prefer smart lib, and dumb code, lol | 19:21 |
| harlowja | but to each there own :-P | 19:22 |
| greghaynes | but yes, its probably overkill here | 19:22 |
| SpamapS | http://doc.ntp.org/4.2.6/ntpd.html is quite good btw. :) | 19:22 |
| *** achanda has quit IRC | 19:22 | |
| *** hasharAway is now known as hashar | 19:23 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/elastic-recheck: Generalize fingerprint for 1446847 https://review.openstack.org/176441 | 19:24 |
| *** jaypipes has quit IRC | 19:24 | |
| SpamapS | clarkb: to harlowja's point, This is a generic problem that can be gotten wrong. | 19:24 |
| SpamapS | clarkb: also in your example, the timeout is not waited for if time goes backward. | 19:25 |
| SpamapS | clarkb: which means a clock correction can lead to spurious errors. | 19:25 |
| clarkb | SpamapS: yes you may timeout early | 19:25 |
| clarkb | SpamapS: but thats a non issue | 19:25 |
| sdague | did we lose a cloud again? | 19:26 |
| clarkb | SpamapS: because your code should already be time out tolerant | 19:26 |
| clarkb | SpamapS: if it isnt then *that* is the bug | 19:26 |
| clarkb | sdague: I turned off hpcloud again yesterday as its presence was making rax less performant | 19:26 |
| SpamapS | I think I could argue that spurious errors are an issue. But this is a seldom-hit issue. :) | 19:26 |
| *** ddieterly has joined #openstack-infra | 19:26 | |
| sdague | clarkb: hmmm... ok? | 19:27 |
| clarkb | sdague: basically hpcloud wasn't booting nodes, but nodepool would still schedule half its workload to it | 19:27 |
| sdague | so now we're just in backlog mode | 19:27 |
| sdague | ok | 19:27 |
| clarkb | sdague: so we were running at half rax capacity + 0 hpcloud | 19:28 |
| clarkb | sdague: by turning off hpcloud we get 100% rax | 19:28 |
| SpamapS | clarkb: in shade's case, you're going to tolerate the timeout, but waste a lot of time and resources deleting and re-creating the server. BUT, I want to stress.. the error rate here, where nodepool's clock will go backwards, should be about once per ever. | 19:28 |
| sdague | I'm surprised to see us waiting > 1 hr for pep8 nodes | 19:28 |
| clarkb | SpamapS: yes exactly | 19:28 |
| mordred | SpamapS: yup. if we timeout a second early once ever it's fine | 19:28 |
| clarkb | SpamapS: and when it does happen everything will work fine | 19:28 |
| clarkb | even if it happend 100 times per day it would still be fine | 19:29 |
| harlowja | SpamapS do we really know how often time goes backwards in nodepool (or vm(s)); that'd be something neat to know, cause I thought vm's are especially affected by clock-shifts (due to hypervisor load...) | 19:29 |
| clarkb | harlowja: hypervisor load should only shift time forward right? | 19:29 |
| harlowja | busy neighbor, blah blah | 19:29 |
| harlowja | i'd hope so, but einstien and i haven't chatted recently | 19:29 |
| clarkb | if my cpu can't process the clock that doesn't make the clock go backwards | 19:30 |
| clarkb | but it may jump ahead when I get cpu cycles | 19:30 |
| harlowja | is this involving gravity and/or spacetime? | 19:30 |
| mordred | clarkb: dunno man. don't forget systemd exists | 19:30 |
| *** sandywalsh has quit IRC | 19:30 | |
| *** sushilkm has joined #openstack-infra | 19:30 | |
| mordred | clarkb: I can no longer predict anything that happens on a linux system, because systemd may just decide to, youknow, lennart something | 19:30 |
| *** sushilkm has left #openstack-infra | 19:30 | |
| SpamapS | I like how systemd is basically the new thing that can cause anything to happen. | 19:30 |
| SpamapS | harlowja: As clarkb says, vms are known for skipping seconds, but not for repeating them. | 19:31 |
| *** dizquierdo has quit IRC | 19:31 | |
| *** sandywalsh has joined #openstack-infra | 19:31 | |
| *** achanda has joined #openstack-infra | 19:31 | |
| SpamapS | harlowja: It's a corner case. A real inconsistency, that is interesting, but I think not a priority. | 19:31 |
| harlowja | sure, i hope so, i'm not really sure since never ran much extensive testing of this | 19:31 |
| clarkb | sdague: I think some of that is the gerrit event stream died again too | 19:32 |
| harlowja | SpamapS sure; take the review or not, either is fine with me :-P | 19:32 |
| devananda | clock can go backwards if ntpd screws up, or is misconfigured | 19:32 |
| harlowja | i just work here, lol | 19:32 |
| SpamapS | It goes up there with race conditions as something we'd love to have a real way to test, but since the matrix of inputs is every second that a test runs for... | 19:32 |
| devananda | though ntpd _should_ merely slow a clock down (make seconds slightly longer) when it detects that it is skewed ahead | 19:32 |
| SpamapS | devananda: you're a little late, we established that it's an unlikely problem to have. :) | 19:33 |
| *** sarob has joined #openstack-infra | 19:33 | |
| devananda | SpamapS: cool. yah, i haven't read all the backscroll | 19:33 |
| sdague | clarkb: ah | 19:33 |
| SpamapS | not impossible, just unlikely. | 19:33 |
| sdague | is that going to recover, or are we in lost job land? | 19:33 |
| SpamapS | And TIL that ntpd _is_ allowed to go backwards, just... very unlikely to do so. | 19:33 |
| *** ayoung has joined #openstack-infra | 19:34 | |
| clarkb | sdague: any jobs that did start should be fine | 19:34 |
| clarkb | sdague: s/start/queue/ | 19:34 |
| clarkb | sdague: we may have missed starting jobs for changes during the time gerrit wasn't emitting events, I think likely many got rechecked resulting in the backlog | 19:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: beaker: enable voting for puppet-(keystone/openstacklib) https://review.openstack.org/176343 | 19:35 |
| *** ddieterly has quit IRC | 19:36 | |
| *** jamesmcarthur has left #openstack-infra | 19:39 | |
| *** samuelBartel has quit IRC | 19:39 | |
| *** shakamunyi has joined #openstack-infra | 19:40 | |
| clarkb | sdague: I have confirmed queued jobs are still starting, so we haven't stopped the system again for some reason | 19:40 |
| sdague | clarkb: ok, no worries | 19:41 |
| *** david-lyle has joined #openstack-infra | 19:41 | |
| clarkb | mordred: greghaynes in 168603 why do we keep a global openstack cloud client? | 19:42 |
| clarkb | mordred: greghaynes that means we will use one client per test process which may leak state | 19:42 |
| mordred | clarkb: one sec | 19:43 |
| clarkb | oh hrm I guess the old code did the same | 19:43 |
| clarkb | so meh | 19:43 |
| mordred | clarkb: yah. | 19:43 |
| *** mmedvede has quit IRC | 19:43 | |
| greghaynes | clarkb: Yes, its a fake - they need to share state | 19:43 |
| mordred | clarkb: minimal changes | 19:43 |
| clarkb | greghaynes: you only need shared state per test not per test runner | 19:43 |
| clarkb | but this is how the old code worked so meh | 19:43 |
| *** sarob_ has joined #openstack-infra | 19:44 | |
| *** stevemar has joined #openstack-infra | 19:44 | |
| *** akshai has joined #openstack-infra | 19:45 | |
| *** ddieterly has joined #openstack-infra | 19:45 | |
| greghaynes | clarkb: yea, not sure how to punch that through though? I guess in the test cleanup reset the client? | 19:46 |
| clarkb | greghaynes: a | 19:46 |
| clarkb | but not a problem I just didn't realize the old code did this too until I had scrolled further | 19:46 |
| greghaynes | heh | 19:47 |
| greghaynes | well, I dont mind fixing it | 19:47 |
| greghaynes | ill do it in followup | 19:47 |
| *** sarob has quit IRC | 19:47 | |
| *** sarob_ is now known as sarob | 19:47 | |
| openstackgerrit | Merged openstack-infra/shade: Test flavor cache and add invalidation https://review.openstack.org/174707 | 19:47 |
| openstackgerrit | Merged openstack-infra/shade: Add test for caching in list_images https://review.openstack.org/174717 | 19:47 |
| *** mrunge has joined #openstack-infra | 19:48 | |
| *** mrunge has joined #openstack-infra | 19:48 | |
| *** mrmartin has joined #openstack-infra | 19:48 | |
| *** ZZelle_ has joined #openstack-infra | 19:49 | |
| *** ddieterly has quit IRC | 19:50 | |
| *** dkranz has quit IRC | 19:50 | |
| *** akshai has quit IRC | 19:50 | |
| EmilienM | AJaeger: so you think doc-team mays be also interested by the backport-potential commit flag ? | 19:51 |
| *** annegentle has quit IRC | 19:52 | |
| *** ociuhandu has quit IRC | 19:52 | |
| *** dangers_away is now known as dangers | 19:52 | |
| openstackgerrit | Merged openstack-infra/shade: Explicitly request cloud name in test_caching https://review.openstack.org/175698 | 19:52 |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Add update_machine method https://review.openstack.org/173996 | 19:53 |
| *** openstackgerrit has quit IRC | 19:54 | |
| *** openstackgerrit has joined #openstack-infra | 19:55 | |
| *** mmedvede has joined #openstack-infra | 19:55 | |
| AJaeger_ | EmilienM: Yeah, see https://wiki.openstack.org/wiki/Documentation/HowTo#Git_commit_messages_and_backports | 19:56 |
| EmilienM | AJaeger_: cool. Though jogo is bring some thoughts about the 'automatic' thing, that will bring conflict issues... | 19:56 |
| jogo | EmilienM: yeah, cherry-picks need humans eyes | 19:57 |
| jogo | we already have a process for this I thaught | 19:57 |
| *** ibiris is now known as ibiris_away | 19:57 | |
| EmilienM | jogo: most of the puppet patches we backport are without conflict | 19:57 |
| jogo | tagging bugs in launchpad | 19:57 |
| EmilienM | +1 for LP | 19:58 |
| EmilienM | AJaeger_: do you think we can go ahead with https://review.openstack.org/#/c/176070/ ? | 19:58 |
| EmilienM | jogo: what do you propose then? | 19:58 |
| jogo | EmilienM: just using LP | 19:58 |
| AJaeger_ | jogo, EmilienM for openstack-manuals we have a few directories where we backport nearly everything - and there such an automatic tool would really be great | 19:58 |
| *** radez is now known as radez_g0n3 | 19:58 | |
| jogo | and having people go over the list of things marked as backport | 19:59 |
| jogo | it wouldn't be hard to make a tool to query LP for things to backport | 19:59 |
| AJaeger_ | if you backport 3 patches in a different order than they were original, you might have conflicts | 19:59 |
| *** subscope_ has quit IRC | 19:59 | |
| jogo | and try to do so locally so a human can resolve conflicts manually | 19:59 |
| *** Rockyg has joined #openstack-infra | 19:59 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add test for create_image with glance v1 https://review.openstack.org/175699 | 19:59 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: Add tests and invalidation for glance v2 upload https://review.openstack.org/176024 | 19:59 |
| jogo | AJaeger_ EmilienM: so I propose tools that help humans do this faster | 20:00 |
| clarkb | mordred: greghaynes one small thing on 168603 | 20:00 |
| SpamapS | greghaynes: ^ issues fixed | 20:00 |
| SpamapS | mordred: ^ | 20:00 |
| jogo | not automated 100% | 20:00 |
| AJaeger_ | EmilienM: 176070 has a very long line... | 20:00 |
| EmilienM | AJaeger_: is it an issue? | 20:00 |
| openstackgerrit | Michael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.11.0 https://review.openstack.org/173449 | 20:00 |
| AJaeger_ | jogo: we have a script that does this manually - just pass it the review and it backports and let's you review manually... | 20:00 |
| mordred | SpamapS: woot | 20:01 |
| mordred | clarkb: on it | 20:01 |
| AJaeger_ | EmilienM: no, I'm not happy but I'll approve ;) | 20:01 |
| jogo | AJaeger_: 'git review -X' does that | 20:01 |
| *** dimtruck is now known as zz_dimtruck | 20:01 | |
| jogo | AJaeger_: but you can hook that up to launchpad to find the list of things to backport as well | 20:01 |
| AJaeger_ | jogo: yes, the script uses git review -X | 20:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add third party CI working group tools repository https://review.openstack.org/175520 | 20:02 |
| AJaeger_ | jogo: http://paste.openstack.org/show/205137/ | 20:02 |
| AJaeger_ | jogo, that's what we basically use - hooking into launchpad sounds nice | 20:03 |
| fungi | AJaeger_: i'm working on the new patch to attempt to address your consistency concern with conflation of "project" vs "repository" in the infra manual. it's daunting. you'll see in a bit if i manage to finish it | 20:03 |
| mordred | clarkb, greghaynes: I think we should do both - we should have one cloud with cloud and no auth info, one with cloud and auth info, and one with no cloud ... and we should test that the values you get back via shade/os-client-config match our expectations in each case | 20:04 |
| jogo | AJaeger_: thankfully LP has a decent REST API | 20:04 |
| clarkb | fungi: 171286 has my +2 and I rechecked it to get good test results post pycparser fix | 20:04 |
| *** tqtran is now known as tqtran_afk | 20:04 | |
| clarkb | mordred: +1 | 20:04 |
| *** otter768 has joined #openstack-infra | 20:04 | |
| AJaeger_ | fungi: Wow! I agree with you that this should be separate patch - that's why I didn't -1 jeblair's patch. | 20:04 |
| openstackgerrit | Joshua Harlow proposed openstack-infra/shade: Have the task class actually wait for results to be set https://review.openstack.org/176459 | 20:04 |
| greghaynes | mordred: sounds good | 20:04 |
| jogo | greghaynes: what ever happened to the wheel mirror? | 20:05 |
| greghaynes | jogo: sitting for reviews forever | 20:05 |
| greghaynes | still there | 20:05 |
| greghaynes | had to rebase a bit ago | 20:05 |
| clarkb | I reviewed it :) | 20:05 |
| *** ildikov has quit IRC | 20:05 | |
| AJaeger_ | fungi: looking forward to that, thanks! | 20:05 |
| fungi | clarkb: thanks! | 20:05 |
| jogo | fungi: want to review it too? | 20:05 |
| *** patrickeast has joined #openstack-infra | 20:06 | |
| greghaynes | it being: https://review.openstack.org/#/q/status:open+branch:master+topic:feature/wheel-mirror,n,z | 20:06 |
| fungi | jogo: review what? | 20:06 |
| openstackgerrit | Merged openstack-infra/system-config: New nodes should install puppet from install_puppet.sh https://review.openstack.org/172936 | 20:07 |
| jogo | fungi: greghaynes wheel mirror of goodness | 20:07 |
| *** shashankhegde has joined #openstack-infra | 20:07 | |
| fungi | oh, yes i do want to review it. let's see if i manage to find time to review it | 20:07 |
| *** zz_dimtruck is now known as dimtruck | 20:08 | |
| *** dangers is now known as dangers_away | 20:08 | |
| *** dangers_away is now known as dangers | 20:08 | |
| jogo | fungi: thanks | 20:09 |
| *** otter768 has quit IRC | 20:09 | |
| mrmartin | hi | 20:09 |
| clarkb | mrmartin: good evening | 20:10 |
| mrmartin | fungi, I've question on this askbot-staging. So I had a quite nice progress today with the puppet scripts | 20:10 |
| mrmartin | but how could we handle the situation when the puppet-askbot manifests breaks the compatibility with previous version, and we still need to operate the ask.o.o based on existing ones? | 20:11 |
| mrmartin | can we pinpoint ask.o.o's puppet module to the actual version somehow? | 20:11 |
| *** sputnik13 has joined #openstack-infra | 20:11 | |
| mrmartin | or just simply shutdown the puppet on that node until resolve all dependency issues? | 20:11 |
| *** e0ne has quit IRC | 20:12 | |
| *** achanda has quit IRC | 20:12 | |
| *** teran has quit IRC | 20:13 | |
| clarkb | mrmartin: we cannot, we would need to be backward and forward compatible | 20:14 |
| mrmartin | great | 20:14 |
| *** sabeen3 has quit IRC | 20:14 | |
| clarkb | mrmartin: it seems to work for the most part at least with jenkins-dev, etherpad-dev, review-dev and so on | 20:14 |
| *** agentle_ has joined #openstack-infra | 20:14 | |
| *** annegentle has joined #openstack-infra | 20:14 | |
| clarkb | enforces a bit of discipline that is good to have | 20:14 |
| *** fawadkhaliq has quit IRC | 20:15 | |
| mrmartin | I have a major issue with the deployment model, actually we are deploying from pip which brings in all of the requirements | 20:15 |
| *** sabeen has joined #openstack-infra | 20:15 | |
| mrmartin | dependencies | 20:15 |
| mrmartin | but if I checking out the project directly from github repo, what is the proper way to pull the missing deps? a, deploy the packages from ubuntu repo b, do a pip install -r requirements.txt ? | 20:16 |
| clarkb | mrmartin: if you want greater control of the dependencies you can install them all first then install askbot | 20:16 |
| clarkb | mrmartin: pip install /path/to/project should work | 20:16 |
| mrmartin | so typically, if a new askbot version brings in a different dependency in requirements.txt, the deploy can brake too | 20:17 |
| openstackgerrit | Merged openstack-infra/project-config: Add puppet stackforge modules to tripleo jobs https://review.openstack.org/176070 | 20:17 |
| *** HeOS has quit IRC | 20:18 | |
| clarkb | mrmartin: yes | 20:19 |
| clarkb | mrmartin: but we want to know that so I am not sure we would change it? | 20:20 |
| *** ildikov has joined #openstack-infra | 20:20 | |
| mrmartin | ok, deployment into a virtualenv can solve this new dependency problem? | 20:20 |
| clarkb | mrmartin: no, because the deps can still chnge within the virtualenv and break the deployment | 20:20 |
| clarkb | mrmartin: the virtualenv can isolate it, but not prevent it | 20:21 |
| mrmartin | and here comes the blue-green model :D | 20:21 |
| mrmartin | deploying into different venv slots? | 20:21 |
| clarkb | mrmartin: so I don't think this is a problem | 20:21 |
| clarkb | mrmartin: if the staging server breaks that is why it exists | 20:21 |
| mrmartin | ok. | 20:21 |
| mrmartin | I try to not over-engineer it | 20:21 |
| clarkb | mrmartin: then we only update the production server at known good points in staging | 20:21 |
| mrmartin | I guess some deps already missing from requirements.txt | 20:22 |
| clarkb | fun | 20:22 |
| mrmartin | :) | 20:22 |
| mrmartin | ok, great, thnx, I'll play with that tomorrow | 20:23 |
| clarkb | heads up system-config core, I think there are at least two bugs in the apply test that may make our apply integration test not quite an integration test | 20:23 |
| clarkb | I need to wrap my head around it but will share shortly | 20:24 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: networking-nec: add ability to create and manage branches https://review.openstack.org/176466 | 20:24 |
| clarkb | (but be careful approving module changes) | 20:24 |
| *** rlandy has quit IRC | 20:25 | |
| *** ldnunes_ has joined #openstack-infra | 20:26 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Reverse dependency relationship in project creation https://review.openstack.org/173485 | 20:27 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Project does not mean repository? Inconceivable! https://review.openstack.org/176469 | 20:27 |
| fungi | AJaeger_: ^ | 20:27 |
| AJaeger_ | fungi: nice subject ;) | 20:28 |
| fungi | AJaeger_: i'm not personally convinced that change makes the guides more readable, but it's at least a suitable straw-man for discussion | 20:28 |
| *** annegentle has quit IRC | 20:28 | |
| *** agentle_ has quit IRC | 20:28 | |
| clarkb | ok one bug is not a bug thats good | 20:29 |
| fungi | problem is that even though the tc has decreed "project" means something different now, most of our tools and basically all our config files still use "project" to mean "repository" | 20:29 |
| *** annegentle has joined #openstack-infra | 20:29 | |
| clarkb | (I thought integration modules may have been overridden by install_modules.sh after zuul-cloner runs but puppetdb is not in integration modules and the integration modules logic looks correct) | 20:29 |
| fungi | this is why changing the meaning of terms in use is so much more painful than picking new terms not yet in use | 20:29 |
| waynr | dang reviews are taking quite a while today | 20:30 |
| *** yamamoto has joined #openstack-infra | 20:31 | |
| fungi | waynr: yep, zuul has about 800 jobs waiting on available workers at the moment http://status.openstack.org/zuul/ | 20:31 |
| clarkb | the other issue is we can't have blank lines inside of a node def due to how our csplit works | 20:32 |
| fungi | waynr: assuming by reviews you mean check results | 20:32 |
| clarkb | so the openstack_project::server decoupling is tripping over that in a few places | 20:32 |
| *** sputnik13 has quit IRC | 20:32 | |
| waynr | yeah that's what i meant | 20:32 |
| clarkb | and this is going to require better parsing than a regex to fix | 20:32 |
| AJaeger_ | fungi, that's a significant change, I'll review tomorrow closely | 20:32 |
| *** sandywalsh has quit IRC | 20:33 | |
| clarkb | unless I make some other assumptions about the code which I acutally probably can maybe do | 20:33 |
| *** sandywalsh has joined #openstack-infra | 20:34 | |
| fungi | AJaeger_: i tried to clarify "project team" in some places and use "package" in other places where relevant, and use "repository" when talking about actual git repositories, but there are still many, many grey areas | 20:34 |
| *** sputnik13 has joined #openstack-infra | 20:34 | |
| AJaeger_ | fungi: yeah, noticed - and that's when I decided I need to review this slowly ;) | 20:35 |
| *** dannywilson has joined #openstack-infra | 20:35 | |
| AJaeger_ | It's not enough to review the change, this needs rewriting of the whole documents | 20:35 |
| *** yamamoto has quit IRC | 20:35 | |
| fungi | yes. yes it does | 20:36 |
| AJaeger_ | enhancing your commit message with the above comment about these three words would help others to understand this better | 20:36 |
| fungi | previously "project" potentially meant multiple things depending on the surrounding context | 20:36 |
| AJaeger_ | I meant "reading again" | 20:36 |
| fungi | can do | 20:37 |
| AJaeger_ | fungi: yeah, project had multiple meanings indeed. | 20:37 |
| AJaeger_ | thanks, fungi | 20:37 |
| *** sambetts has quit IRC | 20:38 | |
| *** sambetts has joined #openstack-infra | 20:39 | |
| *** stevemar has quit IRC | 20:41 | |
| *** rbradfor has quit IRC | 20:41 | |
| sdague | clarkb / fungi - https://review.openstack.org/#/c/175391/ could use review - this adds the stable/kilo branch support, and Depends-On a series of fixes we needed that were discovered by trying to do that the first time | 20:42 |
| *** hdd has quit IRC | 20:42 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-manual: Project does not mean repository? Inconceivable! https://review.openstack.org/176469 | 20:42 |
| fungi | AJaeger_: updated | 20:42 |
| *** dannywilson has quit IRC | 20:42 | |
| fungi | sdague: awesome. looking now | 20:43 |
| *** dannywilson has joined #openstack-infra | 20:43 | |
| AJaeger_ | fungi, great! | 20:43 |
| *** stevemar has joined #openstack-infra | 20:45 | |
| sdague | and we're holding requirements unfreeze until https://review.openstack.org/#/c/175391/ lands so that we're not doing juno -> liberty upgrades with new requirements | 20:45 |
| pleia2 | cinerama: meant to ask you the other day, did you ever create an issue in the zanata tracker for project creation stuff? (bugzilla makes my brain hurt, cannot find!) | 20:46 |
| *** ArxCruz has quit IRC | 20:46 | |
| *** pc_m has quit IRC | 20:46 | |
| *** mrmartin has quit IRC | 20:46 | |
| *** luqas has joined #openstack-infra | 20:47 | |
| cinerama | pleia2: oh to the effect that there's no configurability of roles so everyone can create a project? | 20:47 |
| pleia2 | cinerama: that's the one | 20:47 |
| pleia2 | saw mention of it when I was browsing the logs yesterday | 20:48 |
| cinerama | pleia2: no - shall i do that now or would you like to do the honours? | 20:48 |
| pleia2 | I'm honestly not sure what to do if we can't restrict that, just tell people to be good? :) | 20:48 |
| pleia2 | cinerama: you should go for it | 20:48 |
| cinerama | pleia2: i kind of got sidetracked dealing with some of the client issues | 20:48 |
| pleia2 | cinerama: understandable :) just trying to close loops here | 20:48 |
| *** ArxCruz has joined #openstack-infra | 20:49 | |
| *** AJaeger_ has quit IRC | 20:49 | |
| cinerama | pleia2: oh that's right, i have to create an account for that bugzilla and then file. that's probably how it went in my "ok, later" bin | 20:50 |
| pleia2 | hehe | 20:50 |
| pleia2 | it's so nice that the interface auto-generates your zanata.ini for the client <3 | 20:51 |
| *** luqas has quit IRC | 20:51 | |
| clarkb | sdague: will look in just a moment, I have a system-config bug fix I need to push up | 20:52 |
| *** achanda has joined #openstack-infra | 20:53 | |
| sdague | clarkb: no prob | 20:54 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Allow blank lines in node{} with apply test https://review.openstack.org/176482 | 20:54 |
| clarkb | fungi: pleia2 ^ that is the bug, its still not perfect but should be a bit more flexible now | 20:55 |
| clarkb | now I need to -1 some changes and have them rebase on htat | 20:55 |
| *** hdd has joined #openstack-infra | 20:55 | |
| *** dboik_ has quit IRC | 20:56 | |
| fungi | oh that's a fun one to have to track down | 20:57 |
| fungi | have we let any other bugs slip in because of that? | 20:58 |
| *** david-lyle_ has joined #openstack-infra | 20:58 | |
| fungi | or is the integration test working already with that patch? | 20:59 |
| clarkb | fungi: nothing is broken now but a bunch of proposed changes break without it | 20:59 |
| sdague | also, clarkb / fungi / pleia2 - https://review.openstack.org/#/c/167412 is part of moving trove support out of devstack and into it's own plugin | 20:59 |
| clarkb | fungi: I was debugging why jenkins -1'd some of those new changes which haven't merged and discovered this | 20:59 |
| *** david-lyle has quit IRC | 20:59 | |
| fungi | got it | 20:59 |
| *** adiantum has quit IRC | 20:59 | |
| clarkb | fungi: so I think we are ok for now and am -1'ing things suggestiong they rebase on that fix | 20:59 |
| pleia2 | clarkb: seems more perfect than splitting on blank lines | 20:59 |
| clarkb | pleia2: ya its still naieve but less so | 21:00 |
| * pleia2 nods | 21:00 | |
| *** mattfarina has quit IRC | 21:01 | |
| clarkb | I ran the string munging locally and it looks correct but I didn't run the full test fwiw | 21:01 |
| pleia2 | k, I'll give it a go | 21:01 |
| *** ldnunes_ has quit IRC | 21:01 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 21:02 |
| openstackgerrit | Merged openstack-infra/project-config: add project fuel-plugin-nova-nfs https://review.openstack.org/175452 | 21:02 |
| *** baoli has quit IRC | 21:03 | |
| *** tqtran has joined #openstack-infra | 21:05 | |
| cinerama | pleia2: i'll be here for the next hour or so & then i have to go pick up a family member, so if you have any other questions for me, best to ask now | 21:06 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: only force off KEYSTONE_USE_WSGI up until kilo https://review.openstack.org/175531 | 21:06 |
| cinerama | pleia2: in the meantime i have a few bugs to file with zanata | 21:06 |
| openstackgerrit | Merged openstack-infra/project-config: Move tripleo-puppet-elements to tripleo-core https://review.openstack.org/176225 | 21:06 |
| *** kgiusti has left #openstack-infra | 21:07 | |
| *** dizquierdo has joined #openstack-infra | 21:08 | |
| *** ivar-laz_ has joined #openstack-infra | 21:08 | |
| clarkb | sdague: I have approved 175391 though it may not gate right away due to the depends on | 21:08 |
| clarkb | sdague: happy to push it again later | 21:08 |
| sdague | clarkb: cool, thanks | 21:09 |
| *** tqtran has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Add diskimage-builder functests https://review.openstack.org/173585 | 21:11 |
| *** erikmwilson has quit IRC | 21:11 | |
| clarkb | greghaynes: ^ | 21:11 |
| *** ivar-lazzaro has quit IRC | 21:11 | |
| *** dboik has joined #openstack-infra | 21:12 | |
| *** teran has joined #openstack-infra | 21:13 | |
| *** dprince has quit IRC | 21:13 | |
| *** stevemar has quit IRC | 21:13 | |
| pleia2 | cinerama: noted, I'll let you know if I come across anything in the next hour :) | 21:15 |
| *** david-lyle_ has quit IRC | 21:15 | |
| *** david-lyle has joined #openstack-infra | 21:15 | |
| openstackgerrit | Merged openstack-infra/reviewstats: Sync TripleO projects. https://review.openstack.org/176444 | 21:16 |
| *** mrunge has quit IRC | 21:17 | |
| *** ldnunes has quit IRC | 21:17 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Generalize fingerprint for 1446847 https://review.openstack.org/176441 | 21:19 |
| *** gyee has quit IRC | 21:20 | |
| *** HeOS has joined #openstack-infra | 21:20 | |
| *** david-lyle has quit IRC | 21:21 | |
| *** sarob has quit IRC | 21:25 | |
| *** gyee has joined #openstack-infra | 21:26 | |
| lifeless | mordred: ping - https://bugs.launchpad.net/pbr/+bug/1446425 tchaypo and i discussed, unless you see something more, going to close | 21:28 |
| openstack | Launchpad bug 1446425 in PBR "pip only respects 1 version specifier for any particular name" [Undecided,New] | 21:28 |
| *** peristeri has quit IRC | 21:29 | |
| *** openstackgerrit has quit IRC | 21:29 | |
| mordred | lifeless: I defer to your judgemnt - my head is not deep in it | 21:29 |
| *** openstackgerrit has joined #openstack-infra | 21:30 | |
| *** sarob has joined #openstack-infra | 21:31 | |
| *** dboik has quit IRC | 21:31 | |
| *** tnovacik has quit IRC | 21:31 | |
| *** hdd has quit IRC | 21:32 | |
| *** Rockyg has quit IRC | 21:32 | |
| tchaypo | lifeless: one thing that occurred to me | 21:34 |
| *** yamahata has joined #openstack-infra | 21:34 | |
| tchaypo | if pbr is re-invoking pip to deal does that mean you’ll end up with (potentially) different things being installed than if pip was doing all the processing? | 21:35 |
| *** bswartz has quit IRC | 21:35 | |
| tchaypo | but i think the answer is no, because when pbr re-invokes pip it passes in all the dependencies it already has, from memory | 21:35 |
| lifeless | yes, which is a different symptom and problem | 21:35 |
| openstackgerrit | Joe Gordon proposed openstack-infra/system-config: Add subnode logs to logstash https://review.openstack.org/176499 | 21:35 |
| clarkb | pip bug 988 is going to be my most glorious bug opened | 21:35 |
| openstack | bug 988 in Launchpad itself "Malone result headers scrunched in Konqueror+Safari+Opera" [Medium,Fix released] https://launchpad.net/bugs/988 - Assigned to Matthew Paul Thomas (mpt) | 21:35 |
| tchaypo | in any case I think the correct solution there is to fix pip to be consistent rather than break pbr to match pip | 21:36 |
| tchaypo | clarkb: much nicer than that other bug 988 | 21:36 |
| *** dboik has joined #openstack-infra | 21:37 | |
| *** david-lyle has joined #openstack-infra | 21:37 | |
| mordred | clarkb: whats pip bug 988? | 21:38 |
| openstack | bug 988 in Launchpad itself "Malone result headers scrunched in Konqueror+Safari+Opera" [Medium,Fix released] https://launchpad.net/bugs/988 - Assigned to Matthew Paul Thomas (mpt) | 21:38 |
| lifeless | tchaypo: was that a strawman? I don't think anyone suggested breaking pbr... | 21:38 |
| *** dboik has quit IRC | 21:38 | |
| clarkb | mordred: `pip needs a proper dep resolver` | 21:38 |
| clarkb | mordred: I opened a few years ago | 21:38 |
| mordred | clarkb: it is a glorious bug | 21:38 |
| *** dboik has joined #openstack-infra | 21:38 | |
| tchaypo | lifeless: yes | 21:38 |
| clarkb | mordred: but it has become the nexus point of many python/pip related bugs | 21:38 |
| mordred | clarkb: you might wind up creating a black hole | 21:39 |
| *** dustins has quit IRC | 21:40 | |
| *** tonytan4ever has quit IRC | 21:41 | |
| *** sdake has quit IRC | 21:44 | |
| *** annegentle has quit IRC | 21:45 | |
| *** ivar-laz_ has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Setup publishing to pypi for xstatic-* https://review.openstack.org/169274 | 21:45 |
| *** ivar-lazzaro has joined #openstack-infra | 21:45 | |
| *** yamahata has quit IRC | 21:46 | |
| *** dprince has joined #openstack-infra | 21:46 | |
| *** bknudson has quit IRC | 21:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Add stable acls for python-glanceclient https://review.openstack.org/173748 | 21:46 |
| jogo | lifeless: https://pip.readthedocs.org/en/stable/user_guide.html#id21 | 21:48 |
| jogo | err https://pip.readthedocs.org/en/stable/user_guide.html#ensuring-repeatability | 21:48 |
| lifeless | jogo: ECONTEXT? | 21:49 |
| jogo | lifeless: pips recommendation for managing packages and ensuring repeatable installs | 21:49 |
| jogo | lifeless: it says to use --no-deps, and even better it says to not use pip (and use peep?) | 21:50 |
| lifeless | so thats a little stale | 21:50 |
| lifeless | I also have a quarrel with it | 21:51 |
| lifeless | freeze lists everything bar pip and setuptools | 21:51 |
| lifeless | with == | 21:51 |
| lifeless | so, having things randomly shift around with or without --no-deps is at best a bug | 21:51 |
| lifeless | I'd want an example to poke at | 21:51 |
| jogo | lifeless: good point | 21:52 |
| lifeless | the last point is false | 21:52 |
| lifeless | because pypi has been fixed | 21:52 |
| lifeless | no ninja-replacing-of-objects-on-pypi anymore | 21:52 |
| jogo | lifeless: that is what I thought too but the docs are wrong for pip I guess | 21:52 |
| jogo | anyway since you are digging into this stuff now wanted to share the link with you | 21:52 |
| lifeless | thanks | 21:53 |
| lifeless | yeah - good, just was like so many balls in air, not sure which one you refered to :) | 21:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 21:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove REST links from inventory metadata https://review.openstack.org/175903 | 21:53 |
| openstackgerrit | Merged openstack-infra/project-config: Remove stable-compat-jobs for projects that have stable branchs https://review.openstack.org/173478 | 21:54 |
| *** bhuvan has quit IRC | 21:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Add glean to infra https://review.openstack.org/172749 | 21:54 |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Make default from address mandatory https://review.openstack.org/176393 | 21:54 |
| *** dboik has quit IRC | 21:55 | |
| *** dboik has joined #openstack-infra | 21:55 | |
| *** cdent has quit IRC | 21:56 | |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Add cachetools requirement https://review.openstack.org/152953 | 21:57 |
| jogo | lifeless: at least they aren't swords | 21:57 |
| *** dprince has quit IRC | 21:58 | |
| *** ivar-laz_ has joined #openstack-infra | 21:58 | |
| *** aduarte has quit IRC | 22:00 | |
| *** ivar-lazzaro has quit IRC | 22:01 | |
| *** tqtran_afk is now known as tqtran | 22:01 | |
| *** Swami has quit IRC | 22:04 | |
| lifeless | jogo: heh. | 22:04 |
| lifeless | jogo: | 22:04 |
| lifeless | https://github.com/pypa/pip/pull/2713 | 22:04 |
| *** andreykurilin__ has joined #openstack-infra | 22:05 | |
| *** otter768 has joined #openstack-infra | 22:05 | |
| *** bknudson has joined #openstack-infra | 22:05 | |
| *** esker has quit IRC | 22:07 | |
| *** otter768 has quit IRC | 22:09 | |
| *** signed8b_ has joined #openstack-infra | 22:09 | |
| jogo | nice! | 22:10 |
| *** openstackgerrit has quit IRC | 22:11 | |
| *** openstackgerrit has joined #openstack-infra | 22:11 | |
| clarkb | jenkins04 appears dead | 22:12 |
| *** signed8bit has quit IRC | 22:12 | |
| clarkb | the process is up but not responding to my browser, likely the thread leak again | 22:13 |
| *** dboik has quit IRC | 22:14 | |
| clarkb | the other masters look fine. Working to get a thread count on 04 now | 22:14 |
| tchaypo | dstufft is on the ball | 22:15 |
| clarkb | 7,190 threads | 22:15 |
| clarkb | jenkins that is not healthy stop | 22:15 |
| clarkb | putting it in shutdown mode if I can | 22:16 |
| clarkb | I am also cleaning up ~10 stale nodes on rax that will bump our usable quota by ~10 nodes | 22:17 |
| anteaya | clarkb: is that the same leaking thread issue that we have had for a while? | 22:18 |
| *** shashankhegde has quit IRC | 22:18 | |
| clarkb | anteaya: ya | 22:19 |
| *** ivar-laz_ has quit IRC | 22:19 | |
| clarkb | got 04 into shutdown mode, will likely leave it there then dig out the java gloves tomorrow and see if I can fix it | 22:19 |
| anteaya | and your upgrade didn't solve the issue :( | 22:19 |
| *** ivar-lazzaro has joined #openstack-infra | 22:19 | |
| clarkb | anteaya: it did not | 22:19 |
| clarkb | anteaya: we have observed the bug on both versions of the ssh-slaves plugin | 22:19 |
| anteaya | are java gloves made with dragon hide? | 22:19 |
| anteaya | boo | 22:19 |
| *** jaypipes has joined #openstack-infra | 22:20 | |
| *** david-lyle_ has joined #openstack-infra | 22:20 | |
| *** david-lyle has quit IRC | 22:20 | |
| *** shashankhegde has joined #openstack-infra | 22:21 | |
| *** spzala has quit IRC | 22:21 | |
| *** erlon has quit IRC | 22:21 | |
| *** annegentle has joined #openstack-infra | 22:22 | |
| *** agentle_ has joined #openstack-infra | 22:22 | |
| zaro | fungi: i have verified that deploying jeblair's gerrit change on review-dev works without error: https://review.openstack.org/176523 | 22:23 |
| *** ArxCruz has quit IRC | 22:23 | |
| *** dkranz has joined #openstack-infra | 22:24 | |
| anteaya | zaro: what do you have running on gerrit-dev at the moment? | 22:24 |
| *** ddieterly has joined #openstack-infra | 22:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Update wildfly puppet module version https://review.openstack.org/176409 | 22:26 |
| *** hogepodge has quit IRC | 22:27 | |
| *** hashar has quit IRC | 22:27 | |
| zaro | anteaya: i assume you mean review-dev, it's running whatever is on tip of openstack/2.10.2 branch + https://review.openstack.org/176409 | 22:28 |
| *** ddieterl_ has joined #openstack-infra | 22:28 | |
| *** ddieterly has quit IRC | 22:29 | |
| anteaya | I do mean review-dev, sorry | 22:29 |
| anteaya | zaro: that is a patch for an open wildfly module | 22:30 |
| anteaya | are you saying that is running on review-dev? | 22:31 |
| *** sdake has joined #openstack-infra | 22:31 | |
| zaro | opps, wrong #, it's actaully https://review.openstack.org/176523 and yes review-dev | 22:31 |
| *** fbo has quit IRC | 22:31 | |
| anteaya | ah yes, thanks that makes sense | 22:32 |
| *** hogepodge has joined #openstack-infra | 22:32 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Set EnablePacemaker for the HA job https://review.openstack.org/176286 | 22:37 |
| *** fbo has joined #openstack-infra | 22:37 | |
| *** pradk has joined #openstack-infra | 22:41 | |
| *** pradk has quit IRC | 22:41 | |
| *** dkranz has quit IRC | 22:43 | |
| *** bhuvan has joined #openstack-infra | 22:44 | |
| *** amitgandhinz has quit IRC | 22:45 | |
| *** agentle_ has quit IRC | 22:45 | |
| *** annegentle has quit IRC | 22:45 | |
| *** Swami has joined #openstack-infra | 22:46 | |
| *** shakamunyi has quit IRC | 22:46 | |
| *** gordc has quit IRC | 22:47 | |
| *** dimtruck is now known as zz_dimtruck | 22:48 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/release-tools: Include 'source_url' in notes and use readme parser function https://review.openstack.org/176528 | 22:48 |
| *** apmelton is now known as apmelton_ | 22:50 | |
| *** apmelton_ is now known as apmelton_away | 22:50 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Refactored event worker plugin structure into plugin directory. https://review.openstack.org/176530 | 22:51 |
| *** nelsnelson has quit IRC | 22:51 | |
| *** Swami has quit IRC | 22:53 | |
| *** ZZelle_ has quit IRC | 22:54 | |
| *** david-lyle_ has quit IRC | 22:57 | |
| *** melwitt has quit IRC | 22:57 | |
| *** dizquierdo has quit IRC | 22:57 | |
| *** Rockyg has joined #openstack-infra | 22:58 | |
| jhesketh | Morning | 22:59 |
| *** andreykurilin__ has quit IRC | 22:59 | |
| clarkb | good morning | 23:00 |
| *** ChuckC has quit IRC | 23:00 | |
| *** stevemar has joined #openstack-infra | 23:01 | |
| anteaya | morning jhesketh | 23:01 |
| anteaya | I'm being diconnected again from my etherpad in firefox | 23:01 |
| jhesketh | o/ | 23:02 |
| anteaya | I have worked for several hours without being disconnected and I just got disconnected twice in about 5 minutes | 23:02 |
| anteaya | it seems to come in waves but I don't know what the trigger is | 23:02 |
| *** sabeen has quit IRC | 23:04 | |
| *** tjones2 has joined #openstack-infra | 23:04 | |
| *** tjones1 has quit IRC | 23:05 | |
| *** spzala has joined #openstack-infra | 23:06 | |
| *** jd__ has quit IRC | 23:08 | |
| *** jd__ has joined #openstack-infra | 23:08 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 23:08 | |
| *** doug-fish has left #openstack-infra | 23:08 | |
| *** eharney has quit IRC | 23:09 | |
| *** bswartz has joined #openstack-infra | 23:09 | |
| *** yamamoto has joined #openstack-infra | 23:10 | |
| *** ajo has quit IRC | 23:10 | |
| anteaya | I will also note that etherpad in chromium is able to deal with a multiline copy/paste and format it the way it was pasted rather than in firefox which if I paste into a bullet point puts bullets at the beginning of every new line even if it is one passage | 23:13 |
| *** thedodd has quit IRC | 23:13 | |
| *** yamamoto has quit IRC | 23:14 | |
| clarkb | anteaya: did it work in firefox before the upgrade? | 23:15 |
| clarkb | or is chromium half fixing an existing bug? | 23:15 |
| *** fawadkhaliq has joined #openstack-infra | 23:15 | |
| anteaya | firefox worked fine as far as I can remember prior to about 2 days ago | 23:17 |
| anteaya | I never really worked in etherpad in chromium before yesterday as I am just used to firefox and use it without thinking about it | 23:17 |
| anteaya | but today I am thinking about it | 23:17 |
| clarkb | gotcha | 23:18 |
| anteaya | so I don't know what chromium did with regard to formating prior to today, nor for taht matter had I spent much time copy pasting and using bullets in etherpad in firefox | 23:18 |
| anteaya | I'm doing a nova-net to neutron ml round up | 23:18 |
| anteaya | much to copy paste | 23:18 |
| anteaya | and for this work chromium, whatever it is doing, it proving itself to be less work for me to get what I want | 23:20 |
| *** fawadkhaliq has quit IRC | 23:20 | |
| *** ChuckC has joined #openstack-infra | 23:21 | |
| clarkb | mordred: SpamapS Shrews comment on 175699 about general shade behavior that you may find interesting | 23:22 |
| clarkb | anteaya: I think as a general rule chrom* has the largest user share so many applications work better with it | 23:23 |
| clarkb | anteaya: sort of like how IE6 was "best" for compatibility for so long | 23:23 |
| *** david-lyle has joined #openstack-infra | 23:24 | |
| clarkb | mordred: and for 175903 does links come from the cloud? if so wouldn't it be appropriate to let users decide if they don't want that information? I don't know what the links metadata is fwiw | 23:24 |
| *** ChuckC has quit IRC | 23:25 | |
| *** Rockyg has quit IRC | 23:25 | |
| *** ChuckC has joined #openstack-infra | 23:25 | |
| anteaya | that would make sense | 23:26 |
| *** markvoelker has quit IRC | 23:26 | |
| *** hichihara has joined #openstack-infra | 23:26 | |
| *** mriedem is now known as mriedem_away | 23:26 | |
| anteaya | and I have been disconnected from the etherpad in firefox 3 more times but yet to be disconnected in chrome | 23:26 |
| *** amotoki has quit IRC | 23:27 | |
| clarkb | mordred: found it in the api docs | 23:27 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:27 | |
| clarkb | mordred: so its a link to that resource, but shade already knows how toget that resource | 23:27 |
| clarkb | mordred: so its not useful? | 23:28 |
| *** sputnik13 has quit IRC | 23:29 | |
| greghaynes | clarkb: thats why my reading of the API indicates | 23:30 |
| *** ddieterl_ has quit IRC | 23:31 | |
| mordred | clarkb: it's not useful in the inventory | 23:35 |
| mordred | clarkb: it's useful if it's part of the payload to do further rest calls | 23:35 |
| mordred | clarkb: also - yeah list_images() should return a [] - thank you nice catch | 23:36 |
| mordred | clarkb, SpamapS, Shrews: also - I just rewrote that inventory patch (sorry, should have wip'd the one in gerrit) - new upload coming | 23:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Every worker now gets its own session. https://review.openstack.org/176545 | 23:36 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Pre-resolve author record for events. https://review.openstack.org/176546 | 23:36 |
| mordred | clarkb: I think I just found a MUCH nicer way to do it in shade that will let the ansible-specific things be in ansible and the re-usable things be in shade without making either one too weird | 23:37 |
| mordred | (Also, in so doing, found two other bugs - so it was worth hacking on regardless) | 23:37 |
| *** sdake has quit IRC | 23:39 | |
| *** dannywilson has quit IRC | 23:39 | |
| *** claudiub has quit IRC | 23:42 | |
| *** Longgeek has quit IRC | 23:42 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add inventory command to shade https://review.openstack.org/175891 | 23:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove REST links from inventory metadata https://review.openstack.org/175903 | 23:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Have get_image_name return an image_name https://review.openstack.org/176549 | 23:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Return an empty list on volume_list https://review.openstack.org/176550 | 23:43 |
| mordred | greghaynes: ^^ you'll enjoy 176549 | 23:43 |
| mordred | Shrews, SpamapS: we need to circle back on the behavior on volume_list I think | 23:44 |
| mordred | because right now it's just passing through cinder - which may not exist for the user in the cloud - this is one where we need to see if the user has cinder, if not, try nova | 23:45 |
| greghaynes | mordred: TIL, image is not name | 23:45 |
| greghaynes | er, id is not name | 23:45 |
| mordred | :) | 23:45 |
| SpamapS | mordred: Oh, yeah, thats entirely true. I just think we haven't run into a cloud like that yet. | 23:46 |
| SpamapS | though, is nova-volume as prevalent as nova-network ? | 23:46 |
| greghaynes | mordred: what is with the WIP on https://review.openstack.org/#/c/168634/ ? | 23:46 |
| SpamapS | I thought cinder stomped it out of existence rather quickly. | 23:46 |
| mordred | SpamapS: well.... | 23:47 |
| mordred | SpamapS: thing is - vexxhost and RunAbove do not have cinder at all | 23:47 |
| cinerama | pleia2: back now | 23:47 |
| mordred | SpamapS: I mean, I can also put a try/except into the inventory meta code | 23:47 |
| *** pc_m has joined #openstack-infra | 23:47 | |
| fungi | clarkb: mind if i delete all ready nodes from jenkins04 so it doesn't keep squatting them? | 23:47 |
| SpamapS | mordred: but do they even have volumes then? | 23:48 |
| clarkb | fungi nope | 23:48 |
| mordred | SpamapS: prolly not | 23:48 |
| mordred | SpamapS: so yeah - we can prolly skip the nova volume api attempt | 23:48 |
| SpamapS | mordred: so isn't it appropriate to error out? | 23:48 |
| mordred | SpamapS: maybe so | 23:48 |
| mordred | SpamapS: it's possible I'm wrongthinking this | 23:48 |
| fungi | clarkb: done. thanks! | 23:48 |
| *** yamamoto has joined #openstack-infra | 23:48 | |
| SpamapS | Also agreed that list_something should return a list btw. ;) | 23:48 |
| fungi | hopefully that'll speed things up a little | 23:48 |
| * SpamapS is just catching up. | 23:48 | |
| mordred | greghaynes: that's nodepool consuming a resource api in shade - so we should make sure that we're happy with its get/list/search interface | 23:49 |
| greghaynes | ok | 23:49 |
| greghaynes | just shade API instability | 23:49 |
| mordred | greghaynes: yah - wanted to make sure we're ready to lock that api | 23:50 |
| mordred | greghaynes: essentially, once nodepool starts using it, we need to consider it _stable_ and deal with transition paths and stuff | 23:50 |
| greghaynes | Yep | 23:50 |
| *** ddieterly has joined #openstack-infra | 23:50 | |
| mordred | SpamapS: I tink list_images returning a dict is a holdover from when we were caching in a dict | 23:50 |
| *** ajmiller has quit IRC | 23:51 | |
| dansmith | is the gate sick? | 23:51 |
| SpamapS | Or just a benign attempt at returning a more useful data structure. | 23:51 |
| fungi | something seems to have broken all tempest-based jobs again (maybe all devstack-based jobs?) | 23:51 |
| fungi | dansmith: just noticed the same. playing "spot the bug" now | 23:51 |
| dansmith | fungi: okay | 23:52 |
| palendae | fungi: Not necessarily, os-ansible-deployment isn't running either | 23:52 |
| palendae | And we don't use devstack | 23:52 |
| mordred | SpamapS: we could ALSO do that - but I'd want it to be consistent across all of our list_foo calls | 23:52 |
| fungi | http://logs.openstack.org/87/176387/3/gate/gate-tempest-dsvm-full/4ce29bd//logs/devstacklog.txt.gz#_2015-04-22_23_26_18_033 | 23:53 |
| dansmith | I'm waiting for mine to die so I can look at the logs | 23:53 |
| dansmith | devstack failed to start tho | 23:53 |
| fungi | "ImportError: No module named pathlib" | 23:53 |
| mordred | SpamapS, morganfainberg: OH WOW | 23:53 |
| dansmith | suh-weet | 23:53 |
| fungi | looks like a django_pyscss issue | 23:53 |
| morganfainberg | mordred, hi | 23:53 |
| dansmith | fungi: now you play "who released a lib today" right? | 23:53 |
| mtreinish | dansmith, fungi: isn't pathlib in stdlib? | 23:53 |
| mordred | morganfainberg: http://paste.openstack.org/show/205152/ | 23:53 |
| morganfainberg | dansmith: isn't that every day in openstack? | 23:53 |
| mtreinish | oh not on python 2 | 23:53 |
| dansmith | morganfainberg: *hangs head* ... yes :/ | 23:54 |
| mordred | morganfainberg: the depth of that | 23:54 |
| mordred | exception | 23:54 |
| morganfainberg | mordred: omg | 23:54 |
| mordred | morganfainberg: yay for late binding | 23:54 |
| morganfainberg | mordred: i... | 23:54 |
| mordred | so, I construct a cinder_client with a keystone Session | 23:54 |
| morganfainberg | i half expected it to hit max depth there. | 23:54 |
| fungi | dansmith: yeah, that's the current roll of the die anyway | 23:54 |
| SpamapS | mordred: impressive stacktrace ;) | 23:54 |
| * mordred goes to look to see if there is a saner thing to do ... it _may_ still be reasonable to trap this rather than complain | 23:54 | |
| fungi | dansmith: hoping i don't draw a go to jail card | 23:55 |
| dansmith | fungi: heh | 23:55 |
| greghaynes | fungi: django_pyscss just released... | 23:55 |
| fungi | survey says... https://pypi.python.org/pypi/django-pyscss | 23:55 |
| fungi | indeedie | 23:55 |
| greghaynes | :) | 23:55 |
| mtreinish | SpamapS: heh, I still prefer this one: http://logs.openstack.org/95/176095/1/check/gate-cinder-python27/747e8f0/console.html#_2015-04-22_15_27_20_571 | 23:55 |
| SpamapS | mordred: I think EndpointNotFound is a nice specific thing you can turn into a shade exception. | 23:56 |
| mtreinish | which gets dumped on ever cinder unit test run | 23:56 |
| mordred | yah | 23:56 |
| mtreinish | more than once per run I should say | 23:56 |
| SpamapS | mtreinish: best part about that one is that it is a perfectly normal thing to have happen. ;) | 23:56 |
| SpamapS | ok, time to go fetch the children | 23:56 |
| * SpamapS disappears | 23:56 | |
| clarkb | mtreinish wat | 23:57 |
| fungi | sdague's openstack/requirements:tools/what-broke.py script puts django-pyscss 2.0.0 at the top of the suspect pile too, followed by python-glanceclient 0.17.1, kombu 3.0.26, libvirt-python 1.2.14, python-novaclient 2.24.1 and cliff 1.12.0 (all released 2015-04-22) | 23:57 |
| fungi | but yeah, given the traceback, django-pyscss is the clear winner this round | 23:58 |
| fungi | "you fail at python 2.x" | 23:58 |
| greghaynes | Can we start making awards | 23:58 |
| fungi | "Added support for pyScss 1.3 and Python 3" (and dropped support for any python older than that) | 23:59 |
| fungi | so! this raises an interesting challenge | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!