| *** otherwiseguy has joined #openstack-infra | 00:00 | |
| *** ArxCruz has quit IRC | 00:00 | |
| *** daya_k has joined #openstack-infra | 00:02 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 00:03 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Fingerprint for bug 1360011 https://review.openstack.org/116140 | 00:04 |
|---|---|---|
| uvirtbot | Launchpad bug 1360011 in neutron "SSH Auth fails in AdvancedNetworkOps scenario" [High,New] https://launchpad.net/bugs/1360011 | 00:04 |
| *** smcginnis has joined #openstack-infra | 00:05 | |
| smcginnis | Hey anyone around that might know why I am getting this following Jay Pipes third party CI instructions? | 00:06 |
| smcginnis | In the zuul debug logs: | 00:06 |
| smcginnis | SSHException: not a valid DSA private key file | 00:06 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Maintain the trigger cache after reconfiguring https://review.openstack.org/114761 | 00:06 |
| *** bhuvan has quit IRC | 00:06 | |
| fungi | smcginnis: i've seen that error when you set it to use a file which is not an ssh private key, or one which is encrypted (you need one with no passphrase for this) | 00:07 |
| smcginnis | fungi: Hmm, I know the one I generated following the blog instructions is not envrypted. | 00:08 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Fingerprint for bug 1360011 https://review.openstack.org/116140 | 00:08 |
| uvirtbot | Launchpad bug 1360011 in neutron "SSH Auth fails in AdvancedNetworkOps scenario" [High,New] https://launchpad.net/bugs/1360011 | 00:08 |
| smcginnis | fungi: Can you point me at where zuul is configured to make sure it is actually set to use my cert? | 00:08 |
| *** lcheng has joined #openstack-infra | 00:08 | |
| *** otherwiseguy has quit IRC | 00:09 | |
| *** _buzz has quit IRC | 00:09 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: BuildSet now has a representation https://review.openstack.org/115126 | 00:10 |
| *** RicoLin has joined #openstack-infra | 00:10 | |
| *** _buzz has joined #openstack-infra | 00:10 | |
| *** nelsnelson has quit IRC | 00:11 | |
| *** alexpilotti has joined #openstack-infra | 00:11 | |
| *** nelsnelson has joined #openstack-infra | 00:12 | |
| smcginnis | fungi: OK, nevermind. I found /etc/zuul/zuul.conf | 00:12 |
| fungi | smcginnis: ahh, good. i was about to pull up an example copy of our config template for it | 00:12 |
| smcginnis | fungi: Thanks. Sometimes I ask things before looking at the obvious. ;) | 00:13 |
| *** otherwiseguy has joined #openstack-infra | 00:13 | |
| fungi | smcginnis: yeah, looks like gerrit.sshkey is the parameter you want, and we set ours to /var/lib/zuul/ssh/id_rsa | 00:14 |
| *** annegentle has joined #openstack-infra | 00:14 | |
| *** Ryan_Lane1 has quit IRC | 00:14 | |
| smcginnis | fungi: That is what mine is set to, and cat'ing that file looks like a legitimate private key. | 00:15 |
| clarkb | the error says its not a dsa key | 00:15 |
| clarkb | are dsa and rsa keys getting mixed? | 00:16 |
| fungi | clarkb: yeah, though that's a fallthrough since paramiko first tries to see if it's an rsa key, then if not tries to see if it's a dsa key, then in typical fashion spurts a misleading error | 00:16 |
| clarkb | oh fun | 00:16 |
| fungi | it really should say "i can't figure out what kind of ssh private key this is" | 00:17 |
| smcginnis | clarkb: Thanks, that was going to be my next question. I have an RSA key. | 00:17 |
| fungi | instead it just gets overly specific about the last thing it tried | 00:17 |
| smcginnis | So I doubled checked that the public key is the one we used when requesting our CI account and it matches. The private key looks good. Anything else I can look at? | 00:18 |
| *** ZZelle has quit IRC | 00:18 | |
| asselin | anyone run into this issue? Duplicate declaration: Package[python-lxml] is already declared in file /root/config/modules/zuul/manifests/init.pp at line 64; cannot redeclare at /root/config/modules/nodepool/manifests/init.pp:39 on node ci-repomaster.novalocal | 00:19 |
| fungi | smcginnis: is it possible the zuul user doesn't have read access for that file? | 00:19 |
| smcginnis | Good idea, I'll check. | 00:19 |
| fungi | asselin: yeah, you'll probably need to make both of those conditional on whether the package is already installed, since you're trying to put zuul and nodepool on a single server | 00:20 |
| *** gokrokve has joined #openstack-infra | 00:20 | |
| *** baoli has joined #openstack-infra | 00:20 | |
| smcginnis | fungi: 400 zuul:zuul. It should be able to read it fine. Should only need read access, right? | 00:21 |
| asselin | fungi, too bad puppet's not smart enough to figure that out :( | 00:21 |
| fungi | smcginnis: i believe so, yes... but did you check the parent directories as well, or su to zuul and try to cat that file?> | 00:21 |
| fungi | asselin: yes, we curse it for that regularly | 00:21 |
| *** gokrokve has quit IRC | 00:21 | |
| asselin | lol | 00:21 |
| *** gokrokve has joined #openstack-infra | 00:22 | |
| smcginnis | fungi: Yep, zuul user is able to cat the file just fine. | 00:22 |
| fungi | smcginnis: huh. try to use a normal ssh client to connect to your gerrit system account with that key? | 00:23 |
| * smcginnis googling how to ssh with specific key file | 00:24 | |
| fungi | smcginnis: something like 'ssh -p 29418 -i /var/lib/zuul/ssh/id_rsa youruser@review.openstack.org gerrit ls-projects' | 00:24 |
| fungi | smcginnis: but obviously put whatever your proper username in gerrit is | 00:24 |
| *** baoli has quit IRC | 00:25 | |
| fungi | (for that system account associated with the key) | 00:25 |
| smcginnis | As zuul user I was able to get a project list successfully using "ssh -p 29418 -i /var/lib/zuul/ssh/id_rsa dell-storagecenter-ci@review.openstack.org gerrit ls-projects" | 00:26 |
| *** gokrokve has quit IRC | 00:26 | |
| fungi | smcginnis: okay, so that's a usable key | 00:27 |
| fungi | narrows down possible reasons zuul/paramiko might complain about it at least | 00:27 |
| clarkb | is maybe the path zuul is using wrong? | 00:27 |
| smcginnis | sshkey path in zuul.conf does point to the right location. | 00:28 |
| jeblair | smcginnis: any chance you have the wrong username configured in zuul.conf? | 00:28 |
| smcginnis | clarkb: fungi: Oh hold on. May be a dumb error. | 00:29 |
| jeblair | smcginnis: i see auth failures from "dell-storage-ci" in gerrit | 00:29 |
| jeblair | smcginnis: but "dell-storagecenter-ci" succeeds | 00:29 |
| smcginnis | Nope. I think I originally had the account name wrong in the zuul.conf file and I hadn't restarted the zuul service since I changed it. | 00:30 |
| smcginnis | Restarted it and didn't see the error right away in debug log, but once things settled down it's back to the same thing. | 00:30 |
| fungi | ohhhhh, i wonder if paramiko even throws that same error message when it can't authenticate to the remote server | 00:30 |
| smcginnis | jeblair: Is the user name stored multiple places? | 00:30 |
| jeblair | smcginnis: i'm still seeing attempts for dell-storage-ci | 00:31 |
| smcginnis | fungi: Looks like that error may be misleading. | 00:31 |
| smcginnis | jeblair: I've restarted so it shouldn't be. I'll do some grep'ing and see where it might still have the original mistake. | 00:31 |
| smcginnis | jeblair: Thanks for checking on that. | 00:31 |
| smcginnis | grep -r storage /etc/zuul/* only shows me my corrected name in the zuul.conf file. | 00:32 |
| *** masayukig has quit IRC | 00:32 | |
| smcginnis | Do I maybe need to restart any other services? | 00:32 |
| jeblair | smcginnis: may want to double check that zuul actually stopped | 00:32 |
| *** mikedillion has quit IRC | 00:33 | |
| *** ramashri has quit IRC | 00:33 | |
| *** masayukig has joined #openstack-infra | 00:33 | |
| jeblair | looks fairly continuous in the zuul log; i don't see a gap | 00:33 |
| smcginnis | jeblair: service zuul stop reports none running bit service zuul status says it is running. | 00:34 |
| smcginnis | jeblair: pkill zuul no effect. | 00:34 |
| clarkb | https://jenkins06.openstack.org/job/gate-nova-docs/5736/console is weird | 00:34 |
| clarkb | why do we need docs.python.org? | 00:34 |
| jeblair | smcginnis: the process is called 'zuul-server' | 00:34 |
| smcginnis | Regular kill won't make them go away. | 00:35 |
| jeblair | clarkb: 'intersphinx' sphinx plugin | 00:35 |
| smcginnis | What is the correct way to restart zuul? | 00:35 |
| *** emagana has quit IRC | 00:35 | |
| smcginnis | I suppose I could just reboot. | 00:35 |
| jeblair | smcginnis: i usually use '/etc/init.d/zuul stop'; it's a hard stop. 'restart' tries to do a graceful restart. | 00:36 |
| jeblair | smcginnis: looks like you got it. | 00:36 |
| *** emagana has joined #openstack-infra | 00:36 | |
| smcginnis | kill -9 | 00:36 |
| smcginnis | jeblair: Starting up again. Debug log looks OK so far. | 00:37 |
| *** daya_k has quit IRC | 00:37 | |
| *** tomoe_ has joined #openstack-infra | 00:39 | |
| smcginnis | Thanks everyone. Looks like it may be working fine now. My CI didn't kick off, but the zuul piece seems to be OK. | 00:39 |
| *** gokrokve has joined #openstack-infra | 00:39 | |
| smcginnis | On to the next problem. :) | 00:39 |
| *** emagana has quit IRC | 00:40 | |
| *** ChuckC has quit IRC | 00:41 | |
| *** masayukig has quit IRC | 00:41 | |
| *** masayukig has joined #openstack-infra | 00:42 | |
| *** masayukig has quit IRC | 00:42 | |
| *** masayukig has joined #openstack-infra | 00:43 | |
| *** smcginnis has quit IRC | 00:56 | |
| *** nosnos has joined #openstack-infra | 00:59 | |
| *** rcarrillocruz has quit IRC | 01:00 | |
| *** harlowja is now known as harlowja_away | 01:00 | |
| *** annegentle has quit IRC | 01:02 | |
| *** Ryan_Lane has quit IRC | 01:03 | |
| *** rcarrillocruz has joined #openstack-infra | 01:04 | |
| *** Ryan_Lane has joined #openstack-infra | 01:06 | |
| *** lcheng has quit IRC | 01:07 | |
| *** Ryan_Lane has quit IRC | 01:07 | |
| *** dims has quit IRC | 01:09 | |
| *** mriedem has joined #openstack-infra | 01:09 | |
| *** armax has left #openstack-infra | 01:10 | |
| *** yamahata has joined #openstack-infra | 01:10 | |
| *** armax has joined #openstack-infra | 01:10 | |
| *** armax has left #openstack-infra | 01:10 | |
| *** Ryan_Lane has joined #openstack-infra | 01:11 | |
| *** annegentle has joined #openstack-infra | 01:12 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Replace all check/gate: noop with a template https://review.openstack.org/115500 | 01:15 |
| mrodden | so when new nodes that are more static-ish are added, do they get configured with puppet-apply or do they check into the puppetmaster and get their stuff from there.... ? | 01:16 |
| *** dims has joined #openstack-infra | 01:17 | |
| mrodden | trying to get more familiar with the puppet side of things... i'm more of a chef guy usually | 01:17 |
| *** masayukig has quit IRC | 01:18 | |
| *** marcoemorais has quit IRC | 01:18 | |
| clarkb | mrodden: the long lived "static" nodes run puppet agent and communicate with the master. However, we don't run puppet agent as a daemon. Instead we have ansible trigger puppet so that we get a little orchestration out of it | 01:19 |
| mrodden | oh. it runs it periodically i'm assuming? | 01:20 |
| lifeless | clarkb: were you aware of the https://review.openstack.org/#/c/110486/ effort ? | 01:20 |
| fungi | mrodden: every 15 minutes | 01:20 |
| *** yaguang has joined #openstack-infra | 01:20 | |
| *** baoli has joined #openstack-infra | 01:21 | |
| clarkb | lifeless: not at all | 01:21 |
| jeblair | lifeless: nodepool has per-process mysql db more in the way you describe (i think you may have helped review it?) | 01:21 |
| clarkb | lifeless: I did show ^ that code to zzzeek | 01:22 |
| clarkb | and the gertty sqlite migration stuff | 01:22 |
| *** Sukhdev has quit IRC | 01:22 | |
| lifeless | I mean its cool that the instance hook can do what they need, but it totally wasn't the intent... and doing it this way will preclude the other | 01:23 |
| clarkb | lifeless: ya I think it should be a -2 honestly | 01:23 |
| lifeless | clarkb: I can't -2 on that tree | 01:23 |
| clarkb | lifeless: as soon as any other project uses that code we are stuck using testr to run those tests | 01:23 |
| clarkb | and this is where I get frustrated with oslo | 01:23 |
| lifeless | btw the bad versions thing | 01:24 |
| *** masayukig has joined #openstack-infra | 01:24 | |
| lifeless | nova stable was broken by the jsonschema thing too | 01:24 |
| lifeless | that fix is on its 5th recheck | 01:24 |
| lifeless | once it lands | 01:24 |
| clarkb | the codify bad practices all over the place then we are stuck with them | 01:24 |
| lifeless | then the setup.cfg version fix can enqueue | 01:24 |
| lifeless | and then we can move forward on pbr again | 01:24 |
| clarkb | the test cross process locking, oslo.config's weird put the config all over the place, ^, and so on | 01:25 |
| clarkb | lifeless: noted | 01:25 |
| lifeless | there's a few murano old releases that can't CI cleanly too | 01:25 |
| *** Sukhdev has joined #openstack-infra | 01:26 | |
| lifeless | but they're wedged for other reasons, so this won't make that recovery significantly | 01:26 |
| lifeless | worse | 01:26 |
| *** baoli has quit IRC | 01:26 | |
| lifeless | and there's os-purge or whatever, but thats not entangled in the gate | 01:26 |
| *** Daisy has joined #openstack-infra | 01:26 | |
| *** yaguang_ has joined #openstack-infra | 01:27 | |
| jeblair | i've -1d that patch | 01:28 |
| clarkb | reading zzzeek's comment I don't understand why he would want to go with that approach | 01:28 |
| clarkb | if you want py.test parallel to work you will do what nodepool does | 01:28 |
| clarkb | https://github.com/zzzeek/sqlalchemy/blob/54592942c4a9c3d6d891519082555f8081026445/lib/sqlalchemy/testing/plugin/pytestplugin.py#L47 may explain it | 01:29 |
| mrodden | jeblair: do only certain versions of python-jenkins get published to pypi.python.org? | 01:29 |
| clarkb | mrodden: no, I think there were issues with publishing python-jenkins that haven't all been sorted out | 01:30 |
| mrodden | oh | 01:31 |
| clarkb | zaro: can we publish the working version? what do we need to do for that/ | 01:31 |
| *** masayukig has quit IRC | 01:31 | |
| *** yaguang has quit IRC | 01:31 | |
| *** masayukig has joined #openstack-infra | 01:32 | |
| mrodden | zaro directed me to jeblair last time i asked about a release | 01:32 |
| clarkb | oh | 01:32 |
| mrodden | but zaro is the only individual in the ptl group with tagging priv.s | 01:33 |
| mrodden | that doesn't seem to be the issue though, there is a 3.2 and i think even 3.3 that aren't on pypi | 01:33 |
| mrodden | so i wasn't sure what happened there | 01:33 |
| clarkb | mrodden: the releases were broken | 01:34 |
| mrodden | i mean, there is a 3.2 tag and 3.3 tag | 01:34 |
| mrodden | oh | 01:34 |
| clarkb | mrodden: releases are manual because at the time those releases did not fit into our release tooling | 01:34 |
| clarkb | mrodden: the working release has not been put on pypi. (I think 3.3 actually works) | 01:34 |
| mrodden | ok | 01:34 |
| mrodden | i can maybe pull that into my local bandersnatch mirror...? | 01:34 |
| clarkb | mrodden: possibly if you make an sdist and put it on disk there | 01:35 |
| clarkb | not sure what will happen if/when 3.3 shows up on pypi | 01:35 |
| mrodden | wonder if it will clean it up next run | 01:35 |
| jeblair | yeah, i'm going to make a 3.3 release tomorrow | 01:35 |
| jeblair | it's manual | 01:35 |
| jeblair | (since it's pre-pbr) | 01:35 |
| jeblair | we don't do that sort of thing around here very often :) | 01:35 |
| mrodden | i c | 01:35 |
| mrodden | was wondering why it was manual | 01:36 |
| jeblair | yeah, all future releases will be automated | 01:36 |
| pleia2 | camunoz: thanks for the email | 01:36 |
| mrodden | i actually started using pbr for some of my personal projects | 01:37 |
| mrodden | its pretty handy | 01:37 |
| camunoz | pleia2: no worries! pinged you earlier to see if you could upload another public key to that server as I am working remotely today, but sean helped me out with that | 01:37 |
| jeblair | mrodden: yeah, i don't even know what to do without it anymore :) | 01:38 |
| clarkb | jeblair: I was about to say that. I had to debug a setup.py (a real one) a little while back and it was weird | 01:38 |
| jeblair | mrodden: in a similar vein, i don't know how to use github. :) | 01:38 |
| pleia2 | camunoz: ah! sorry about that, coffee meeting went longer than expected :) glad you got sorted | 01:38 |
| jeblair | i'm like 'i have to clone in the web? then set up a remote to point to my clone? then push to that clone? then click some stuff?' | 01:39 |
| jeblair | 'git review' | 01:39 |
| mrodden | yeah <3 git review | 01:39 |
| clarkb | jeblair: even better is when you clone a thing then later decide to fork it. you have to read the git remote man page to figure out how to change your origin remote | 01:39 |
| mrodden | although when you have to explain advance push commands to people after they are used to using git review a lot... | 01:40 |
| jeblair | which is pretty much all i ever do | 01:40 |
| StevenK | jeblair: Launchpad gets a large number of questions asking where the clone button is, and the answer is "No, just push it, LP will figure it out" | 01:40 |
| *** harlowja_away is now known as harlowja | 01:40 | |
| *** unicell has quit IRC | 01:40 | |
| clarkb | StevenK: but bzr made it easy | 01:40 |
| clarkb | bzr clone | 01:40 |
| clarkb | lp:foo | 01:40 |
| StevenK | Yup | 01:41 |
| StevenK | That part of bzr is nice, it's just the other parts :-) | 01:41 |
| StevenK | I'm now spoilt by multiple branches in one repo | 01:41 |
| r1chardj0n3s | fungi: do the jenkins VMs use virtualenv installed from ubuntu or from pypi? | 01:42 |
| lifeless | so sqlaclchemy uses py.test? | 01:42 |
| clarkb | StevenK: and lightweight branches | 01:42 |
| *** annegentle has quit IRC | 01:42 | |
| lifeless | I'd be happy enough to help things be more compatible there | 01:42 |
| clarkb | lifeless: I think it can use it and it uses a plugin system? | 01:42 |
| clarkb | lifeless: honestly I think that is an anti pattern | 01:42 |
| StevenK | clarkb: bzr has lightweight checkouts, but it requires some fiddling to set up | 01:42 |
| lifeless | clarkb: py.test is? | 01:42 |
| clarkb | aren't these test frameworks supposed to hae a runner compatiblity layert | 01:42 |
| lifeless | boom tish | 01:42 |
| clarkb | lifeless: no, the special code to make py.test work | 01:42 |
| clarkb | at least when I switched projects to testr I ensured testr and nose both worked | 01:43 |
| lifeless | right :) | 01:43 |
| *** alexpilotti has quit IRC | 01:44 | |
| *** yaguang_ has quit IRC | 01:46 | |
| *** yaguang has joined #openstack-infra | 01:46 | |
| mrodden | so.... ansible runs this python script that runs puppet agent that checks into puppet master and then gathers the module and runs them ? | 01:47 |
| *** alexpilotti has joined #openstack-infra | 01:47 | |
| mrodden | openstack-infra/config/blob/master/modules/ansible/files/library/config_management/puppet | 01:47 |
| clarkb | mrodden: it just runs a few things in a specific order with parallel branching where possible | 01:48 |
| mrodden | it == ansible? | 01:48 |
| *** alexpilotti has quit IRC | 01:48 | |
| clarkb | mrodden: yes | 01:49 |
| *** gokrokve has quit IRC | 01:49 | |
| clarkb | so that python file is just tellign ansible how to run puppet and check the return codes | 01:50 |
| mrodden | yeah | 01:50 |
| fungi | r1chardj0n3s: pretty sure we're installing the latest tox from pypi and it either pulls in or vendors its own virtualenv | 01:50 |
| clarkb | the actual orchestration happens in modules/openstack_project/files/ansible/remote_puppet.yaml | 01:50 |
| clarkb | mrodden: ^ | 01:50 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add oslo.serialization to devstack-vm-gate-wrap.sh https://review.openstack.org/112994 | 01:50 |
| mrodden | ah k, i'll check that out | 01:51 |
| clarkb | mrodden: and the separation there is puppet module for ansible to run puppet (wow that got confusing fast) and openstacks use of the ansible module to run puppet | 01:51 |
| r1chardj0n3s | fungi: well, the good news is that I can reprodce the problem on a local ubuntu system | 01:51 |
| r1chardj0n3s | fungi: the bad news is I have, literally, no idea what the hell is going on :) | 01:51 |
| clarkb | mrodden: that basically says run puppet on git nodes, then on review.o.o, then on everything else | 01:51 |
| lifeless | clarkb: so when I say be more compatible, I mean work with him to get what sqlalchemy/oslo.db need in a clean fashion | 01:51 |
| clarkb | mrodden: and when the number of nodes in each step is >1 it will do them in parallel but sync before it moves to the next step | 01:51 |
| clarkb | lifeless: right, do what nodepool does | 01:52 |
| clarkb | lifeless: which should work in any test runner | 01:52 |
| mrodden | ok, so its pretty much a sequential orchestration thing | 01:52 |
| clarkb | mrodden: yup | 01:52 |
| lifeless | clarkb: note that py.test is less a test runner and more a totally different environment for running tests in | 01:52 |
| lifeless | clarkb: it has a shim to run unittest tests, but thats not its native model | 01:52 |
| clarkb | mrodden: we needed that because creating new projects safely means updating the git mirrors, then gerrit, then everything else | 01:53 |
| lifeless | clarkb: similar to nose, but nose is much less different vs stdlib | 01:53 |
| fungi | r1chardj0n3s: python packaging, huh? crazy stuff that | 01:53 |
| r1chardj0n3s | fungi: in short, there's a "local" directory inside the virtualenv that is only created on ubuntu, and it contain symlinks to the non-"local" "lib", "bin", etc. virtualenv directories (where stuff is actually installed) and that "local/lib" is in the sys.path earlier and so it all falls about laughing. | 01:53 |
| clarkb | lifeless: ugh then why does everyone keep wanting to use it with openstack | 01:53 |
| clarkb | lifeless: thats like saying we should rewrite all of our tests | 01:53 |
| lifeless | clarkb: precisely | 01:53 |
| r1chardj0n3s | fungi: there's some weird-ass debian /local/ stuff going on here | 01:53 |
| lifeless | clarkb: that is exactly what they want | 01:53 |
| clarkb | lifeless: I don't even... I guess if they want to rewrite ~100k tests they should have a go | 01:53 |
| fungi | r1chardj0n3s: debian patched python, huh? crazy stuff that | 01:54 |
| r1chardj0n3s | fungi: those wacky guys | 01:54 |
| *** KanagarajM2 has quit IRC | 01:54 | |
| r1chardj0n3s | fungi: so now I figure out how to work around it; shouldn't be impossible... | 01:54 |
| clarkb | r1chardj0n3s: fungi: don't look at centos6 the patching that happened there is scary | 01:54 |
| r1chardj0n3s | :) | 01:54 |
| clarkb | their 2.6 has 3.3 in it | 01:54 |
| fungi | clarkb: rhel, but yah | 01:54 |
| *** zul has joined #openstack-infra | 01:55 | |
| mrodden | "their 2.6 has 3.3 in it" .... wut? :/ | 01:55 |
| fungi | centos just suffers from reusing it | 01:55 |
| clarkb | mrodden: yes red hat has backported bits of 3.3 (maybe 3.2) into 2.6 | 01:55 |
| clarkb | mrodden: it broke eventlet for a lot of projects because monkeypatching didn't work right | 01:55 |
| clarkb | they fixed this in eventlet not rhel | 01:56 |
| mrodden | all of a sudden i don't feel bad about my essex-folsom hybrid of nova | 01:56 |
| fungi | clarkb: technically so does debian (in their case to support multiple interpreters) | 01:56 |
| r1chardj0n3s | fungi: ok, time to liberally apply os.path.realpath :) | 01:56 |
| clarkb | fungi: the path thing? imo that should be standard | 01:56 |
| clarkb | fungi: because the alternative is what happens on rhel and its way broken | 01:56 |
| fungi | clarkb: yeah | 01:56 |
| r1chardj0n3s | time for lunch first | 01:57 |
| *** baohua has joined #openstack-infra | 01:57 | |
| clarkb | since we are in a ranting mood >_> can we talk neutron incubator | 01:57 |
| r1chardj0n3s | actually, this job takes ages; patch and rerun and then I can lunch ;) | 01:57 |
| clarkb | I haven't found the definitive thread to disagree with that plan anyone know where it is? | 01:57 |
| * fungi is about to get ready for sleep. later all | 01:58 | |
| clarkb | fungi: gnite | 01:58 |
| *** gargola has quit IRC | 01:58 | |
| mrodden | is that the actual plan or a proposal ? neutron-incubator | 01:59 |
| clarkb | mrodden: there is a proposal, let me find it | 02:00 |
| clarkb | mrodden: https://wiki.openstack.org/wiki/Network/Incubator there doesn't seem to be a good forum for discussing it yet though | 02:00 |
| clarkb | mrodden: I don't like the idea of it because it makes the assumption that the least logical place to put neutron code is in the neutron repo | 02:01 |
| clarkb | its just paperwork and red tape | 02:01 |
| *** Ryan_Lane has quit IRC | 02:01 | |
| clarkb | code graduates via code review... | 02:01 |
| mrodden | oh they want to kick out VPNaaS and the other L3 things | 02:01 |
| clarkb | it goes from proposed to merged | 02:02 |
| clarkb | you don't have to enable it by default if you find that it needs work | 02:02 |
| jeblair | i'm totally okay with them kicking them out :) | 02:02 |
| jeblair | however, if that's what they want to do, let's not engineer a whole process around the fiction that things would go into neutron this way | 02:02 |
| mrodden | yeah we have a way of building technical systems and metrics to solve political issues | 02:02 |
| clarkb | and oslo-incubator is a thing we are tring to remove? or at least working very hard to not use extensively. So taking that model and using it elsewhere seems like a step back | 02:03 |
| jeblair | ++ (or --) | 02:03 |
| mrodden | code review stats is a great example | 02:03 |
| jeblair | clarkb: oslo-incubator code graduates out into real repos, that last part at least is sensible | 02:03 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for cinder/keystoneclient bug 1360034 https://review.openstack.org/116158 | 02:03 |
| uvirtbot | Launchpad bug 1360034 in cinder "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20" [Undecided,New] https://launchpad.net/bugs/1360034 | 02:03 |
| jeblair | clarkb: graduating _into_ a repo is not sensible | 02:03 |
| clarkb | jeblair: agreed. and my understanding is that over time oslo incubator will die on the vine | 02:03 |
| *** nosnos has quit IRC | 02:03 | |
| jeblair | (if you can exist as an external library, well, you should just stay as an external library) | 02:03 |
| jeblair | cause that's the best way to modularize around a known interoperability mechanism | 02:04 |
| *** nosnos has joined #openstack-infra | 02:04 | |
| jeblair | (people before us have invented good ideas sometimes) | 02:04 |
| *** unicell has joined #openstack-infra | 02:04 | |
| clarkb | ++ | 02:05 |
| pleia2 | hooray zanata w/ wildfly install directions on the etherpad now https://etherpad.openstack.org/p/zanata-install | 02:05 |
| *** zul has quit IRC | 02:05 | |
| clarkb | pleia2: nice | 02:06 |
| pleia2 | and my hpcloud instance is sending emails, I thought they didn't do that | 02:06 |
| mrodden | seems like they want to say "yes we love new contributions" but also "no we don't because you make our core stuff unstable" | 02:06 |
| clarkb | pleia2: they do but are severely rate limited iirc | 02:06 |
| clarkb | pleia2: so not completely blocked and if you don't send much will be fine | 02:06 |
| pleia2 | clarkb: ah, that explains why it's been sporatic :) | 02:06 |
| *** nosnos_ has joined #openstack-infra | 02:08 | |
| *** nosnos has quit IRC | 02:08 | |
| clarkb | mrodden: I think there is some good stuff in there too. The express requirement for open source implementations that work, and the concept of code reaching a "stable" point | 02:08 |
| mrodden | yes | 02:09 |
| mrodden | agreed | 02:09 |
| clarkb | mrodden: I just think we can do that without the extra paperwork | 02:09 |
| mrodden | my idea was to fork neutron and rip out the vendor plugins | 02:09 |
| mrodden | because i just want a stable networking as a service thing | 02:09 |
| mrodden | don't care about nicera or bigswitch or whatever | 02:09 |
| mrodden | openvswitch + linuxbridge and some firewall/l3 magic is all i care about | 02:09 |
| jeblair | "neutron-lite" | 02:10 |
| mrodden | yes | 02:10 |
| mrodden | was going with 'neutrino' if possible :) | 02:10 |
| lifeless | mrodden: so if I may suggest | 02:11 |
| lifeless | mrodden: I'd like to break it up instead | 02:11 |
| lifeless | mrodden: one little API for DHCP management | 02:11 |
| lifeless | mrodden: one little API for router management. There is /no/ connection needed between them that can't be addressed by one calling the other in a cleanly disconnected way. | 02:11 |
| lifeless | mrodden: Ironic, for instance, is adding a dhcp-api abstraction internally so that some folk can deploy it without Neutron | 02:12 |
| mrodden | interesting | 02:12 |
| lifeless | which would be a lot less attractive I think if there was a trivial microservice that they could stub out in an afternoon | 02:12 |
| lifeless | mrodden: I might be a big fan of small do-one-thing APIs | 02:13 |
| lifeless | (but do it well) | 02:13 |
| *** otherwiseguy has quit IRC | 02:13 | |
| mrodden | yeah | 02:13 |
| mrodden | only issue i see is we have a lot of rest apis we call already in the chain to spawn a VM | 02:13 |
| mrodden | it does add some significant issues and latency... | 02:13 |
| lifeless | mrodden: I don't think it needs to add any latency: we have a token from keystone already, so no new roundtrips there | 02:14 |
| mrodden | there is still issues if the token is expired before we get that far | 02:15 |
| mrodden | not sure if we ever came up with a good solution to that | 02:15 |
| lifeless | mrodden: sure, but not any different to today | 02:15 |
| mrodden | right | 02:15 |
| mrodden | the other one is that neutron and nova are tightly coupled with the nova neutron callback api | 02:15 |
| mrodden | which makes me sad everytime i think about it | 02:16 |
| mrodden | haven't thought of a better solution yet though | 02:16 |
| lifeless | mrodden: webhooks | 02:17 |
| mrodden | the one time pad URLs would be cool though | 02:18 |
| lifeless | mrodden: nova should give neutron a callback url (e.g. nova/servers/$server-uuid/vnics/$id) with the vif plug call | 02:18 |
| mrodden | post JSON body to neutron that says "hit me at these urls for these events" ? | 02:18 |
| lifeless | mrodden: neutron should POST to that when that vnic plugs successfully (or fails) | 02:18 |
| mrodden | yeah that would work | 02:19 |
| lifeless | then for Ironic which also cares about port callbacks (it wants the port to be in DHCP before powering on the server) Ironic can give a different URL in its port-update call | 02:19 |
| lifeless | and Neutron would then POST to that. | 02:19 |
| lifeless | If the call is just success/fail we arguably don't even need auth | 02:20 |
| mrodden | yeah | 02:20 |
| lifeless | since the server uuid is nondeterministics | 02:20 |
| lifeless | and if you can see the server uuid you are either the user or the admin | 02:20 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for cinder minidom parse bug 1360034 https://review.openstack.org/116158 | 02:21 |
| uvirtbot | Launchpad bug 1360034 in cinder "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20" [Undecided,New] https://launchpad.net/bugs/1360034 | 02:21 |
| *** baoli has joined #openstack-infra | 02:22 | |
| *** arnaud has quit IRC | 02:24 | |
| *** lcheng has joined #openstack-infra | 02:24 | |
| *** maxbit has joined #openstack-infra | 02:26 | |
| *** baoli has quit IRC | 02:26 | |
| *** stevemar has joined #openstack-infra | 02:32 | |
| *** rcarrillocruz has quit IRC | 02:34 | |
| *** dims has quit IRC | 02:34 | |
| mrodden | the more i think of it, i would appreciate a nice stable DHCP as a Service thing | 02:36 |
| *** otherwiseguy has joined #openstack-infra | 02:36 | |
| *** rcarrillocruz has joined #openstack-infra | 02:38 | |
| *** Sukhdev has quit IRC | 02:40 | |
| lifeless | mrodden: me too!. also it would make doing e.g. differnet backends (isc-dhcpd anyone?) easier, less code to look at | 02:40 |
| *** arnaud has joined #openstack-infra | 02:41 | |
| *** maxbit has quit IRC | 02:45 | |
| *** arnaud has quit IRC | 02:45 | |
| *** nelsnelson has quit IRC | 02:46 | |
| *** maxbit has joined #openstack-infra | 02:46 | |
| *** harlowja is now known as harlowja_away | 02:49 | |
| r1chardj0n3s | \o/ we have a pip-missing-reqs failure in nova by way of jenkins :) | 02:49 |
| *** annegentle has joined #openstack-infra | 02:50 | |
| *** KanagarajM has joined #openstack-infra | 02:51 | |
| *** lcheng has quit IRC | 02:53 | |
| *** nosnos_ has quit IRC | 02:56 | |
| *** maxbit_ has joined #openstack-infra | 03:04 | |
| *** maxbit has quit IRC | 03:06 | |
| *** mriedem has quit IRC | 03:06 | |
| *** homeless has quit IRC | 03:07 | |
| *** lcheng has joined #openstack-infra | 03:07 | |
| *** maxbit has joined #openstack-infra | 03:07 | |
| *** maxbit_ has quit IRC | 03:08 | |
| *** rcarrillocruz has quit IRC | 03:09 | |
| *** lcheng has quit IRC | 03:13 | |
| *** rcarrillocruz has joined #openstack-infra | 03:13 | |
| *** arnaud has joined #openstack-infra | 03:18 | |
| *** asettle has joined #openstack-infra | 03:18 | |
| *** krtaylor has joined #openstack-infra | 03:20 | |
| openstackgerrit | A change was merged to openstack/requirements: Avoid httpretty version 0.8.3 https://review.openstack.org/113239 | 03:21 |
| *** maxbit has quit IRC | 03:22 | |
| *** maxbit_ has joined #openstack-infra | 03:22 | |
| *** baoli has joined #openstack-infra | 03:23 | |
| *** gyee has quit IRC | 03:24 | |
| *** baoli has quit IRC | 03:27 | |
| *** maxbit has joined #openstack-infra | 03:28 | |
| *** maxbit_ has quit IRC | 03:32 | |
| *** lcheng has joined #openstack-infra | 03:34 | |
| *** dims has joined #openstack-infra | 03:35 | |
| *** weshay has quit IRC | 03:37 | |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Test puppet only on precise to fix the gate https://review.openstack.org/116169 | 03:39 |
| *** dims has quit IRC | 03:39 | |
| *** nosnos has joined #openstack-infra | 03:40 | |
| pleia2 | jeblair: I made you and mordred stick puppets for puppetconf! https://www.flickr.com/photos/pleia2/14808962909/sizes/z/ | 03:43 |
| clarkb | pleia2: those are amazing | 03:44 |
| pleia2 | hehe | 03:44 |
| clarkb | is that next month? | 03:44 |
| pleia2 | yeah | 03:44 |
| clarkb | probably down the street from you | 03:44 |
| pleia2 | yeah, like 2 blocks away | 03:44 |
| pleia2 | oh my gosh, twilight sparkle has pink sunglasses | 03:44 |
| clarkb | I don't knwo what that is but if they are for the puppet you must | 03:44 |
| pleia2 | she's a my little pony sitting on my desk | 03:45 |
| pleia2 | haha PERFECT https://www.flickr.com/photos/pleia2/14809145127/sizes/z/ | 03:47 |
| clarkb | wow | 03:49 |
| *** annegentle has quit IRC | 03:52 | |
| *** esker has joined #openstack-infra | 03:53 | |
| *** amotoki has joined #openstack-infra | 03:54 | |
| *** adalbas has quit IRC | 03:58 | |
| *** ominakov has joined #openstack-infra | 04:03 | |
| *** jaypipes has joined #openstack-infra | 04:05 | |
| *** rcarrillocruz has quit IRC | 04:05 | |
| *** rcarrillocruz has joined #openstack-infra | 04:08 | |
| StevenK | clarkb: Can you have a poke at https://review.openstack.org/114869 , please? | 04:08 |
| *** garyh has quit IRC | 04:09 | |
| *** praneshp has quit IRC | 04:10 | |
| *** _Alexandra_ has joined #openstack-infra | 04:11 | |
| *** asettle has quit IRC | 04:13 | |
| *** _Alexandra_ is now known as asettle | 04:13 | |
| *** pcrews has quit IRC | 04:14 | |
| *** harlowja_at_home has joined #openstack-infra | 04:15 | |
| *** annegentle has joined #openstack-infra | 04:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add python33 jobs for os-{apply,collect}-config https://review.openstack.org/114869 | 04:19 |
| clarkb | lifeless: https://jenkins01.openstack.org/job/gate-heat-python26/9168/consoleFull did python-fixtures release a brkoen release? | 04:20 |
| clarkb | the answer according to pypi is most likely | 04:20 |
| clarkb | lifeless: http://bazaar.launchpad.net/~python-fixtures/python-fixtures/trunk/revision/76 looks to blame | 04:22 |
| *** Ryan_Lane has joined #openstack-infra | 04:22 | |
| zaro | mrodden, clarkb: i do not have permission to upload python-jenkins to pypi | 04:22 |
| clarkb | zaro: ya, jeblair said he is going to do it tomorrow | 04:22 |
| *** baoli has joined #openstack-infra | 04:23 | |
| zaro | where's fungi ? | 04:25 |
| lifeless | clarkb: on it, fixing. | 04:27 |
| clarkb | fungi: <- right there? | 04:27 |
| clarkb | lifeless: thank you | 04:27 |
| *** baoli has quit IRC | 04:28 | |
| lifeless | clarkb: 0.3.16 on pypi. | 04:31 |
| lifeless | clarkb: not verified on 2.6 (who has that shit anymore) but should be fixed. | 04:31 |
| *** rpodolyaka1 has joined #openstack-infra | 04:31 | |
| clarkb | lifeless: thanks | 04:32 |
| clarkb | lifeless: rhel6 | 04:32 |
| clarkb | and suse | 04:32 |
| lifeless | clarkb: like I said | 04:32 |
| lifeless | clarkb: who | 04:32 |
| clarkb | people use it appaerntly | 04:32 |
| *** tsg has quit IRC | 04:33 | |
| *** tgohad has joined #openstack-infra | 04:33 | |
| *** yamahata_ has quit IRC | 04:33 | |
| zaro | fungi: I think Dave mean remove the abililty to add/remove httpPassword and ssh-key from set-account commands, https://gerrit-review.googlesource.com/#/c/59487 | 04:36 |
| *** dims has joined #openstack-infra | 04:36 | |
| *** Daisy has quit IRC | 04:37 | |
| zaro | fungi: which would those would require admins to change. I don't see a problem with that, do you? | 04:38 |
| *** ppai has joined #openstack-infra | 04:38 | |
| zaro | *would/ would mean | 04:39 |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Replace all check/gate: noop with a template https://review.openstack.org/115500 | 04:39 |
| *** wenlock has joined #openstack-infra | 04:39 | |
| lifeless | clarkb: yes, but I mean developers :) | 04:39 |
| *** garyh has joined #openstack-infra | 04:40 | |
| *** dims has quit IRC | 04:41 | |
| HenryG | I came here looking for possible info on py26 unit test issues with fixtures. I see some chatter but I am not sure what the status is? | 04:42 |
| lifeless | HenryG: its fixed, the next bandwersnatch mirror push should correct it | 04:43 |
| *** SumitNaiksatam has quit IRC | 04:43 | |
| HenryG | lifeless: yay! thanks | 04:44 |
| clarkb | lifeless: mirror is updated | 04:44 |
| HenryG | lifeless: This is the symptom? http://logs.openstack.org/32/110432/5/check/gate-neutron-python26/1ba6a5e/testr_results.html.gz | 04:44 |
| *** SumitNaiksatam has joined #openstack-infra | 04:44 | |
| *** wenlock has quit IRC | 04:45 | |
| HenryG | lifeless: can I recheck now? Bug #? | 04:45 |
| lifeless | HenryG: not aware of a bug #, yes you can recheck | 04:45 |
| HenryG | lifeless: tyvm! | 04:45 |
| lifeless | oh yay, from taht oslo.db change " Let it be known that this is entirely incorrect, as well as that I *despise* testr, and I'd much prefer we use pytest. " | 04:45 |
| lifeless | thats lovely, lets get all critical of things | 04:45 |
| *** gokrokve has joined #openstack-infra | 04:47 | |
| *** harlowja_at_home has quit IRC | 04:48 | |
| zaro | fungi: sorry, i think i stated my previous thought incorrectly. not remove the command just remove the commands from modifyAccount capability. only gerrit admins can use that command | 04:48 |
| *** Daisy has joined #openstack-infra | 04:48 | |
| *** annegentle has quit IRC | 04:50 | |
| *** Ryan_Lane has quit IRC | 04:51 | |
| *** hdd has joined #openstack-infra | 04:51 | |
| *** praneshp has joined #openstack-infra | 04:52 | |
| *** hemna has quit IRC | 04:53 | |
| *** esker has quit IRC | 04:53 | |
| *** lcheng has quit IRC | 04:54 | |
| *** rpodolyaka has quit IRC | 04:55 | |
| *** Vega has joined #openstack-infra | 04:55 | |
| *** rpodolyaka has joined #openstack-infra | 04:55 | |
| *** Vega is now known as Guest47474 | 04:56 | |
| *** hdd has quit IRC | 04:56 | |
| *** hdd has joined #openstack-infra | 04:57 | |
| *** praneshp has quit IRC | 04:57 | |
| *** ominakov has quit IRC | 04:58 | |
| *** hdd has quit IRC | 05:00 | |
| *** asettle has quit IRC | 05:01 | |
| *** tgohad has quit IRC | 05:01 | |
| *** praneshp has joined #openstack-infra | 05:05 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Add attrdict to global requirements https://review.openstack.org/116175 | 05:07 |
| *** arnaud__ has joined #openstack-infra | 05:08 | |
| stevemar | zuul, why you so red! | 05:08 |
| clarkb | stevemar: fixtures had a release that was not happy with python 2.6 | 05:09 |
| clarkb | stevemar: was fixed and things should clear up | 05:09 |
| stevemar | clarkb, ah, good to know sir | 05:10 |
| *** signed8bit_ZZZzz has quit IRC | 05:10 | |
| stevemar | clarkb, so if something is in the gate queue now... odds are it shouldn't hit that problem? | 05:10 |
| clarkb | stevemar: ya | 05:11 |
| clarkb | our pypi mirror has the fixed version | 05:11 |
| clarkb | so any tests that download that version as of about 25 minutes ago should be fine | 05:11 |
| *** flaviof_zzz is now known as flaviof | 05:14 | |
| *** rcarrillocruz has quit IRC | 05:18 | |
| *** rcarrillocruz has joined #openstack-infra | 05:18 | |
| *** ArxCruz has joined #openstack-infra | 05:19 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Propose new Code Review -1 text https://review.openstack.org/116176 | 05:19 |
| *** baoli has joined #openstack-infra | 05:24 | |
| *** otherwiseguy has quit IRC | 05:25 | |
| *** wenlock has joined #openstack-infra | 05:25 | |
| *** tsg has joined #openstack-infra | 05:25 | |
| *** annegentle has joined #openstack-infra | 05:26 | |
| *** reed has quit IRC | 05:26 | |
| *** HeOS has joined #openstack-infra | 05:28 | |
| *** baoli has quit IRC | 05:28 | |
| *** tomoe__ has joined #openstack-infra | 05:44 | |
| *** tomoe_ has quit IRC | 05:44 | |
| *** harlowja_away has quit IRC | 05:44 | |
| *** tomoe_ has joined #openstack-infra | 05:45 | |
| *** rcarrillocruz has quit IRC | 05:51 | |
| *** gokrokve has quit IRC | 05:54 | |
| *** _nadya_ has joined #openstack-infra | 05:55 | |
| *** rcarrillocruz has joined #openstack-infra | 05:55 | |
| *** Daisy has quit IRC | 05:58 | |
| *** annegentle has quit IRC | 06:04 | |
| *** annegentle has joined #openstack-infra | 06:05 | |
| *** xchu has joined #openstack-infra | 06:06 | |
| *** marun has quit IRC | 06:08 | |
| pleia2 | clarkb: fixed in git \o/ (the self profiles issue) https://github.com/translate/pootle/issues/3251 | 06:08 |
| pleia2 | so at least there's that :) | 06:09 |
| *** chandankumar has joined #openstack-infra | 06:09 | |
| *** loki184 has joined #openstack-infra | 06:09 | |
| *** annegentle has quit IRC | 06:10 | |
| *** HeOS has quit IRC | 06:11 | |
| *** arnaud__ has quit IRC | 06:12 | |
| *** stevemar has quit IRC | 06:20 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/nodepool: [WIP] capacity testing for nodepool https://review.openstack.org/116189 | 06:21 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/nodepool: [WIP] capacity testing for nodepool https://review.openstack.org/116189 | 06:22 |
| *** _nadya_ has quit IRC | 06:23 | |
| *** baoli has joined #openstack-infra | 06:25 | |
| *** dims has joined #openstack-infra | 06:25 | |
| *** flaper87|afk is now known as flaper87 | 06:29 | |
| *** baoli has quit IRC | 06:29 | |
| *** dims has quit IRC | 06:30 | |
| *** tiantian has joined #openstack-infra | 06:30 | |
| tiantian | hi all, sorry to trouble | 06:30 |
| *** tsg has quit IRC | 06:31 | |
| tiantian | http://paste.openstack.org/show/98571/, we can't git-review today | 06:31 |
| tiantian | do anybody can help? | 06:31 |
| *** k4n0 has joined #openstack-infra | 06:35 | |
| tiantian | do anybody can help? | 06:35 |
| *** annegentle has joined #openstack-infra | 06:35 | |
| tiantian | http://paste.openstack.org/show/98571/, we can't git-review today | 06:35 |
| StevenK | tiantian: Please don't re-ask the same question, people will see it and reply at some point. | 06:36 |
| tiantian | ok, sorry :) | 06:37 |
| StevenK | tiantian: "Bad file number" means you can not connect, are you sure there isn't a firewall in the way blocking your connection? | 06:38 |
| *** annegentle has quit IRC | 06:42 | |
| *** arnaud has quit IRC | 06:44 | |
| tiantian | StevenK: tks, before today, it works, I will to check | 06:46 |
| *** imeerovi_ has joined #openstack-infra | 06:47 | |
| *** afazekas has joined #openstack-infra | 06:48 | |
| *** gokrokve has joined #openstack-infra | 06:48 | |
| *** che-arne has joined #openstack-infra | 06:49 | |
| *** imeerovi has quit IRC | 06:49 | |
| *** gokrokve has quit IRC | 06:53 | |
| *** praneshp has quit IRC | 06:55 | |
| *** loki184 has quit IRC | 06:57 | |
| *** Nikita_Koshikov has joined #openstack-infra | 06:58 | |
| *** rcarrillocruz has quit IRC | 06:59 | |
| *** rcarrillocruz has joined #openstack-infra | 07:03 | |
| *** skolekonov has joined #openstack-infra | 07:04 | |
| *** jlibosva has joined #openstack-infra | 07:04 | |
| *** rpodolyaka1 has quit IRC | 07:05 | |
| *** p5ntangle has joined #openstack-infra | 07:05 | |
| *** doude has joined #openstack-infra | 07:08 | |
| *** LuckyVega has joined #openstack-infra | 07:17 | |
| *** _nadya_ has joined #openstack-infra | 07:17 | |
| *** rcarrillocruz has quit IRC | 07:18 | |
| *** SumitNaiksatam has quit IRC | 07:20 | |
| *** Guest47474 has quit IRC | 07:21 | |
| *** SumitNaiksatam has joined #openstack-infra | 07:21 | |
| *** rcarrillocruz has joined #openstack-infra | 07:22 | |
| *** camunoz has quit IRC | 07:23 | |
| *** baoli has joined #openstack-infra | 07:26 | |
| *** SumitNaiksatam has quit IRC | 07:28 | |
| *** baoli has quit IRC | 07:30 | |
| *** HeOS has joined #openstack-infra | 07:35 | |
| *** Guest87807 has joined #openstack-infra | 07:37 | |
| *** bradjones has quit IRC | 07:39 | |
| *** sdake has joined #openstack-infra | 07:41 | |
| *** AlexF has joined #openstack-infra | 07:42 | |
| *** jp_at_hp has joined #openstack-infra | 07:43 | |
| *** wenlock has quit IRC | 07:48 | |
| *** Nikita_Koshikov has quit IRC | 07:49 | |
| *** gokrokve has joined #openstack-infra | 07:49 | |
| *** tomoe_ has quit IRC | 07:52 | |
| *** tomoe_ has joined #openstack-infra | 07:53 | |
| *** Nikita_Koshikov has joined #openstack-infra | 07:53 | |
| *** gokrokve has quit IRC | 07:53 | |
| *** MaxV has joined #openstack-infra | 07:56 | |
| *** LuckyVega has quit IRC | 07:58 | |
| *** _nadya_ has quit IRC | 07:58 | |
| *** jpich has joined #openstack-infra | 08:02 | |
| *** sunrenjie6 has joined #openstack-infra | 08:04 | |
| *** bradjones has joined #openstack-infra | 08:07 | |
| *** bradjones has quit IRC | 08:07 | |
| *** bradjones has joined #openstack-infra | 08:07 | |
| *** ociuhandu has quit IRC | 08:09 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Enable parallelism on DVR job https://review.openstack.org/116200 | 08:10 |
| *** dims has joined #openstack-infra | 08:14 | |
| *** derekh has joined #openstack-infra | 08:14 | |
| *** dtantsur|afk is now known as dtantsur | 08:15 | |
| *** dims has quit IRC | 08:19 | |
| *** chandankumar has quit IRC | 08:19 | |
| *** Longgeek has joined #openstack-infra | 08:23 | |
| *** rcarrillocruz has quit IRC | 08:23 | |
| *** Alexei_987 has joined #openstack-infra | 08:24 | |
| *** rcarrillocruz has joined #openstack-infra | 08:25 | |
| *** loki184 has joined #openstack-infra | 08:26 | |
| *** baoli has joined #openstack-infra | 08:26 | |
| *** mmaglana has quit IRC | 08:28 | |
| *** che-arne|2 has joined #openstack-infra | 08:28 | |
| *** jamielennox is now known as jamielennox|away | 08:29 | |
| *** che-arne has quit IRC | 08:30 | |
| *** baoli has quit IRC | 08:31 | |
| *** pblaho has joined #openstack-infra | 08:32 | |
| *** Longgeek has quit IRC | 08:35 | |
| *** Longgeek has joined #openstack-infra | 08:35 | |
| *** _nadya_ has joined #openstack-infra | 08:35 | |
| HeOS | fungi, clarkb, zaro which results with clean db on 2.8 version? | 08:37 |
| *** annegentle has joined #openstack-infra | 08:37 | |
| *** hashar has joined #openstack-infra | 08:40 | |
| *** che-arne|2 has quit IRC | 08:41 | |
| *** AlexF has quit IRC | 08:42 | |
| *** annegentle has quit IRC | 08:43 | |
| *** e0ne has joined #openstack-infra | 08:45 | |
| *** ihrachyshka has joined #openstack-infra | 08:46 | |
| *** AlexF has joined #openstack-infra | 08:47 | |
| *** lcheng has joined #openstack-infra | 08:47 | |
| *** gokrokve has joined #openstack-infra | 08:48 | |
| *** lcheng has quit IRC | 08:49 | |
| *** zz_naotok has quit IRC | 08:50 | |
| *** zz_naotok has joined #openstack-infra | 08:51 | |
| *** ihrachyshka has quit IRC | 08:52 | |
| *** gokrokve has quit IRC | 08:53 | |
| *** loki184 has quit IRC | 08:53 | |
| *** cnesa7 has joined #openstack-infra | 08:54 | |
| *** ihrachyshka has joined #openstack-infra | 08:55 | |
| *** e0ne has quit IRC | 08:57 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:59 | |
| *** Nikita_Koshikov has quit IRC | 09:00 | |
| *** mmaglana_ has joined #openstack-infra | 09:01 | |
| *** _nadya_ has quit IRC | 09:01 | |
| *** e0ne has joined #openstack-infra | 09:03 | |
| *** ZZelle has joined #openstack-infra | 09:04 | |
| *** hashar_ has joined #openstack-infra | 09:04 | |
| *** mmaglana_ has quit IRC | 09:05 | |
| *** xchu has quit IRC | 09:06 | |
| *** mkollaro has joined #openstack-infra | 09:07 | |
| *** dizquierdo has joined #openstack-infra | 09:07 | |
| *** hashar has quit IRC | 09:07 | |
| *** Nikita_Koshikov has joined #openstack-infra | 09:07 | |
| *** loki184 has joined #openstack-infra | 09:08 | |
| *** jp_at_hp has quit IRC | 09:09 | |
| *** ihrachyshka has quit IRC | 09:13 | |
| *** ihrachyshka has joined #openstack-infra | 09:15 | |
| *** AlexF has quit IRC | 09:20 | |
| *** loki184 has quit IRC | 09:23 | |
| *** rick__ has joined #openstack-infra | 09:24 | |
| *** chandankumar has joined #openstack-infra | 09:25 | |
| rick__ | Does any one know how to export ci result or log to the specified path? | 09:25 |
| *** jp_at_hp has joined #openstack-infra | 09:26 | |
| *** jp_at_hp has quit IRC | 09:26 | |
| *** jp_at_hp has joined #openstack-infra | 09:26 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:27 | |
| *** baoli has joined #openstack-infra | 09:27 | |
| rick__ | anteaya: HI. | 09:28 |
| *** pelix has joined #openstack-infra | 09:28 | |
| rick__ | clarkb: HI | 09:28 |
| *** hashar_ is now known as hashar | 09:32 | |
| *** _nadya_ has joined #openstack-infra | 09:32 | |
| *** baoli has quit IRC | 09:32 | |
| *** maxbit has quit IRC | 09:33 | |
| *** enikanorov__ has quit IRC | 09:37 | |
| *** annegentle has joined #openstack-infra | 09:37 | |
| *** sunrenjie6 has quit IRC | 09:39 | |
| *** ihrachyshka has quit IRC | 09:40 | |
| *** annegentle has quit IRC | 09:42 | |
| *** enikanorov has joined #openstack-infra | 09:43 | |
| *** gokrokve has joined #openstack-infra | 09:48 | |
| *** ZZelle has quit IRC | 09:50 | |
| *** gokrokve has quit IRC | 09:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: Promote Nova check for docker to silent https://review.openstack.org/114547 | 09:58 |
| *** Guest87807 has quit IRC | 09:58 | |
| *** Vega has joined #openstack-infra | 09:59 | |
| *** Vega is now known as Guest19211 | 09:59 | |
| *** dims has joined #openstack-infra | 09:59 | |
| *** pblaho is now known as pblaho|afk | 10:00 | |
| *** penguinRaider has quit IRC | 10:01 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** mmaglana has joined #openstack-infra | 10:02 | |
| *** rlandy has joined #openstack-infra | 10:03 | |
| *** dims has quit IRC | 10:05 | |
| *** mmaglana has quit IRC | 10:07 | |
| *** malini has joined #openstack-infra | 10:08 | |
| *** pcm_ has joined #openstack-infra | 10:15 | |
| *** pcm__ has joined #openstack-infra | 10:16 | |
| *** yamahata has quit IRC | 10:18 | |
| *** malini has left #openstack-infra | 10:18 | |
| *** penguinRaider has joined #openstack-infra | 10:18 | |
| *** pcm_ has quit IRC | 10:19 | |
| *** ppai has quit IRC | 10:25 | |
| *** baoli has joined #openstack-infra | 10:28 | |
| *** ppai has joined #openstack-infra | 10:28 | |
| *** e0ne has quit IRC | 10:32 | |
| *** baoli has quit IRC | 10:33 | |
| *** sulochan has quit IRC | 10:34 | |
| *** ihrachyshka has joined #openstack-infra | 10:35 | |
| *** sulochan has joined #openstack-infra | 10:36 | |
| *** annegentle has joined #openstack-infra | 10:37 | |
| *** rick__ has quit IRC | 10:39 | |
| *** Nikita_Koshikov has quit IRC | 10:39 | |
| *** annegentle has quit IRC | 10:42 | |
| *** mcgin has joined #openstack-infra | 10:47 | |
| *** gokrokve has joined #openstack-infra | 10:48 | |
| mcgin | Hey there, I've had a change sitting in review for a week with nothing happening on it. I'm wondering if I need to do anything with it to get it reviewed? https://review.openstack.org/#/c/114630/ | 10:48 |
| mcgin | it's gotten a +1 from jenkins | 10:48 |
| *** rcarrillocruz has quit IRC | 10:49 | |
| openstackgerrit | Timur Sufiev proposed a change to openstack-infra/config: Create merlin repo https://review.openstack.org/110325 | 10:50 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Use double-bracket style everywhere in rc*.sh https://review.openstack.org/116221 | 10:51 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Support oslo-incubator in final release scripts https://review.openstack.org/116222 | 10:51 |
| openstackgerrit | Timur Sufiev proposed a change to openstack-infra/config: Create merlin repo https://review.openstack.org/110325 | 10:51 |
| *** rcarrillocruz has joined #openstack-infra | 10:53 | |
| *** gokrokve has quit IRC | 10:53 | |
| *** dimsum_ has joined #openstack-infra | 10:59 | |
| *** alexpilotti has joined #openstack-infra | 10:59 | |
| *** Nikita_Koshikov has joined #openstack-infra | 11:00 | |
| *** mmaglana has joined #openstack-infra | 11:01 | |
| *** pblaho|afk is now known as pblaho | 11:04 | |
| *** mmaglana has quit IRC | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:08 | |
| *** dims_ has joined #openstack-infra | 11:09 | |
| *** dimsum_ has quit IRC | 11:11 | |
| *** rcarrillocruz has quit IRC | 11:12 | |
| *** tiantian has quit IRC | 11:12 | |
| *** rcarrillocruz has joined #openstack-infra | 11:15 | |
| *** nosnos has quit IRC | 11:17 | |
| *** MaxV has quit IRC | 11:17 | |
| *** nosnos has joined #openstack-infra | 11:17 | |
| *** MaxV has joined #openstack-infra | 11:17 | |
| *** Guest87807 has joined #openstack-infra | 11:18 | |
| *** p5ntangle has quit IRC | 11:21 | |
| *** KanagarajM has quit IRC | 11:21 | |
| *** pblaho is now known as pblaho|afk | 11:21 | |
| *** e0ne has quit IRC | 11:21 | |
| *** nosnos has quit IRC | 11:21 | |
| *** e0ne has joined #openstack-infra | 11:22 | |
| *** MaxV has quit IRC | 11:22 | |
| *** yaguang has quit IRC | 11:23 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for 1357677 https://review.openstack.org/114742 | 11:24 |
| *** _nadya_ has quit IRC | 11:24 | |
| *** sulochan has quit IRC | 11:24 | |
| *** dtantsur is now known as dtantsur|lunch | 11:25 | |
| *** ppai has quit IRC | 11:26 | |
| *** rcarrillocruz has quit IRC | 11:28 | |
| *** rcarrillocruz has joined #openstack-infra | 11:28 | |
| *** baoli has joined #openstack-infra | 11:29 | |
| *** hashar has quit IRC | 11:29 | |
| *** baoli has quit IRC | 11:33 | |
| *** hdd has joined #openstack-infra | 11:37 | |
| *** annegentle has joined #openstack-infra | 11:37 | |
| *** jp_at_hp has quit IRC | 11:40 | |
| *** yaguang has joined #openstack-infra | 11:40 | |
| *** unicell has quit IRC | 11:40 | |
| *** _nadya_ has joined #openstack-infra | 11:42 | |
| *** annegentle has quit IRC | 11:42 | |
| *** MaxV has joined #openstack-infra | 11:44 | |
| *** penguinRaider has quit IRC | 11:47 | |
| *** pblaho|afk is now known as pblaho | 11:47 | |
| *** MaxV has quit IRC | 11:48 | |
| *** MaxV has joined #openstack-infra | 11:48 | |
| *** gokrokve has joined #openstack-infra | 11:48 | |
| *** ildikov has joined #openstack-infra | 11:50 | |
| *** ominakov has joined #openstack-infra | 11:52 | |
| *** bigq has joined #openstack-infra | 11:52 | |
| *** MaxV has quit IRC | 11:52 | |
| *** MaxV_ has joined #openstack-infra | 11:52 | |
| *** gokrokve has quit IRC | 11:53 | |
| *** yaguang has quit IRC | 11:55 | |
| *** penguinRaider has joined #openstack-infra | 12:03 | |
| *** HeOS has quit IRC | 12:06 | |
| *** denis_makogon has joined #openstack-infra | 12:07 | |
| *** zul has joined #openstack-infra | 12:08 | |
| *** mbacchi has joined #openstack-infra | 12:13 | |
| *** dprince has joined #openstack-infra | 12:19 | |
| *** HeOS has joined #openstack-infra | 12:20 | |
| *** e0ne has quit IRC | 12:21 | |
| *** rcarrillocruz has quit IRC | 12:23 | |
| *** aysyd has joined #openstack-infra | 12:23 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/config: Import os-net-config project under TripleO https://review.openstack.org/112331 | 12:23 |
| openstackgerrit | A change was merged to openstack/requirements: Add a lower limit to oslosphinx https://review.openstack.org/113423 | 12:26 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/116242 | 12:27 |
| *** rcarrillocruz has joined #openstack-infra | 12:27 | |
| *** e0ne has joined #openstack-infra | 12:27 | |
| *** _nadya_ has quit IRC | 12:28 | |
| *** baoli has joined #openstack-infra | 12:30 | |
| *** baoli has quit IRC | 12:34 | |
| *** dtantsur|lunch is now known as dtantsur | 12:35 | |
| *** weshay has joined #openstack-infra | 12:37 | |
| *** annegentle has joined #openstack-infra | 12:37 | |
| *** zul has quit IRC | 12:40 | |
| *** kgiusti has joined #openstack-infra | 12:40 | |
| *** mmedvede has joined #openstack-infra | 12:41 | |
| *** annegentle has quit IRC | 12:42 | |
| *** che-arne has joined #openstack-infra | 12:42 | |
| *** adalbas has joined #openstack-infra | 12:43 | |
| ihrachyshka | since very recently, I started getting the following in gate unit tests for all branches: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. | 12:43 |
| ihrachyshka | any ideas what's the problem? | 12:43 |
| ihrachyshka | I see that ne testtools were released very recently (today) | 12:44 |
| ihrachyshka | *new | 12:44 |
| ihrachyshka | ah, here it is: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 | 12:44 |
| ihrachyshka | so probably we should pin versions for now | 12:44 |
| HeOS | Who knows why in gerrit (v. 2.9) database no system users? Where Gerrit takes them? | 12:47 |
| HeOS | In gerrit v. 2.8.X i can see this users in database. | 12:47 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/requirements: Pin testtools version to < 0.9.36 https://review.openstack.org/116267 | 12:48 |
| ihrachyshka | I've sent review for testtools issue: ^^ | 12:48 |
| *** gokrokve has joined #openstack-infra | 12:48 | |
| HeOS | Is it bug or feature? | 12:48 |
| *** kgiusti has quit IRC | 12:50 | |
| *** Nikita_Koshikov has left #openstack-infra | 12:50 | |
| *** p5ntangle has joined #openstack-infra | 12:51 | |
| *** Alexei_987 has quit IRC | 12:51 | |
| *** dizquierdo has quit IRC | 12:52 | |
| *** gokrokve has quit IRC | 12:53 | |
| *** kgiusti has joined #openstack-infra | 12:53 | |
| *** baohua has quit IRC | 12:55 | |
| *** jp_at_hp has joined #openstack-infra | 12:55 | |
| *** lcheng has joined #openstack-infra | 12:57 | |
| *** hdd has quit IRC | 12:58 | |
| *** hdd has joined #openstack-infra | 12:58 | |
| *** ominakov has quit IRC | 12:59 | |
| *** hashar has joined #openstack-infra | 12:59 | |
| *** yfried has joined #openstack-infra | 13:00 | |
| *** dkranz has quit IRC | 13:00 | |
| *** mmaglana has joined #openstack-infra | 13:01 | |
| *** dkranz has joined #openstack-infra | 13:02 | |
| *** tomoe__ has joined #openstack-infra | 13:02 | |
| dhellmann | ihrachyshka: which projects report that failure? | 13:02 |
| ihrachyshka | dhellmann: I saw neutron, glance | 13:03 |
| ihrachyshka | though maybe there are more | 13:03 |
| *** _nadya_ has joined #openstack-infra | 13:03 | |
| dhellmann | ihrachyshka: ok, it might be better to just let those 2 projects fix their tests if it's only a couple of them | 13:03 |
| dhellmann | ihrachyshka: pushing out the requirements change and then rolling it back across all projects may take more time | 13:04 |
| dhellmann | ihrachyshka: so if only a few projects are blocked, then it's easy enough for them to fix that | 13:04 |
| fungi | HeOS: the google gerrit developers are trying to move more data out of the database and into git repositories. you might want to ask in #gerrit for details | 13:05 |
| ihrachyshka | ok, let me do some elastic magic | 13:05 |
| sdague | ihrachyshka: glance failure seemed unrelated | 13:05 |
| *** tomoe_ has quit IRC | 13:05 | |
| sdague | yeh, I'd rather focus on projects having to fix their own unit tests on something like this. Right now I'm seeing neutron unit tests that are passing gate side, so I'd suggest more targetted approach | 13:05 |
| *** mmaglana has quit IRC | 13:06 | |
| *** _nadya_ has quit IRC | 13:06 | |
| HeOS | fungi, thanks. | 13:06 |
| *** dims_ has quit IRC | 13:07 | |
| *** p5ntangle has quit IRC | 13:07 | |
| *** dimsum_ has joined #openstack-infra | 13:08 | |
| *** mwagner_lap has quit IRC | 13:09 | |
| ihrachyshka | sdague: dhellmann: hm, indeed, I can't find glance failures other than this one: http://logs.openstack.org/49/109749/1/gate/gate-glance-python27/a014b25/testr_results.html.gz | 13:10 |
| ihrachyshka | and neutron fails in havana only (?) | 13:10 |
| ihrachyshka | so yeah, glance fails too | 13:11 |
| openstackgerrit | chandan kumar proposed a change to openstack/requirements: Added amqplib>=1.0.2 package to the global-requirements https://review.openstack.org/107653 | 13:13 |
| *** aviau has quit IRC | 13:14 | |
| *** zul has joined #openstack-infra | 13:15 | |
| *** annegentle has joined #openstack-infra | 13:17 | |
| *** markmc has joined #openstack-infra | 13:17 | |
| fungi | tiantian left. hopefully figured out the firewall issue... "bad file number" errors from ssh were new to me. apparently that's a windowsism (which would explain why i'd never seen it before) | 13:18 |
| fungi | mcgin: i suspect the jjb reviewers are a bit backed up... you're specifically looking for reviews from hashar, zaro, msabramo, mgagne and the others listed at https://review.openstack.org/#/admin/groups/194,members | 13:20 |
| *** dustins has joined #openstack-infra | 13:21 | |
| hashar | o/ | 13:22 |
| fungi | ihrachyshka: were you running into that issue only under python 2.6, or 2.7 as well? i believe python-fixtures had issues on 2.6 which were corrected with a brown-bag fix shortly after it was noticed... (fixed in 0.3.16 it looked like) | 13:24 |
| *** bknudson has quit IRC | 13:24 | |
| *** ominakov has joined #openstack-infra | 13:24 | |
| ihrachyshka | fungi: which issue? with new testtools? see: https://review.openstack.org/#/c/110303/ (one of the latest runs for glance) | 13:25 |
| *** mriedem has joined #openstack-infra | 13:25 | |
| fungi | ihrachyshka: i was wondering whether it was actually the python-fixtures issue and just looked like it was manifesting through testtools (they're part of the same suite) | 13:26 |
| *** aviau has joined #openstack-infra | 13:26 | |
| fungi | but yeah, this seems to be impacting python 2.7 jobs as well? potentially a different problem | 13:26 |
| fungi | lifeless: ^ if you're not asleep yet | 13:27 |
| ihrachyshka | different one for sure. I've already found how to fix it in neutron (and it was in havana only, so simple backport does the job) | 13:27 |
| StevenK | fungi: He flew in from NZ today, and so is probably comatose. | 13:27 |
| fungi | "TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp." | 13:27 |
| *** mfer has joined #openstack-infra | 13:28 | |
| fungi | that looks like good advice regardless | 13:28 |
| fungi | StevenK: noted--thanks! | 13:28 |
| fungi | hrm... looking at the failing glance 2.7 run there, it looks like it *could* be pythonhashseed related instead | 13:29 |
| StevenK | I think http://paste.openstack.org/show/98730/ will fix the fallout from the new testtools | 13:30 |
| StevenK | Which I can propose quickly if required | 13:30 |
| fungi | StevenK: it would be a great way to confirm, for sure | 13:30 |
| *** baoli has joined #openstack-infra | 13:30 | |
| fungi | but, like, this mismatch error is just plan silly: http://logs.openstack.org/03/110303/4/check/gate-glance-python26/b166fa8/console.html#_2014-08-22_08_53_38_669 | 13:31 |
| *** hdd has quit IRC | 13:32 | |
| fungi | is assertEqual() really smart enough to deal with unordered data types? | 13:32 |
| StevenK | fungi: Nope, it's not clever enough to do so | 13:32 |
| fungi | that's the sort of problem which makes me suspect this cropped up as an ordering difference due to new tox's pythonhashseed randomization | 13:33 |
| *** pcrews has joined #openstack-infra | 13:33 | |
| fungi | or that perhaps we're looking at manifestations of separate issues triggered by different changes (new tox for some, new testtools for others?) | 13:34 |
| StevenK | fungi: I think you're right | 13:34 |
| *** baoli has quit IRC | 13:34 | |
| *** dmsimard_away is now known as dmsimard | 13:35 | |
| fungi | regardless, these are all bugs in the tests, not the tools. changes in the tools just brought them into view | 13:36 |
| *** annegentle has quit IRC | 13:36 | |
| *** julim has joined #openstack-infra | 13:37 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make unit tests around bug_urls_map order indifferent https://review.openstack.org/116278 | 13:39 |
| *** kgiusti has quit IRC | 13:43 | |
| *** mmedvede has quit IRC | 13:44 | |
| *** mmedvede has joined #openstack-infra | 13:45 | |
| *** kgiusti has joined #openstack-infra | 13:45 | |
| *** armax has joined #openstack-infra | 13:46 | |
| *** Nikita_Koshikov has joined #openstack-infra | 13:47 | |
| *** lcheng has quit IRC | 13:47 | |
| *** bknudson has joined #openstack-infra | 13:48 | |
| *** gokrokve has joined #openstack-infra | 13:48 | |
| *** AlexF has joined #openstack-infra | 13:49 | |
| *** dizquierdo has joined #openstack-infra | 13:49 | |
| *** gokrokve has quit IRC | 13:52 | |
| *** maxbit has joined #openstack-infra | 13:54 | |
| *** russellb is now known as rustlebee | 13:54 | |
| *** signed8bit has joined #openstack-infra | 13:54 | |
| *** mwagner_lap has joined #openstack-infra | 13:55 | |
| *** _nadya_ has joined #openstack-infra | 13:55 | |
| *** k4n0 has quit IRC | 13:55 | |
| *** leifmadsen is now known as blitzrage | 13:56 | |
| *** pdmars has joined #openstack-infra | 13:58 | |
| *** pdmars has quit IRC | 13:59 | |
| *** hashar_ has joined #openstack-infra | 14:01 | |
| *** mmaglana has joined #openstack-infra | 14:01 | |
| *** kgiusti has quit IRC | 14:01 | |
| *** zz_dimtruck is now known as dimtruck | 14:02 | |
| *** alkari has joined #openstack-infra | 14:03 | |
| *** kgiusti has joined #openstack-infra | 14:04 | |
| *** hashar has quit IRC | 14:04 | |
| *** mmaglana has quit IRC | 14:06 | |
| *** eharney has quit IRC | 14:06 | |
| *** wenlock has joined #openstack-infra | 14:07 | |
| *** hdd has joined #openstack-infra | 14:07 | |
| *** stevemar has joined #openstack-infra | 14:09 | |
| *** YorikSar has quit IRC | 14:09 | |
| *** oomichi has quit IRC | 14:11 | |
| kgiusti | Hi - any pointers/advice on how to test a new CI job I'm proposing for config? https://review.openstack.org/#/c/115752/ | 14:12 |
| mriedem | fungi: could we get this promoted so we can get e-r working again and get a few gate bug queries through, since it seems the gate is kind of hosed right now https://review.openstack.org/#/c/116278/ | 14:12 |
| kgiusti | The job depends on devstack and olso.messaging changes that have yet to be merged. | 14:13 |
| *** bigq has quit IRC | 14:13 | |
| *** mmaglana has joined #openstack-infra | 14:13 | |
| mriedem | kgiusti: salv-orlando and arosen have done some patches before where they change the actual refs to be cloned for different projects to test them together, maybe you could do something similar - the patch never merges, it's just for cross-project interdependency testing | 14:15 |
| stevemar | where is the code for 'openstack proposal bot' hosted? | 14:16 |
| stevemar | and how does it determine what projects to propose changes to? | 14:16 |
| stevemar | also, good morning | 14:16 |
| fungi | mriedem: elastic-recheck isn't part of a shared queue, so as soon as it gets test results it should head straight into the gate and merge shortly thereafter | 14:17 |
| mriedem | fungi: that job has been queued in zuul for awhile (as in not a normal while) | 14:17 |
| fungi | mriedem: 37 minutes it says | 14:18 |
| mriedem | stevemar: i'd assume this http://git.openstack.org/cgit/openstack-infra/gerritbot/ | 14:18 |
| mriedem | fungi: yeah that's a long time | 14:18 |
| mriedem | e-r things usually move through quickly | 14:18 |
| fungi | uhhh | 14:18 |
| fungi | when the gate pipeline is demanding a lot of resources, the check pipeline starves a bit and looks like it currently takes a couple hours to return initial results | 14:19 |
| mtreinish | stevemar: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/gerritbot/files/gerritbot_channel_config.yaml | 14:19 |
| *** david-lyle has joined #openstack-infra | 14:19 | |
| fungi | i'm not sure i'd call half an hour unusually long except when there's nothing going on | 14:19 |
| stevemar | thanks mtreinish | 14:20 |
| *** marcusvrn has joined #openstack-infra | 14:20 | |
| kgiusti | mriedem: thanks, I'll check those out. But honestly, I'm totally noob and I really don't even know how to run CI locally in order to test it. :( Any docs etc on how to get started? | 14:20 |
| fungi | mriedem: so, this fix for elastic-recheck is preventing you from landing pending elastic-recheck signatures for existing bugs? | 14:20 |
| fungi | mriedem: just trying to get clear on the rush before i try to jump the normal testing cycle for it | 14:20 |
| mtreinish | fungi: yeah hashseed broke the unit tests | 14:20 |
| *** msabramo has quit IRC | 14:21 | |
| *** mmaglana has quit IRC | 14:21 | |
| fungi | i'll see if i can force it into the gate (not the same as promoting it within the gate as was originally requested--that wouldn't have helped this) | 14:21 |
| *** tonytan4ever has joined #openstack-infra | 14:21 | |
| stevemar | mtreinish, i had initially selected that project, but it's readme says "Gerritbot is an IRC bot that will notify IRC channels of Gerrit events", so i skipped it :( | 14:21 |
| *** mmaglana has joined #openstack-infra | 14:22 | |
| fungi | stevemar: the "openstack proposal bot" is really just a jenkins job. the list of projects it proposes to are the projects.txt in the openstack/requirements repo | 14:22 |
| mriedem | kgiusti: you just want to run tempest? | 14:23 |
| stevemar | fungi, ahhh there we go! | 14:23 |
| mtreinish | stevemar: oh, I assumed based on mriedem's message you were looking for the gerrit irc bot | 14:23 |
| mtreinish | I didn't see your original message | 14:23 |
| fungi | stevemar: that's also the list of projects which should have the requirements list enforced on them--they're synonymous for design reasons (we only propose requirements sync for projects which we enforce requirements on) | 14:23 |
| mriedem | kgiusti: i'd probably get a devstack setup with the change you need for devstack, then clone in your oslo.messaging change and restack (restart the services), then run tempest | 14:23 |
| stevemar | mtreinish, you associate me with the likes of mriedem :( | 14:24 |
| mriedem | kgiusti: http://docs.openstack.org/developer/tempest/ | 14:24 |
| stevemar | fungi, hmm, so the only -spec project on this list is keystone, are the others not enforced? | 14:24 |
| fungi | stevemar: probably not, no. any reason a -spec project needs requirements synchronized? i guess it can't hurt, but it's not like downstream distributions and deployers are going to be packaging/redistributing/rebuilding it | 14:25 |
| *** sdake_ has joined #openstack-infra | 14:26 | |
| *** mmaglana has quit IRC | 14:26 | |
| mriedem | the only requirements in those are like sphinx right? | 14:26 |
| kgiusti | mriedem: ok, that _sounds_ like what I need, assuming I can get tempest to fire off my new job. I've got changes to the zuul layout.yaml too - can I get tempest to exercise those? (Thanks for the help, btw!) | 14:27 |
| fungi | mriedem: mtreinish: as luck would have it, the change you wanted me to rush into the gate is failing tests | 14:27 |
| mriedem | kgiusti: tempest doesn't have anything to do with the infra config change | 14:27 |
| *** gargola has joined #openstack-infra | 14:28 | |
| fungi | ./elastic_recheck/tests/unit/test_stream.py:156:24: E127 continuation line over-indented for visual indent | 14:28 |
| mriedem | gfdi | 14:28 |
| mtreinish | oops sry | 14:28 |
| mriedem | :P | 14:28 |
| mtreinish | I just ran the unit tests locally, I forgot about pep8 | 14:28 |
| *** nelsnelson has joined #openstack-infra | 14:28 | |
| mriedem | rookie mistake | 14:28 |
| *** doug-fish has joined #openstack-infra | 14:28 | |
| * mriedem goes to get more coffee before making a similar error | 14:29 | |
| *** pblaho is now known as pblaho|afk | 14:29 | |
| stevemar | mriedem, yeah, mainly sphinx and|or oslosphinx | 14:30 |
| *** annegentle has joined #openstack-infra | 14:30 | |
| stevemar | mriedem, you has disappointed jenkins, womp womp | 14:30 |
| *** wenlock has quit IRC | 14:31 | |
| *** hashar_ is now known as hashar | 14:32 | |
| *** amotoki has quit IRC | 14:33 | |
| *** datsun180b has joined #openstack-infra | 14:33 | |
| mriedem | stevemar: me? or mtreinish? | 14:33 |
| mriedem | we are only the same person on fridays | 14:33 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Make unit tests around bug_urls_map order indifferent https://review.openstack.org/116278 | 14:33 |
| *** mriedem is now known as mtrenish-dplgnr | 14:34 | |
| mtreinish | fungi, mriedem: ^^^ That should fix it | 14:34 |
| fungi | oh, right, it's friday! | 14:34 |
| *** pblaho|afk is now known as pblaho | 14:34 | |
| mtreinish | although it turns out I broke flake8 locally so it was still by eye :) | 14:34 |
| mtrenish-dplgnr | +W | 14:34 |
| stevemar | mtreinish, mtrenish-dplgnr clearly i need coffee | 14:34 |
| mtreinish | mtrenish-dplgnr: heh | 14:34 |
| * stevemar skulks away in shame | 14:35 | |
| *** mtreinish is now known as mriedem-dplgnr | 14:35 | |
| mtrenish-dplgnr | oh wow this is getting confusing | 14:35 |
| mriedem-dplgnr | mtrenish-dplgnr: yep, let's see how long we can keep it up :) | 14:35 |
| fungi | mtreinish^2: it's back in the gate now | 14:35 |
| *** msabramo has joined #openstack-infra | 14:35 | |
| mriedem-dplgnr | I'm sure everyone will want to kill us by the end of the day... | 14:35 |
| *** prad has joined #openstack-infra | 14:35 | |
| *** chandankumar has quit IRC | 14:36 | |
| *** ihrachyshka has quit IRC | 14:36 | |
| fungi | however, when it comes to the climax of the film, i won't know which of you is the real mtrenish/mriedem and which one i'm supposed to shoot | 14:36 |
| mtrenish-dplgnr | hmm, if this were 'face off' would i want to be travolta or nicolas cage | 14:37 |
| mtrenish-dplgnr | fungi: the answer is always shoot both | 14:37 |
| fungi | yep, good point | 14:37 |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Uninstall puppet before running puppet jobs https://review.openstack.org/115994 | 14:38 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make unit tests around bug_urls_map order indifferent https://review.openstack.org/116278 | 14:38 |
| mtrenish-dplgnr | let the rebases begin | 14:38 |
| mriedem-dplgnr | ^^^ hey I got it right :) | 14:38 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for cinder minidom parse bug 1360034 https://review.openstack.org/116158 | 14:38 |
| uvirtbot | Launchpad bug 1360034 in cinder "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20" [Undecided,New] https://launchpad.net/bugs/1360034 | 14:38 |
| *** homeless has joined #openstack-infra | 14:39 | |
| mtrenish-dplgnr | so did something change in the nodes wrt lxml recently? like on 8/20? | 14:40 |
| mtrenish-dplgnr | i'm seeing lots of xml parsing errors failing runs in all branches since then | 14:40 |
| *** eharney has joined #openstack-infra | 14:41 | |
| fungi | mtrenish-dplgnr: i believe lxml may also suffer from ordering issues | 14:41 |
| mtrenish-dplgnr | oh that would suck | 14:42 |
| fungi | checking | 14:42 |
| mriedem-dplgnr | mtrenish-dplgnr: there was a bug in tempest around ordering | 14:42 |
| mriedem-dplgnr | adam_g: pushed a fix last night | 14:42 |
| mriedem-dplgnr | mtrenish-dplgnr: https://review.openstack.org/#/c/116147/ | 14:43 |
| mtrenish-dplgnr | mriedem-dplgnr: would that explain this? http://logs.openstack.org/15/115815/1/check/check-grenade-dsvm/61a9020/logs/grenade.sh.txt.gz#_2014-08-22_05_25_07_843 | 14:43 |
| fungi | looking at a few issues for lxml, it looks like attribute ordering may be influenced by dict ordering internally, but still researching | 14:44 |
| mriedem-dplgnr | mtrenish-dplgnr: yes that was the behvaior we were seeing | 14:44 |
| mriedem-dplgnr | mtrenish-dplgnr: although that specific one should have been fixed by the pythonhashseed=0 backport on tempest | 14:44 |
| mriedem-dplgnr | mtrenish-dplgnr: https://review.openstack.org/#/c/115963/ | 14:45 |
| mriedem-dplgnr | but that only merged 7hrs ago | 14:45 |
| *** mmaglana has joined #openstack-infra | 14:45 | |
| *** masayukig has quit IRC | 14:45 | |
| *** Nikita_Koshikov has quit IRC | 14:45 | |
| stevemar | fungi, mtrenish-dplgnr, so it looks like the *-specs projects usually have a few more than just sphinx/oslosphinx | 14:45 |
| mtrenish-dplgnr | mriedem-dplgnr: yeah when e-r merges my query i think this is our top gate bug atm https://bugs.launchpad.net/cinder/+bug/1360034 | 14:45 |
| uvirtbot | Launchpad bug 1360034 in cinder "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20" [Undecided,New] | 14:45 |
| *** p5ntangle has joined #openstack-infra | 14:46 | |
| mriedem-dplgnr | mtrenish-dplgnr: ok, yeah the backport should have fixed that now | 14:47 |
| mtrenish-dplgnr | so cinder has a latent bug then | 14:47 |
| mtrenish-dplgnr | jgriffith: ^ | 14:47 |
| *** gokrokve has joined #openstack-infra | 14:48 | |
| mtrenish-dplgnr | did clarkb have a bug for the hashseed work he was doing? | 14:48 |
| fungi | mtrenish-dplgnr: yeah, lemme dig it up | 14:49 |
| mriedem-dplgnr | mtrenish-dplgnr: but what is tickling it on the cinder side? because cinder isn't being run with a random hashseed in the gate just tempest | 14:49 |
| *** sandywalsh has quit IRC | 14:50 | |
| mtrenish-dplgnr | mriedem-dplgnr: it's gotta be related, cinder randomly explodes on all branches since 8/20 when doing some minidom parsing of xml | 14:50 |
| mtrenish-dplgnr | of a request | 14:50 |
| mtrenish-dplgnr | that timestamp lines up with the tempest failures | 14:50 |
| mriedem-dplgnr | mtrenish-dplgnr: I'm thinking that it's the same tempest bug | 14:50 |
| mriedem-dplgnr | because with a random hashseed tempest was sending invalid xml | 14:51 |
| mtrenish-dplgnr | oh hrm | 14:51 |
| mriedem-dplgnr | and you know being xml the errors make no sense | 14:51 |
| mtrenish-dplgnr | i'll update that query then | 14:51 |
| *** masayukig has joined #openstack-infra | 14:51 | |
| *** wenlock has joined #openstack-infra | 14:51 | |
| mtrenish-dplgnr | what 'line 1 column 1 is sad' isn't helpful? | 14:51 |
| *** bknudson has quit IRC | 14:52 | |
| fungi | mtrenish-dplgnr: bug 1348818 | 14:52 |
| uvirtbot | Launchpad bug 1348818 in designate "Unittests do not succeed with random PYTHONHASHSEED value" [High,In progress] https://launchpad.net/bugs/1348818 | 14:52 |
| mtrenish-dplgnr | fungi: thanks | 14:52 |
| *** sandywalsh has joined #openstack-infra | 14:52 | |
| mtrenish-dplgnr | checking to see if the cinder fialures have dropped off too | 14:53 |
| *** gokrokve has quit IRC | 14:53 | |
| mtrenish-dplgnr | mriedem-dplgnr: yup, the cinder query drops off this morning | 14:53 |
| mtrenish-dplgnr | sweet | 14:53 |
| zaro | morning | 14:56 |
| zaro | fungi: did you see my questions about modifySsh capability change in scrollback? | 14:57 |
| *** juzuluag has joined #openstack-infra | 14:57 | |
| *** mflobo has quit IRC | 14:58 | |
| *** sandywalsh has quit IRC | 14:58 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Classify tempest lxml hashseed bug 1360034 https://review.openstack.org/116158 | 14:58 |
| uvirtbot | Launchpad bug 1360034 in tempest "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20; lxml random hashseed failure" [Critical,Fix committed] https://launchpad.net/bugs/1360034 | 14:58 |
| *** sdake has quit IRC | 14:59 | |
| fungi | zaro: yes. to summarize my concerns, we want to be able to prevent a user in an account manager role from being able to add/modify the ssh keys and api passwords of administrators, while allowing them to create new system user accounts with ssh keys. also we want to make sure that when they do create new user accounts, they can only add them to groups over which they already have control | 14:59 |
| *** sdake has joined #openstack-infra | 14:59 | |
| *** sandywalsh has joined #openstack-infra | 14:59 | |
| *** jp_at_hp has quit IRC | 14:59 | |
| *** yfried has quit IRC | 14:59 | |
| fungi | zaro: those precautions, however they're accomplished, should hopefully accomplish what we need without risk of privilege escalation | 15:00 |
| *** signed8bit has quit IRC | 15:03 | |
| *** signed8bit has joined #openstack-infra | 15:03 | |
| *** sabeen1 has quit IRC | 15:04 | |
| *** otherwiseguy has joined #openstack-infra | 15:04 | |
| *** sabeen has joined #openstack-infra | 15:04 | |
| *** pblaho has quit IRC | 15:05 | |
| *** rcarrillocruz has quit IRC | 15:06 | |
| *** rcarrillocruz has joined #openstack-infra | 15:06 | |
| *** ihrachyshka has joined #openstack-infra | 15:08 | |
| *** Sukhdev has joined #openstack-infra | 15:09 | |
| *** chuckC has joined #openstack-infra | 15:10 | |
| *** rlandy has quit IRC | 15:11 | |
| *** ArxCruz has quit IRC | 15:12 | |
| *** gokrokve has joined #openstack-infra | 15:12 | |
| *** maelfius has joined #openstack-infra | 15:12 | |
| *** morganfainberg has quit IRC | 15:14 | |
| *** tnurlygayanov has quit IRC | 15:14 | |
| *** tnurlygayanov has joined #openstack-infra | 15:14 | |
| *** maelfius is now known as morganfainberg | 15:14 | |
| zaro | fungi: ok. remove capability for account manager to muck with http password and ssh keys would satify #1 & #2, i have not tested #3 though but i think that's just nesting groups so should work. | 15:15 |
| *** YorikSar has joined #openstack-infra | 15:15 | |
| *** mkollaro has quit IRC | 15:15 | |
| fungi | zaro: right. i suspect the complication, from a pure scoping perspective, is letting them add an ssh key when creating a new account but not letting them change or add an ssh key to an existing (at least administrator, but that distinction seems to be arbitrary based on group membership) account | 15:16 |
| *** tsg has joined #openstack-infra | 15:16 | |
| *** doude has quit IRC | 15:16 | |
| *** doude has joined #openstack-infra | 15:17 | |
| *** mtrenish-dplgnr is now known as mtreinish-dplgnr | 15:17 | |
| *** skolekonov has quit IRC | 15:19 | |
| *** amitpp has joined #openstack-infra | 15:19 | |
| *** reed has joined #openstack-infra | 15:20 | |
| *** dansmith is now known as superdan | 15:20 | |
| *** sdake has quit IRC | 15:21 | |
| openstackgerrit | Steve Martinelli proposed a change to openstack-infra/config: Add zaqar to specs.openstack.org https://review.openstack.org/116310 | 15:21 |
| *** jgrimm has joined #openstack-infra | 15:22 | |
| *** signed8bit has joined #openstack-infra | 15:27 | |
| *** ihrachyshka has quit IRC | 15:28 | |
| *** amitpp has quit IRC | 15:29 | |
| *** tonytan4ever has quit IRC | 15:30 | |
| *** tonytan4ever has joined #openstack-infra | 15:31 | |
| *** baoli has joined #openstack-infra | 15:32 | |
| *** pballand has joined #openstack-infra | 15:32 | |
| *** alaski is now known as lascii | 15:32 | |
| *** ominakov has quit IRC | 15:33 | |
| *** mkollaro has joined #openstack-infra | 15:33 | |
| *** denis_makogon has quit IRC | 15:33 | |
| *** Nikita_Koshikov has joined #openstack-infra | 15:36 | |
| *** AlexF has quit IRC | 15:36 | |
| *** annegentle has quit IRC | 15:36 | |
| *** bknudson has joined #openstack-infra | 15:36 | |
| *** baoli has quit IRC | 15:36 | |
| *** IvanBerezovskiy has left #openstack-infra | 15:36 | |
| *** ihrachyshka has joined #openstack-infra | 15:38 | |
| *** amitpp has joined #openstack-infra | 15:38 | |
| *** msabramo has quit IRC | 15:39 | |
| *** lcheng has joined #openstack-infra | 15:39 | |
| *** _nadya_ has quit IRC | 15:40 | |
| *** ominakov has joined #openstack-infra | 15:40 | |
| *** praneshp has joined #openstack-infra | 15:40 | |
| *** annegentle has joined #openstack-infra | 15:41 | |
| *** ominakov has quit IRC | 15:42 | |
| *** _buzz has quit IRC | 15:43 | |
| *** amotoki has joined #openstack-infra | 15:43 | |
| *** ominakov has joined #openstack-infra | 15:44 | |
| *** dane_leblanc has joined #openstack-infra | 15:44 | |
| *** AlexF has joined #openstack-infra | 15:44 | |
| zaro | fungi: https://gerrit-review.googlesource.com/#/c/59540 | 15:45 |
| *** annegent_ has joined #openstack-infra | 15:45 | |
| *** annegentle has quit IRC | 15:46 | |
| *** ihrachyshka has quit IRC | 15:46 | |
| jeblair | zaro: did you want 0.3.2 and 0.3.3 both released, or just 0.3.3? | 15:46 |
| *** annegent_ has quit IRC | 15:46 | |
| zaro | jeblair: only 0.3.3 | 15:47 |
| *** dimtruck is now known as zz_dimtruck | 15:47 | |
| jeblair | zaro: ok, thx | 15:47 |
| *** gokrokve_ has joined #openstack-infra | 15:48 | |
| *** annegentle has joined #openstack-infra | 15:48 | |
| *** Guest87807 has quit IRC | 15:49 | |
| zaro | jeblair: i'll bump the version to 0.4.0 after you release to signify post-pbr, sound ok? | 15:49 |
| *** MaxV_ has quit IRC | 15:49 | |
| *** MaxV has joined #openstack-infra | 15:50 | |
| jeblair | zaro: sounds good | 15:50 |
| fungi | zaro: post-pbr you don't need to have version info in the repo at all, right? you just tag the version you want when the time comes... | 15:51 |
| *** gokrokve has quit IRC | 15:51 | |
| *** trinaths has joined #openstack-infra | 15:52 | |
| *** vhoward has joined #openstack-infra | 15:52 | |
| *** jlibosva has quit IRC | 15:52 | |
| *** afazekas has quit IRC | 15:53 | |
| *** gokrokve_ has quit IRC | 15:53 | |
| *** MaxV has quit IRC | 15:54 | |
| *** _buzz has joined #openstack-infra | 15:54 | |
| jeblair | yeah, i think the version stuff has been correctly removed; double checking | 15:56 |
| zaro | fungi: sorry, you said what i meant. | 15:56 |
| *** ominakov has quit IRC | 15:56 | |
| fungi | zaro: yep, i agree with your comment on that review too | 15:57 |
| *** ildikov has quit IRC | 15:57 | |
| *** hashar has quit IRC | 15:57 | |
| *** AlexF has quit IRC | 15:57 | |
| *** lcheng has quit IRC | 15:58 | |
| *** KanagarajM has joined #openstack-infra | 15:58 | |
| *** ominakov has joined #openstack-infra | 15:58 | |
| openstackgerrit | Alan Pevec proposed a change to openstack-infra/config: Add periodic Icehouse jobs https://review.openstack.org/115933 | 15:59 |
| zaro | jeblair: do you know where the version info in the jjb doc comes from? http://ci.openstack.org/jenkins-job-builder/ | 15:59 |
| *** Nikita_Koshikov has quit IRC | 15:59 | |
| jeblair | (yes, python-jenkins looks to be correctly pbr-ified, so pushing a tag should work) | 16:00 |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| jeblair | zaro: https://pypi.python.org/pypi/python-jenkins/0.3.3 | 16:00 |
| jeblair | zaro: looking into your question | 16:00 |
| *** alexpilotti has quit IRC | 16:00 | |
| jeblair | zaro: so spinx doc versions are set in the conf.py in the doc dir | 16:01 |
| jeblair | zaro: usually they are set by hand | 16:01 |
| jeblair | zaro: but jjb uses pbr | 16:01 |
| jeblair | doc/source/conf.py:version = jenkins_jobs_version.canonical_version_string() | 16:01 |
| jeblair | doc/source/conf.py:from jenkins_jobs.version import version_info as jenkins_jobs_version | 16:01 |
| jeblair | from pbr.version import VersionInfo | 16:01 |
| jeblair | version_info = VersionInfo('jenkins_jobs') | 16:01 |
| zaro | jeblair: thanks. makes sense | 16:02 |
| *** jaypipes is now known as leakypipes | 16:02 | |
| jeblair | zaro: those last two things are what you need to do to get a version from pbr | 16:02 |
| *** lcheng has joined #openstack-infra | 16:02 | |
| jeblair | zaro: so you could either do it directly in conf.py, or add a version.py file like jjb and zuul have (probably what i'd do) | 16:02 |
| *** rfolco has quit IRC | 16:02 | |
| jeblair | zaro: (that makes it convenient for including in other places, like command line output, etc) | 16:03 |
| *** dizquierdo has quit IRC | 16:05 | |
| *** annegentle has quit IRC | 16:06 | |
| *** amitpp has quit IRC | 16:08 | |
| KanagarajM | jeblair: I have submitted a patch https://review.openstack.org/109494 for new project creation and could you have a look? | 16:09 |
| *** amitpp has joined #openstack-infra | 16:11 | |
| jeblair | KanagarajM: lgtm, thanks for the updates :) | 16:11 |
| jeblair | fungi: can you help me understand this? http://logs.openstack.org/05/116105/2/check/gate-zuul-python26/e242845/console.html | 16:12 |
| *** emagana has joined #openstack-infra | 16:13 | |
| fungi | oh awesome | 16:13 |
| fungi | huh | 16:13 |
| fungi | looks like a busted centos6 vm | 16:13 |
| KanagarajM | jeblair: Thanks. | 16:13 |
| jeblair | fungi: in the way that you were looking into last week? | 16:14 |
| *** ominakov has quit IRC | 16:14 | |
| fungi | jeblair: maybe, though this is a slightly different manifestation than i saw before. more likely this is one which didn't get properly aborted on some bad puppet | 16:14 |
| fungi | an image update i mean | 16:14 |
| jeblair | ah ok | 16:14 |
| clarkb | fungi: because puppet returned eg 1? | 16:15 |
| fungi | i'll see if any other nodes booted from the same snapshot have this behavior (if it's possible to find one still) | 16:15 |
| jeblair | fungi: i'll look into the image build logs | 16:15 |
| fungi | clarkb: right, saw a similar behavior with the bundler conflict | 16:15 |
| *** mwagner_lap has quit IRC | 16:16 | |
| fungi | jeblair: the image update logs may not help. we haven't done automatic image updates since the 19th, so this *could* have been booted from a manually updated image | 16:16 |
| *** pballand has quit IRC | 16:16 | |
| clarkb | do we have image builds working again (we restarted nodepool after fungi's fix) | 16:16 |
| jeblair | fungi: i see that now :) | 16:16 |
| *** AlexF has joined #openstack-infra | 16:16 | |
| fungi | clarkb: we do have the patch in, but i didn't restart nodepool until after today's would have triggered | 16:16 |
| fungi | i was hoping to catch it last night during a lull but there was never a lull while i was around | 16:17 |
| fungi | so i finally gave in and did it this morning, except i misremembered the start time and did it about 45 minutes too late to have helped for today | 16:17 |
| jeblair | so if it's a bad image, it could still be the current one | 16:18 |
| clarkb | fungi: so the image above would have had to be from my rebuidls on the 20th? | 16:18 |
| *** Sukhdev has quit IRC | 16:18 | |
| fungi | jeblair: right | 16:18 |
| fungi | clarkb: probably, yes | 16:18 |
| openstackgerrit | Matt Kovacs proposed a change to openstack-infra/config: Adding project saml-generator https://review.openstack.org/110378 | 16:19 |
| fungi | that node was created 2014-08-21 23:49:41 according to the nodepool log | 16:19 |
| clarkb | hrm thats a day later | 16:20 |
| clarkb | (I ask ebacuse I was trying to watch the builds for issues after the bundler thing and don't recall anything) | 16:20 |
| fungi | newest image for that is 43.90 hours old | 16:20 |
| jeblair | oh interesting, the new python-jenkins has broken the layout compare script | 16:20 |
| jeblair | i'm going to dig into that | 16:21 |
| clarkb | fungi: ya ok so it lines up | 16:21 |
| fungi | so that's what it should have booted from, suggesting that if it's a problem image then other current nodes of that type and region should also be screwed | 16:21 |
| clarkb | fungi: I don't recall seeing any errors in that build | 16:21 |
| jeblair | zaro: ping | 16:21 |
| *** signed8bit is now known as signed8bit_ZZZzz | 16:22 | |
| jeblair | zaro: python-jenkins 0.3.3 is broken; it can't pip install | 16:22 |
| jeblair | File "/tmp/foo/build/python-jenkins/setup.py", line 5, in <module> | 16:22 |
| jeblair | with open('requirements.txt') as f: | 16:22 |
| jeblair | IOError: [Errno 2] No such file or directory: 'requirements.txt' | 16:22 |
| fungi | i've held 104.130.128.96 to investigate further | 16:22 |
| *** signed8bit_ZZZzz is now known as signed8bit | 16:22 | |
| zaro | jeblair: i think that was fixed. | 16:23 |
| *** AlexF has quit IRC | 16:23 | |
| *** pballand has joined #openstack-infra | 16:23 | |
| * zaro checks | 16:23 | |
| *** rfolco has joined #openstack-infra | 16:23 | |
| fungi | currently running this job https://jenkins05.openstack.org/job/gate-ironic-python26/762/console | 16:23 |
| clarkb | fungi: it did rpm: command not found in rax dfw | 16:23 |
| clarkb | fungi: maybe we are seeing that behavior come back | 16:23 |
| jeblair | zaro: https://github.com/stackforge/python-jenkins/blob/0.3.3/setup.py#L5 | 16:23 |
| *** ramashri has joined #openstack-infra | 16:23 | |
| fungi | clarkb: yeah, 'which rpm' works fine on the held node from the same image | 16:24 |
| fungi | and /etc/sudoers.d/jenkins-sudo exists on the held node too | 16:25 |
| *** mcgin has quit IRC | 16:25 | |
| fungi | checking my logstash queries for those three symptom bugs now | 16:25 |
| zaro | jeblair: this was the fix http://git.openstack.org/cgit/stackforge/python-jenkins/commit/?h=3.0-fixes&id=4e8eae6c5b897f46ba35f5dc7d2aa077924ff27e | 16:25 |
| fungi | yep, two of the three queries start returning hits again as of the 20th | 16:26 |
| jeblair | zaro: yeah, except that depends on requirements.txt being in the current directory | 16:26 |
| jeblair | zaro: i will propose a very simple fix | 16:26 |
| fungi | phschwartz: our mystery broken snapshot boots have returned in dfw as of two days ago | 16:27 |
| clarkb | fungi: maybe it went away because we stopped building new images | 16:27 |
| clarkb | fungi: and happened to have images without the symptoms | 16:27 |
| clarkb | fungi: as a hack workaround we can `which rpm` in the nodepool ready scripts | 16:27 |
| *** jpich has quit IRC | 16:27 | |
| phschwartz | fungi: Can you get me some id's so I can look at them? I am in Sat and can bug people directly today | 16:27 |
| fungi | clarkb: well, it had subsided a week ago, and we were still autobuilding new images at the time | 16:27 |
| fungi | phschwartz: you bet | 16:28 |
| openstackgerrit | Kevin L. Mitchell proposed a change to stackforge/python-jenkins: Add MANIFEST.in https://review.openstack.org/116327 | 16:29 |
| *** packet has joined #openstack-infra | 16:30 | |
| jeblair | zaro: https://review.openstack.org/116328 https://review.openstack.org/116329 | 16:30 |
| fungi | clarkb: the queries i've been using as a litmus test... http://paste.openstack.org/show/98810/ | 16:30 |
| clarkb | fungi: thanks | 16:31 |
| jeblair | zaro, clarkb, fungi: the config repo is dead in the water until the python-jenkins situation is resolved. | 16:31 |
| fungi | clarkb: the timing and pattern suggests that they may be booting from your discarded/failed image updates on the 19th/20th? | 16:31 |
| clarkb | jeblair: rgr, reviewing those changes now | 16:31 |
| fungi | jeblair: oh, amazing. switching gears | 16:32 |
| *** chandankumar has joined #openstack-infra | 16:32 | |
| adam_g | fungi, reading backlog.. the xml issues i was looking at in tempest yesterday all came down dict generated xml strings on the client side | 16:32 |
| jeblair | zaro, clarkb, fungi: do we: a) force the python-jenkins changes through and push up 0.3.4 to make it resolved quickly; b) push through package pins in jjb to make it resolve quickly; c) let it resolve slowly and start beer early ? | 16:32 |
| *** baoli has joined #openstack-infra | 16:33 | |
| clarkb | jeblair: its only 9am so beer this early is probably a bad idea :) both of your changes lgtm and I would be ok with forcing though and pushing 0.3.4 | 16:33 |
| fungi | adam_g: yep, that sounds like what i would suspect for the random hashseed impact | 16:33 |
| jeblair | zaro: does that sound good to you? | 16:33 |
| zaro | jeblair: ++ | 16:34 |
| fungi | jeblair: i would go for force and not lose a friday to drunken revelry | 16:34 |
| *** openstackgerrit has quit IRC | 16:34 | |
| jeblair | i am disappointed in you both | 16:34 |
| *** derekh has quit IRC | 16:34 | |
| fungi | i reserve my fridays for drunken working instead | 16:34 |
| clarkb | jeblair: it is the life of a coremudgeon | 16:35 |
| jeblair | zaro: okay those changes are merged; do you want to tag 0.3.4 now? | 16:35 |
| * fungi correlates introduction of bugs to specific weekdays to confirm | 16:35 | |
| zaro | jeblair: will do. | 16:35 |
| *** bhuvan has joined #openstack-infra | 16:35 | |
| jeblair | zaro: cool, let me know when you've pushed it up | 16:35 |
| * jeblair prepares to do the pypi dance again | 16:36 | |
| fungi | clarkb: so i think what has happened is that dfw eventually flushed out older stale/broken snapshots, but then the puppet issues around bundler earlier this week added new ones. it still very much seems to support the possibility that dfw is booting from different snapshots than it claims to be at times, randomly | 16:36 |
| clarkb | fungi: and it starts right there on 8-20 at least for the one query I ran | 16:37 |
| * fungi nods | 16:37 | |
| jeblair | our occasionally producing broken images appears to be a critical debugging tool | 16:37 |
| *** baoli has quit IRC | 16:37 | |
| *** zz_dimtruck is now known as dimtruck | 16:37 | |
| *** gokrokve has joined #openstack-infra | 16:38 | |
| fungi | according to phschwartz their logs indicate that nova *claims* to have booted from the same snapshot uuid as we asked, which suggests the broken is at a lower level | 16:38 |
| phschwartz | fungi: From what we saw, correct | 16:39 |
| jeblair | i was about to ask if there was a tempest test that covers this, but since tempest is black-box, it wouldn't know (it would see no more than what phschwartz has seen) | 16:39 |
| jeblair | though i suppose it could test for the existence of a file or something | 16:39 |
| jeblair | (after actually logging into a node booted from an image) | 16:39 |
| phschwartz | Let me see if I can get our glance guys looking again | 16:39 |
| fungi | phschwartz: i'll get you a list of server uuids here shortly and try to correlate them to image uuids | 16:40 |
| phschwartz | fungi: ty | 16:41 |
| mriedem-dplgnr | jeblair: tempest test for what? | 16:41 |
| *** MaxV has joined #openstack-infra | 16:41 | |
| *** dane_leblanc has quit IRC | 16:42 | |
| clarkb | mriedem-dplgnr: that nick is really confusing btw | 16:42 |
| clarkb | mriedem-dplgnr: checking that glance + nova booted the image you wnted them to boot | 16:42 |
| fungi | clarkb: it's no more confusing than mtreinish-dplgnr is | 16:42 |
| jeblair | was just asking in -qa | 16:43 |
| mriedem-dplgnr | heh | 16:43 |
| *** alexpilotti has joined #openstack-infra | 16:44 | |
| *** msabramo has joined #openstack-infra | 16:46 | |
| *** AlexF has joined #openstack-infra | 16:47 | |
| *** dimtruck is now known as zz_dimtruck | 16:47 | |
| *** Ryan_Lane has joined #openstack-infra | 16:48 | |
| KanagarajM | jeblair: gate job is failed for the review https://review.openstack.org/#/c/109494 . so i tired to give comment 'reverify no bug' but the jobs land in the check queue, but was expending on the gate queue. do i need to use some other comment? | 16:50 |
| *** imcsk8 has quit IRC | 16:50 | |
| *** imcsk8 has joined #openstack-infra | 16:50 | |
| jeblair | KanagarajM: that's normal; it has to have a passing jenkins vote to get into the gate queue, so it goes back to check to get one. one of our jobs is currently broken, and we're working on fixing it. | 16:51 |
| jeblair | KanagarajM: recheck will not help | 16:51 |
| jeblair | KanagarajM: in general, you should only use recheck when you understand the problem and believe that it will not happen again | 16:51 |
| jeblair | KanagarajM: it's not to be used to just keep trying until it works | 16:51 |
| zaro | jeblair: did something change, this command no longer works: git push gerrit VERSION_NUMBER | 16:52 |
| zaro | jeblair: fatal: 'gerrit' does not appear to be a git repository | 16:52 |
| jeblair | zaro: that's a local git error, it probably means you don't have a remote set up for gerrit. maybe you need to run 'git review -s' | 16:53 |
| fungi | zaro: do you have a gerrit remote? if this is a fresh clone of the repo and you didn't create a gerrit remote (for example by running git review, maybe with just the -s option) then it won't exist yet | 16:53 |
| fungi | er, ditto to what jeblair just said ;) | 16:53 |
| *** bknudson has quit IRC | 16:54 | |
| *** maxbit has quit IRC | 16:54 | |
| zaro | i have a .gitreview file with gerrit remote setup. | 16:54 |
| jeblair | zaro: paste the output of 'git remote -v' | 16:55 |
| zaro | ohh i forgot to run git review -s after creating the .gitreview | 16:55 |
| fungi | zaro: it's the .git/config which matters in this case, not .gitreview | 16:55 |
| zaro | just pushed the tag up. | 16:55 |
| *** bdpayne has joined #openstack-infra | 16:55 | |
| fungi | .gitreview is only read by git-review, not by git push | 16:55 |
| KanagarajM | jeblair: sure | 16:56 |
| fungi | git push is looking at your ~/.gitconfig and .git/config | 16:56 |
| KanagarajM | jeblair: once check jenkins gives +1, will gate job will trigger automatically in this case? | 16:57 |
| jeblair | zaro: 0.3.4 is uploaded and installs | 16:58 |
| *** armax has quit IRC | 16:59 | |
| zaro | jeblair: thanks. msabramo, mrodden ^ | 17:00 |
| adam_g | i heard from a bird its possible to get gate fixing things bumped in the queue. if so, is it possible for https://review.openstack.org/#/c/116103/ to cut? been in queue for 7 hrs, ironics gate is blocked on it | 17:00 |
| *** comstud is now known as bearhands | 17:02 | |
| jeblair | adam_g: it's possible to get changes that fix nondeterministic errors in the integrated gate promoted | 17:03 |
| jeblair | adam_g: since that isn't blocking other projects, and any changes to ironic which are enqueued behind it will benefit from it, i'm not sure it fits the bill (but we're getting into a grey area for sure) | 17:04 |
| fungi | (and also in unusually bad situations, changes which improve debuggability/identification of nondeterministic failures) | 17:04 |
| adam_g | jeblair, ah, ok. no problem. | 17:05 |
| fungi | aside from changes related to high-frequency nondeterministic failures and failures impacting gating for the integrated gate, the only other exceptions we generally consider are for severe security fixes and addressing legal issues | 17:06 |
| jeblair | adam_g: it's a moderately disruptive process (it causes a gate reset, so we try not to use it unless it's clearly worth it), and we also just don't want to be in the business of handing out indulgences (which is why we try to adopt a pretty clear policy around it) | 17:07 |
| fungi | also, if you hold your teeth right, that change could merge in less than an hour by zuul's estimate (you just need the 9 ahead of it to also merge successfully) | 17:07 |
| jeblair | the best case now is currently better than the worst case if we promoted it (because it would have to wait for nodes) | 17:08 |
| *** mkollaro has quit IRC | 17:09 | |
| *** e0ne has quit IRC | 17:09 | |
| jeblair | clarkb: i like zaneb's suggestion for the -1 message: "Today, I have chosen to obstruct your work for the greater good of the project" :) | 17:10 |
| adam_g | fungi, jeblair thanks for the explanation. | 17:10 |
| jeblair | somehow i doubt we're going to come to consensus on that one though | 17:10 |
| fungi | i too found that suggestion pretty awesome | 17:11 |
| *** amitpp has quit IRC | 17:11 | |
| clarkb | jeblair: we are utilitarian | 17:12 |
| fungi | phschwartz: so for a start, just to sanity check, i have logs from two instances which we *believe* should have booted from snapshot 4321371d-aad5-45ac-832d-1da4a947d316 | 17:12 |
| *** mkollaro has joined #openstack-infra | 17:12 | |
| fungi | phschwartz: 8ece49ec-ae4b-4224-8c56-cf8efa49b5b3 booted around 2014-08-21 23:49:42 and seems to have been broken in such a way that it looks like it could have booted from a different snapshot entirely | 17:13 |
| fungi | phschwartz: while c8390302-7617-46ff-8b35-12789d622819 booted around 2014-08-22 16:14:56 from that snapshot and seems to have been just fine | 17:13 |
| *** atiwari has joined #openstack-infra | 17:15 | |
| *** trinaths has quit IRC | 17:18 | |
| phschwartz | fungi: hmm, they appear to have both been booted from the same image | 17:18 |
| fungi | phschwartz: http://paste.openstack.org/show/98837/ is a list of 8 instance uuids with their respective approximate boot times which acted like they were not booted from the snapshots we requested (in fact all seem to have booted from older snapshots we had already deleted) | 17:19 |
| *** vponomaryov has quit IRC | 17:19 | |
| fungi | phschwartz: and as before, we continue to only see this in dfw, nowhere else | 17:19 |
| *** Longgeek has quit IRC | 17:19 | |
| *** gyee has joined #openstack-infra | 17:20 | |
| *** Longgeek has joined #openstack-infra | 17:21 | |
| phschwartz | fungi: Let me dig some more, but it looks like at least the first 2 you gave me both booted from the same image, 1 was fine and 1 wasn't which is weird as the base should have been the same | 17:21 |
| fungi | phschwartz: yep. best guess is that for some reason nova/glance think they're handing out a different snapshot than they actually are under some weird and rare circumstances, and so your logs are lying to you | 17:22 |
| *** packet has quit IRC | 17:22 | |
| phschwartz | fugni: yes which makes finding the issue really really hard | 17:22 |
| fungi | phschwartz: particularly, it seems like they're actually booting from snapshots which should no longer exist because we deleted them | 17:23 |
| fungi | though unfortunately i don't think we have any record of what those older snapshot uuids would have been (if we do, i'm unable to figure them out from the nodepool debug log) | 17:24 |
| *** dimsum_ has quit IRC | 17:24 | |
| *** dimsum_ has joined #openstack-infra | 17:25 | |
| leakypipes | anteaya: afternoon, ma'am :) I have a couple colleagues wondering about the 3rd party CI account request for the Coraid cinder CI system. If you have a few moments, feel free to ping me on any status you might know of on that and if they or I need do anything. Thanks in advance! :) | 17:26 |
| *** marun has joined #openstack-infra | 17:27 | |
| *** ArxCruz has joined #openstack-infra | 17:27 | |
| *** Nikita_Koshikov has joined #openstack-infra | 17:28 | |
| *** AlexF has quit IRC | 17:28 | |
| jeblair | fungi: https://review.openstack.org/116349 | 17:28 |
| pleia2 | clarkb: so both our pootle bugs are fixed in git, but it turns out rc1 is really buggy, I'm thinking we go with stable/2.6.0 as suggested here: https://github.com/translate/pootle/issues/3250#issuecomment-53049277 | 17:28 |
| jeblair | fungi: i wonder if that would help | 17:28 |
| * pleia2 frowns at "very buggy" RC | 17:29 | |
| *** tonytan4ever has quit IRC | 17:29 | |
| clarkb | pleia2: :( ok so I should give installing from tip of that stable branch a go? | 17:29 |
| clarkb | pleia2: I can probably do that today | 17:29 |
| pleia2 | clarkb: yeah, that'd be great | 17:29 |
| *** esker has joined #openstack-infra | 17:29 | |
| *** dimsum_ has quit IRC | 17:29 | |
| *** openstackgerrit has joined #openstack-infra | 17:30 | |
| pleia2 | then by Monday we should have both Pootle 2.6 and Zanata w/ Wildfly demos up and running to test \o/ | 17:30 |
| *** pelix has quit IRC | 17:31 | |
| *** Nikita_Koshikov has quit IRC | 17:32 | |
| fungi | jeblair: possibly improvement commented on the review. if that turns out not to be feasible though, i'm +2 on the idea | 17:33 |
| *** zul has quit IRC | 17:33 | |
| *** alkari has quit IRC | 17:33 | |
| *** baoli has joined #openstack-infra | 17:33 | |
| clarkb | fungi: jeblair: should we also try a `which yum && which rpm` ready script check? I suppose that will prevent those nodes from running the unittests with the echos though | 17:35 |
| *** alkari has joined #openstack-infra | 17:35 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:35 | |
| fungi | clarkb: i think that would likely only catch the ones we've seen which are missing those files or have a screwy path envvar | 17:36 |
| clarkb | fungi: oh good point | 17:36 |
| fungi | clarkb: also, it would allow us, i think, to not notice we had bad images | 17:36 |
| jeblair | fungi: i think your suggestion is a good idea, but i think i'd like to implement that in nodepool itself rather than our scripts | 17:36 |
| *** harlowja has joined #openstack-infra | 17:36 | |
| jeblair | fungi: i was thinking the quick hacky thing could get through and running soonish (at least, it's the fastest of any option we have around here) | 17:36 |
| fungi | jeblair: fair enough. we could do this first for sure | 17:37 |
| jeblair | fungi: and maybe we could swing back later and have nodepool itself do a better job | 17:37 |
| jeblair | (maybe even include more information) | 17:37 |
| fungi | jeblair: yep, totally agree | 17:37 |
| *** baoli has quit IRC | 17:38 | |
| *** tonytan4ever has joined #openstack-infra | 17:39 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:40 | |
| *** tgohad has joined #openstack-infra | 17:40 | |
| *** tsg has quit IRC | 17:40 | |
| *** peluse has quit IRC | 17:41 | |
| *** masayukig has quit IRC | 17:41 | |
| *** peluse has joined #openstack-infra | 17:42 | |
| *** armax has joined #openstack-infra | 17:42 | |
| jgriffith | clarkb: ping | 17:44 |
| *** amotoki has quit IRC | 17:44 | |
| clarkb | jgriffith: pong | 17:45 |
| clarkb | jeblair: fungi: the only other thing I have to add there is maybe the echo should be done in run-docs too? | 17:45 |
| jgriffith | clarkb: hey...just saw your email about tox. Trying to figure out how to "fix" | 17:45 |
| clarkb | jeblair: fungi: though unittests are the biggest offender right? | 17:45 |
| jgriffith | unit tests that call setUp twice, via super | 17:45 |
| fungi | clarkb: well, and we totally miss it on devstack-gate jobs | 17:45 |
| clarkb | jgriffith: the actual fix is to not treat dicts as being ordered. the testtools thing is independent and probably means finding all the occurences where setUp is called twice | 17:46 |
| fungi | clarkb: i took the point as to just get some coverage for now so we have at least a sample we can correlate | 17:46 |
| jeblair | fungi, clarkb: the other common run- scripts have -e set and i didn't want to bother/risk screwing something up | 17:46 |
| *** esker has quit IRC | 17:46 | |
| clarkb | jeblair: ok works for me then | 17:46 |
| *** leakypipes has quit IRC | 17:46 | |
| jeblair | btw, i restarted openstackgerrit | 17:46 |
| *** masayukig has joined #openstack-infra | 17:46 | |
| jgriffith | clarkb: well.. yeah but; Oh, hold on a sec | 17:47 |
| fungi | oh, had it gone to lunch? | 17:47 |
| jgriffith | clarkb: it's actually not the setUp... I might have something. Never mind | 17:47 |
| jgriffith | clarkb: ahhh... I see what they did. addCleanup that calls out to a method that has another tearDown :( | 17:49 |
| jgriffith | clarkb: ok... sorry for the distraction. I think I've got this fixed up | 17:49 |
| clarkb | jgriffith: no problem. It is a bit confusing with several things going on at once | 17:49 |
| *** packet has joined #openstack-infra | 17:50 | |
| *** ominakov has joined #openstack-infra | 17:51 | |
| *** Sukhdev has joined #openstack-infra | 17:52 | |
| *** rcarrillocruz has quit IRC | 17:53 | |
| *** amitpp has joined #openstack-infra | 17:54 | |
| *** zz_gondoi is now known as gondoi | 17:54 | |
| *** packet has quit IRC | 17:55 | |
| *** Ryan_Lane1 has joined #openstack-infra | 17:55 | |
| asselin | Hi I periodically run into this issue where zuul cannot be restarted. any ideas? cat: /var/run/zuul/zuul.pid: No such file or directory | 17:57 |
| *** rcarrillocruz has joined #openstack-infra | 17:57 | |
| *** amitpp has quit IRC | 17:58 | |
| *** bknudson has joined #openstack-infra | 17:58 | |
| clarkb | asselin: if the pid isn't there that usually means zuul died and cleaned up the pid for some other reason | 18:00 |
| clarkb | asselin: in that case you can just start the service | 18:00 |
| clarkb | asselin: but I think we want to debug why it would die | 18:00 |
| asselin | clarkb, zuul is still running. so somehow that pid file is getting deleted....? | 18:00 |
| clarkb | asselin: ya, the pid file shouldn't be deleted if zuul is still running | 18:01 |
| *** tgohad has quit IRC | 18:01 | |
| *** ominakov has quit IRC | 18:01 | |
| *** tgohad has joined #openstack-infra | 18:01 | |
| asselin | clarkb, so maybe there's something going on with my puppet script that updates/restarts zuul and causes this issue....?? | 18:02 |
| clarkb | asselin: that could be | 18:02 |
| clarkb | or maybe with an init script if puppet uses that | 18:02 |
| clarkb | asselin: I would double check that /var/run/zuul exists and that it is indeed empty | 18:03 |
| asselin | it is empty | 18:03 |
| *** ominakov has joined #openstack-infra | 18:03 | |
| *** MaxV has quit IRC | 18:07 | |
| clarkb | I need to afk for a bit to move my atoms from home to office | 18:08 |
| *** leakypipes has joined #openstack-infra | 18:08 | |
| fungi | technically you're moving through spacetime in an attempt to intersect with where your office will be | 18:09 |
| nibalizer | clarkb: technically they were already there, just with very low probability | 18:10 |
| fungi | touché | 18:10 |
| *** marun has quit IRC | 18:10 | |
| phschwartz | lol | 18:11 |
| Vek | haha :) | 18:12 |
| Vek | actually, the size of your wave function would probably have to be significantly larger than the current size of the universe, so the probability is essentially nonexistant, but... | 18:13 |
| * fungi thinks we should begin a new tradition... cosmology fridays | 18:15 | |
| Vek | well, we haven't even begun talking about cosmology. You'd have to start talking about things like inflation to get there :) | 18:15 |
| fungi | that will require substantially more beer | 18:15 |
| * Vek imagines pickled mushrooms when fungi starts talking about beer... | 18:16 | |
| *** mestery has joined #openstack-infra | 18:16 | |
| adam_g | is possible to show CI comments in gerrit reviews by default? | 18:17 |
| fungi | Vek: if we can verge into biology fridays for a moment, yeast *is* technically fungi ;) | 18:17 |
| Vek | hehe, point :) | 18:17 |
| fungi | adam_g: possibly with some javascript and something like greasemonkey | 18:17 |
| adam_g | fungi, ah | 18:17 |
| fungi | adam_g: all that toggle button is doing is setting a js variable i believe | 18:17 |
| adam_g | cool | 18:18 |
| *** markmcclain has joined #openstack-infra | 18:20 | |
| *** e0ne has joined #openstack-infra | 18:20 | |
| *** e0ne has quit IRC | 18:21 | |
| fungi | adam_g: if the need is to be able to see when the most recent results were posted, on the other hand, that might be something which could just be inserted into the table header for each ci | 18:21 |
| fungi | (by the current javascript) | 18:21 |
| *** _nadya_ has joined #openstack-infra | 18:21 | |
| fungi | adam_g: what specifically are you missing that you want to see all the ci comments by default? | 18:21 |
| *** sabeen1 has joined #openstack-infra | 18:22 | |
| *** sabeen has quit IRC | 18:23 | |
| *** mwagner_lap has joined #openstack-infra | 18:24 | |
| adam_g | fungi, lately i've been doing a lot of searching history for when things broke and would prefer to have the full CI history displayed when looking at a review. | 18:24 |
| adam_g | fungi, for typical usage the new display is great | 18:24 |
| *** emagana has quit IRC | 18:24 | |
| *** alkari1 has joined #openstack-infra | 18:25 | |
| *** emagana has joined #openstack-infra | 18:25 | |
| *** annegentle has joined #openstack-infra | 18:25 | |
| *** alkari has quit IRC | 18:25 | |
| *** dimsum_ has joined #openstack-infra | 18:25 | |
| *** e0ne has joined #openstack-infra | 18:25 | |
| *** tomoe__ has quit IRC | 18:26 | |
| *** chandankumar has quit IRC | 18:28 | |
| *** emagana has quit IRC | 18:29 | |
| *** emagana has joined #openstack-infra | 18:29 | |
| *** dimsum_ has quit IRC | 18:30 | |
| *** e0ne has quit IRC | 18:32 | |
| *** cnesa7 has quit IRC | 18:32 | |
| clarkb | adam_g: fungi: the toggle ci button should probably save a cookie that retains that state across changes for you | 18:33 |
| clarkb | similar to the way zuul status does searches and expansion of boxes | 18:33 |
| *** baoli has joined #openstack-infra | 18:34 | |
| *** e0ne has joined #openstack-infra | 18:36 | |
| openstackgerrit | Dolph Mathews proposed a change to openstack/requirements: bump pycadf to 0.6.0 https://review.openstack.org/116360 | 18:37 |
| *** zz_dimtruck is now known as dimtruck | 18:38 | |
| jeblair | well, also this should all just be in gerrit :) | 18:38 |
| openstackgerrit | Dolph Mathews proposed a change to openstack/requirements: bump pycadf to 0.6.0 https://review.openstack.org/116360 | 18:38 |
| *** doude has quit IRC | 18:39 | |
| *** baoli has quit IRC | 18:39 | |
| *** RicoLin has quit IRC | 18:39 | |
| *** ociuhandu has quit IRC | 18:40 | |
| clarkb | apaprently I am bad at afking /me tries again | 18:42 |
| *** Sukhdev has quit IRC | 18:43 | |
| *** _nadya_ has quit IRC | 18:43 | |
| *** e0ne has quit IRC | 18:45 | |
| *** zul has joined #openstack-infra | 18:46 | |
| *** dane_leblanc has joined #openstack-infra | 18:47 | |
| *** rpodolyaka1 has joined #openstack-infra | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:49 | |
| *** dimtruck is now known as zz_dimtruck | 18:49 | |
| *** ildikov has joined #openstack-infra | 18:51 | |
| pleia2 | jeblair: did you see your puppet? | 18:51 |
| fungi | though mordred's ping heart-shaped rockstar sunglasses were hard to top | 18:52 |
| fungi | s/ping/pink/ | 18:52 |
| pleia2 | yeah, mordred is the funner puppet | 18:52 |
| *** dane_leblanc has quit IRC | 18:53 | |
| jeblair | i uh. what? | 18:54 |
| *** ominakov has quit IRC | 18:54 | |
| pleia2 | jeblair: https://www.flickr.com/photos/pleia2/14808962909/ | 18:54 |
| pleia2 | mordred and jeblair puppets will tell the story of open sourcing our infrastructure at puppetconf | 18:54 |
| jeblair | that's wonderful! | 18:54 |
| pleia2 | :) | 18:55 |
| *** prad has quit IRC | 18:55 | |
| *** alexpilotti has quit IRC | 18:56 | |
| *** radez_g0n3 is now known as radez | 18:57 | |
| *** ominakov has joined #openstack-infra | 18:57 | |
| *** annegentle has quit IRC | 18:59 | |
| *** annegentle has joined #openstack-infra | 18:59 | |
| *** dane_leblanc has joined #openstack-infra | 19:00 | |
| *** e0ne has quit IRC | 19:01 | |
| *** dimsum_ has joined #openstack-infra | 19:02 | |
| *** KanagarajM has quit IRC | 19:03 | |
| *** ihrachyshka has joined #openstack-infra | 19:03 | |
| *** Sukhdev has joined #openstack-infra | 19:04 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat UserKeyPairMissing bug 1360415 https://review.openstack.org/116363 | 19:06 |
| uvirtbot | Launchpad bug 1360415 in heat "test_nova_keypair_resources fails with UserKeyPairMissing" [Undecided,New] https://launchpad.net/bugs/1360415 | 19:06 |
| *** HeOS has quit IRC | 19:07 | |
| *** rpodolyaka1 has quit IRC | 19:09 | |
| *** emagana has quit IRC | 19:11 | |
| *** zz_dimtruck is now known as dimtruck | 19:12 | |
| *** emagana has joined #openstack-infra | 19:12 | |
| markvan | fungi: hi, did you ever get a chance to "...compare .*ruby.* packages installed on rax vs hpcloud and see if that turns up any leads"? | 19:12 |
| fungi | markvan: they seemed to be the same | 19:12 |
| fungi | markvan: i take it you're still seeing issues in rax but not hpcloud for those jobs? | 19:13 |
| markvan | fungi: yup, still hitting the issue: http://logs.openstack.org/45/112945/2/check/gate-cookbook-openstack-identity-chef-style/43d4241/console.html | 19:14 |
| fungi | markvan: just a sec, i'll get you a diff of the package list for what's installed on ubuntu trusty workers in hpcloud vs rax | 19:15 |
| markvan | k | 19:15 |
| *** hashar has joined #openstack-infra | 19:16 | |
| *** emagana has quit IRC | 19:16 | |
| markvan | fungi: also seeing that the bundle install order always changes, not sure if that has anything to do with issue, but I have https://review.openstack.org/#/c/114593/ to try to calm that down a bit. | 19:18 |
| *** markmcclain has quit IRC | 19:18 | |
| *** rpodolyaka1 has joined #openstack-infra | 19:21 | |
| jeblair | r1chardj0n3s: is the missing-reqs job expected to pass? | 19:21 |
| clarkb | jeblair: my understanding last night was that it was initially passing when it should fail, but now it fails correctly | 19:22 |
| clarkb | jeblair: nova doens't list a requirement that it depends on | 19:22 |
| jeblair | is silent the correct place for it? | 19:22 |
| clarkb | jeblair: probably not. The fix is "simple" we just need to edit nova's requirements then we can make the job voting | 19:23 |
| *** SumitNaiksatam has quit IRC | 19:23 | |
| clarkb | jeblair: so probably reasonable to make it a check job non voting then switch to voting when nova's requirements are correct | 19:23 |
| *** mkollaro has quit IRC | 19:24 | |
| jeblair | well, that's not much different than silent from a "we're using 37 nodes to run that test" pov | 19:24 |
| jeblair | if we're about to turn it on, that's fine. i just didn't want to run jobs for no reason | 19:24 |
| clarkb | jeblair: thats true, but it may encourage people to fix it? though I assume r1chardj0n3s is already working on that | 19:24 |
| clarkb | I think it is something we should have turned on. Though now that I think of it it should probably go in the "pep8" job | 19:25 |
| clarkb | since it is a static checker of our requirements | 19:25 |
| jeblair | clarkb: or the requirements job? | 19:25 |
| fungi | markvan: http://paste.openstack.org/show/98871/ | 19:25 |
| jeblair | but yeah, shortly after we latch it voting, we should fold it into something; doesn't seem like it should need its own node | 19:26 |
| clarkb | jeblair: ya or the reqs job | 19:26 |
| clarkb | jeblair: ++ | 19:26 |
| clarkb | jeblair: fungi: for 116176 I figure it can ferment over the weekend then early next week I can put it in place (assuming that there is no major disagreement with the change itself) | 19:27 |
| fungi | jeblair: clarkb: yep, already talked to r1chardj0n3s about that. probably add it to the unit tests or something | 19:27 |
| fungi | clarkb: sounds good. in a similar vein, 115759 needs to be tested out | 19:27 |
| *** radez is now known as radez_g0n3 | 19:27 | |
| clarkb | fungi: reviewed | 19:30 |
| clarkb | jeblair: any chance you want to review https://review.openstack.org/#/c/115836/ so that we can get that into the centos6 image builds too? | 19:31 |
| clarkb | pleia2: ^ sorry I didn't provide an example cfg file that would have been a good thing to add to the comments | 19:32 |
| clarkb | mattoliverau: thank you for providing that info | 19:32 |
| clarkb | also I love that the file says "do not edit". I am the sysadmin I shall edit all the files ! | 19:33 |
| *** Hefeweizen has quit IRC | 19:34 | |
| fungi | i'm going to go grab an early dinner before the last of the tourists descend on the brewery, then pitch in once i'm back | 19:35 |
| fungi | bbiaw | 19:35 |
| jeblair | pleia2: i showed your puppets to li.. she said "what's that? oh, well, one of them is you... the one with the hat... but who's the other one... oh, well, the jacket is sparkly, could that be monty?" then i clicked over to the picture with the glasses and she said "oh, yes! that's monty" | 19:35 |
| *** baoli has joined #openstack-infra | 19:35 | |
| *** morganfainberg is now known as morganfainberg_Z | 19:35 | |
| jeblair | pleia2: so, um, i'd say they did pretty well in blind user testing (i gave her no introduction before showing the pics) | 19:35 |
| *** sdake_ has quit IRC | 19:36 | |
| *** sdake has joined #openstack-infra | 19:36 | |
| *** sdake has quit IRC | 19:36 | |
| *** sdake has joined #openstack-infra | 19:36 | |
| *** baoli has quit IRC | 19:40 | |
| *** hdd has quit IRC | 19:40 | |
| *** hdd has joined #openstack-infra | 19:41 | |
| *** mestery has quit IRC | 19:42 | |
| Sukhdev | folks, is this correct IRC to ask devstack related questions? | 19:44 |
| clarkb | Sukhdev: I think usually we point people to the qa channel but here works too | 19:45 |
| *** vhoward has left #openstack-infra | 19:45 | |
| clarkb | jeblair: you may want to read zzzeeks email to the list and maybe point at nodepools fixtures | 19:45 |
| Sukhdev | clarkb: Thanks - let me explain the issue (which I believe is a bug) in devstack - but, I could be wrong :-) | 19:45 |
| Sukhdev | When I run devstack, I see the error captured in this paste http://paste.openstack.org/show/98782/ | 19:46 |
| clarkb | mriedem-dplgnr: maybe you too because you have used testresources a lot in tempest right? | 19:46 |
| *** tonytan4ever has quit IRC | 19:47 | |
| jeblair | clarkb: i mentioned it in the review | 19:47 |
| clarkb | jeblair: well it seems to have gone missing as an option in the mail thread | 19:47 |
| Sukhdev | When I look at the code in devstack at the following line https://github.com/openstack-dev/devstack/blob/33dc8699e6b774cc39b39e0dba97be41bbb47abe/lib/tempest#L62 | 19:47 |
| *** juzuluag has quit IRC | 19:47 | |
| jeblair | clarkb: what's the thread | 19:47 |
| clarkb | jeblair: Acceptable methods for establishing per-test-suite behaviors | 19:48 |
| Sukhdev | and the line 336 https://github.com/openstack-dev/devstack/blob/33dc8699e6b774cc39b39e0dba97be41bbb47abe/lib/tempest#L336 | 19:48 |
| openstackgerrit | A change was merged to openstack-infra/config: namos - Device and service manager https://review.openstack.org/109494 | 19:48 |
| mriedem-dplgnr | clarkb: we actually don't use testresources in tempest, we've got some fixture usage and testscenarios | 19:48 |
| Sukhdev | the paths for the image seem to different. I believe that causes the error captured in the paste | 19:48 |
| mriedem-dplgnr | but for resource sharing it's still all done manually in setupClass | 19:48 |
| jeblair | clarkb: would you agree that there are not acceptable methods for per-test-suite behaviors? | 19:49 |
| clarkb | mriedem-dplgnr: thanks I was mixing up scenarios with resources then | 19:49 |
| mriedem-dplgnr | but we need to change that soon because it's blocking a bunch of things | 19:49 |
| *** tonytan4ever has joined #openstack-infra | 19:49 | |
| Sukhdev | clarkb: does my explanation make sense? | 19:49 |
| clarkb | jeblair: yes, I think that there are ways of doing per test suite behaviors that are not acceptable | 19:50 |
| clarkb | Sukhdev: looking now | 19:50 |
| clarkb | except that paste isn't working :/ | 19:50 |
| Sukhdev | clarkb: weird - let me re-paste it | 19:51 |
| *** tgohad__ has joined #openstack-infra | 19:51 | |
| clarkb | Sukhdev: no its fine it loaded | 19:52 |
| clarkb | the paste server has DB issues occasionally | 19:52 |
| clarkb | Sukhdev: so I think we test that upstream and it works | 19:52 |
| clarkb | Sukhdev: my first guess would be that the s3 image just didn't download | 19:52 |
| Sukhdev | clarkb: it is failing every time - | 19:52 |
| Sukhdev | clarkb: however, look at the paths in the code - which I pointed out - I believe that has something to do with it | 19:53 |
| Sukhdev | clarkb: stack.sh succeeds even with this error - however, when I run tempest tests (test_network_basic_ops) it fails because of wrong image | 19:54 |
| *** tgohad has quit IRC | 19:55 | |
| *** ihrachyshka has quit IRC | 19:55 | |
| clarkb | Sukhdev: right, but we run the boto tests upstream so it must be working somehow | 19:55 |
| Sukhdev | clarkb: I have been trying to chase the issue to see why do I not get right image - and ended up with this information | 19:55 |
| jeblair | clarkb: he asks for a per-test-run fixture... | 19:55 |
| jeblair | clarkb: the fixture that we have in nodepool -- how often is that used? | 19:55 |
| jeblair | clarkb: is it per-test, or per-test runner, or ...? | 19:56 |
| clarkb | jeblair: I think it is per test I am double checking | 19:57 |
| Sukhdev | clarkb: I have no idea about the upstream boto tests, hence, can not comment on it - as to why is it working --- but, does my analysis makes sense to you? | 19:57 |
| clarkb | jeblair: ya, DbTestCase instantiates a new fixture in each test that runs under it via setUp(). This fixture sets up then tears down each one | 19:58 |
| clarkb | jeblair: I think zzzeek wants fancy reuse too whcih is where testresources come from but the nodepool solution is so simple I can't really argue against it | 19:58 |
| clarkb | jeblair: if/when it breaks its fixable because simple | 19:58 |
| clarkb | Sukhdev: I don't know enough about devstack's boto materials to know | 19:59 |
| clarkb | Sukhdev: I really think it would be odd if it used the wrong path for image upload though | 19:59 |
| jeblair | i'm kind of hesitant to weigh in here because i think i'm detecting a bit of an extreme reaction to a -1 | 19:59 |
| Sukhdev | clarkb: I agree - hence, I decided to ask folks here | 19:59 |
| jeblair | and i kind of don't want to get into some emotional thing he's got going on | 19:59 |
| clarkb | jeblair: thats fair | 20:00 |
| Sukhdev | clarkb: who do you believe will know who can help - trying to get my tests to work... | 20:00 |
| clarkb | Sukhdev: http://logs.openstack.org/24/115824/2/check/check-tempest-dsvm-full/837b411/logs/devstacklog.txt.gz#_2014-08-21_01_11_18_408 well it failed there too so maybe that is the issue. You can always change the path and rerun to see if it fixes | 20:00 |
| clarkb | Sukhdev: dtroyer is the person to talk to usually | 20:00 |
| *** juzuluag has joined #openstack-infra | 20:01 | |
| Sukhdev | clarkb: cool - thanks. dtroyer will be here or #oepnstack-dev channel? | 20:01 |
| clarkb | Sukhdev: I think he hangs out in a bunch of channel | 20:02 |
| clarkb | s | 20:02 |
| *** juzuluag has quit IRC | 20:02 | |
| *** paulrad has joined #openstack-infra | 20:02 | |
| dtroyer | Sukhdev: gere, -dev and -qa primarily | 20:02 |
| dtroyer | s/gere/here/ | 20:02 |
| Sukhdev | dtroyer: Ah cool - thanks for responding | 20:03 |
| jeblair | clarkb: and he's core on oslo.db now | 20:03 |
| dtroyer | Sukhdev: trying to catch up… | 20:03 |
| Sukhdev | dtroyer: I do not know if you followed my chat with clarkb | 20:03 |
| *** rpodolyaka1 has quit IRC | 20:03 | |
| *** esker has joined #openstack-infra | 20:04 | |
| dtroyer | Sukhdev: TBH, I didn't write accrc and I don't use it at all so it'll take me a bit to sort it out | 20:05 |
| Sukhdev | dtroyer: what is accrc? | 20:05 |
| dtroyer | that's the script that set up a bunch of stuff for these tests | 20:06 |
| dtroyer | IIRC it was not originally meant for tempest, but it looks like tempest uses some of its artifacts | 20:06 |
| jeblair | clarkb: i have responded in a way that i hope will not trigger a flame war | 20:07 |
| pleia2 | jeblair: hooray for passing blind test | 20:07 |
| Sukhdev | dtroyer: Oh I see…. you can see the paste that i pasted showing the error (which clarkb pointed is happening at other places) - which seems to be silent failure and then the two links showing the code using mismatched path | 20:07 |
| Sukhdev | dtroyer: I was thinking about logging a bug, but, was not sure, hence thought I discuss it here - | 20:08 |
| *** marcoemorais has joined #openstack-infra | 20:09 | |
| *** tsufiev has quit IRC | 20:09 | |
| dtroyer | I think it is worth a bug, possibly something went south with the change to using the CIRROS_* variables rather than hard-coding everything | 20:10 |
| *** tsufiev has joined #openstack-infra | 20:10 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:11 | |
| openstackgerrit | Robbie Harwood proposed a change to stackforge/gertty: Query changes for all projects with one query. https://review.openstack.org/105032 | 20:11 |
| *** eharney has quit IRC | 20:12 | |
| Sukhdev | dtroyer: OK - I will file a bug. | 20:12 |
| Sukhdev | dtroyer: against devstack, right? | 20:12 |
| *** emagana has joined #openstack-infra | 20:12 | |
| *** rwsu has quit IRC | 20:13 | |
| *** ociuhandu has joined #openstack-infra | 20:13 | |
| openstackgerrit | A change was merged to openstack-infra/config: Save and report nodepool image names https://review.openstack.org/116349 | 20:13 |
| dtroyer | Sukhdev: yes. include the link clarkb posted with the other ocurrence too | 20:13 |
| Sukhdev | dtroyer:will do - thanks | 20:13 |
| *** tgohad__ has quit IRC | 20:13 | |
| *** peluse has quit IRC | 20:14 | |
| *** tgohad__ has joined #openstack-infra | 20:14 | |
| lifeless | fungi: that mismatch might be an ordered dict or something | 20:14 |
| *** peluse has joined #openstack-infra | 20:14 | |
| *** nelsnelson has quit IRC | 20:15 | |
| lifeless | fungi: ah no, its comparing literal bytestrings | 20:15 |
| zaro | hashar: you'll need to rebase these jjb template default changes 100020, 105155, 103134 | 20:15 |
| hashar | zaro: ahh | 20:16 |
| hashar | zaro: sorry I have been busy this week :-/ Will attempt to have a look at it tonight, no promise though | 20:17 |
| *** emagana has quit IRC | 20:17 | |
| *** kgiusti has left #openstack-infra | 20:18 | |
| *** sandywalsh has quit IRC | 20:18 | |
| *** ramashri has quit IRC | 20:19 | |
| jesusaurus | how does zuul determine what gets grouped together into a change queue? such as the integrated gate vs the infra/config gate | 20:19 |
| clarkb | jesusaurus: any projects with shared test names in a dependentpipeline | 20:19 |
| openstackgerrit | Ramy Asselin proposed a change to openstack-infra/config: Allow zuul & nodepool to be installed on one VM https://review.openstack.org/116373 | 20:19 |
| clarkb | jesusaurus: for us that is typically gate-dsvm-tempest-full. but it is transitive | 20:20 |
| jesusaurus | ah, so if projects run the same test then they end up in the same group | 20:20 |
| *** emagana has joined #openstack-infra | 20:21 | |
| clarkb | yup | 20:21 |
| jesusaurus | is it possible to explicitly group projects together in layout.yaml? | 20:21 |
| *** Hefeweizen has joined #openstack-infra | 20:21 | |
| *** gordc has joined #openstack-infra | 20:22 | |
| clarkb | jesusaurus: no | 20:22 |
| *** HeOS has joined #openstack-infra | 20:22 | |
| jesusaurus | im thinking about the future puppet module projects | 20:22 |
| clarkb | jesusaurus: for the puppet module projects they should all run eg gate-config-puppet-integration | 20:22 |
| jesusaurus | and im not sure if they will share a test, but we want them in the same group | 20:22 |
| clarkb | which is a thing I just made up | 20:22 |
| jesusaurus | oh that makes sense. just create an integration test that we run on all of them | 20:23 |
| clarkb | yup | 20:23 |
| clarkb | jesusaurus: maybe even modify the existing puppet apply test (that may be what jeblair was talking about yesterday) | 20:23 |
| *** fifieldt has quit IRC | 20:23 | |
| jesusaurus | yeah, but it will need some extra logic to know which project it is testing so that it will do the Right Thing™ with the zuul ref | 20:24 |
| clarkb | jesusaurus: there is work to make zuul do that | 20:24 |
| clarkb | jesusaurus: so yes but you may get it for free | 20:25 |
| *** markmc has quit IRC | 20:25 | |
| jesusaurus | well, with my downstream setup im doing something similar with infra/config and our internal config | 20:25 |
| jesusaurus | its not difficult, but it might take a couple brain cells to scale out past two projects | 20:26 |
| clarkb | jesusaurus: no really it will be done for you :) | 20:26 |
| *** juzuluag has joined #openstack-infra | 20:26 | |
| jesusaurus | oh? how so? | 20:26 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix unbound on rax centos6 nodes. https://review.openstack.org/115836 | 20:26 |
| clarkb | jesusaurus: zuul is growing a zuul-cloner tool that will set up repos for you | 20:27 |
| *** marun has joined #openstack-infra | 20:27 | |
| clarkb | jesusaurus: then all your test needs to do is run | 20:27 |
| jesusaurus | oh, is that what zuul-cloner is for? | 20:27 |
| clarkb | jesusaurus: ya | 20:27 |
| *** eharney has joined #openstack-infra | 20:27 | |
| clarkb | I wouldn't use it just yet | 20:27 |
| jesusaurus | so you give it a list of projects and they get cloned for you in $known_location? | 20:27 |
| clarkb | yup | 20:27 |
| jesusaurus | is this going to replace gerrit git prep then? | 20:28 |
| clarkb | jesusaurus: I think so, the cloning in d-g too | 20:28 |
| jeblair | jesusaurus: yeah, i'm working on setting up a config + puppet module integration test job, mostly because i thought no one else was | 20:28 |
| clarkb | oh hey the unbound fix merged | 20:28 |
| clarkb | jeblair: fungi: I can build new centos6 shortly | 20:28 |
| *** gyee has quit IRC | 20:28 | |
| *** ominakov has quit IRC | 20:29 | |
| jeblair | and yes, finishing zuul-cloner is the first step in that | 20:29 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Switch some oslo library jobs to voting https://review.openstack.org/115959 | 20:30 |
| clarkb | nibalizer: jesusaurus did you get a chance to see my comments on 99990? | 20:30 |
| *** annegentle has quit IRC | 20:30 | |
| nibalizer | i did | 20:30 |
| clarkb | nibalizer: the latest one? | 20:30 |
| nibalizer | if i understand correctly, you want infra/config to track master? | 20:30 |
| * nibalizer looks | 20:30 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Switch some oslo library jobs to voting https://review.openstack.org/115959 | 20:30 |
| clarkb | nibalizer: yup | 20:30 |
| *** otherwiseguy has quit IRC | 20:31 | |
| *** otherwiseguy has joined #openstack-infra | 20:31 | |
| clarkb | nibalizer: and my argument for that is I think it will make us more disciplined in code review and it will keep us away from doing large point to point release updates that are scary | 20:31 |
| jeblair | clarkb: yeah, agree with your comments there, and i think the apply integration test i'm working on will facilitate it | 20:31 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Sort the gerrit failed event bug urls map for predictive tests https://review.openstack.org/116375 | 20:31 |
| jesusaurus | clarkb: yeah, i agree that we want to track master of the puppet modules | 20:31 |
| *** ominakov has joined #openstack-infra | 20:31 | |
| nibalizer | clarkb: works for me | 20:31 |
| *** vhoward has joined #openstack-infra | 20:32 | |
| gordc | clarkb: any advice on how to fix the gate-ceilometer-doc error we're blocked by? | 20:32 |
| jesusaurus | clarkb: nibalizer ive been meaning to set aside time to add more better words to the problem description, but havent been in much of a literary mindset. proposed wordings are very welcome | 20:33 |
| clarkb | gordc: I couldn't reproduce it and I don't think it is related to the infra changes that were made | 20:33 |
| clarkb | gordc: however you could try a hashseed of 0 | 20:33 |
| *** paulrad has quit IRC | 20:33 | |
| jeblair | jesusaurus: i don't think that's the most important thing | 20:33 |
| *** nelsnelson has joined #openstack-infra | 20:33 | |
| clarkb | gordc: so I am stumped | 20:33 |
| gordc | clarkb: yeah, no one can reproduce it... | 20:34 |
| clarkb | gordc: we could try holding a slave and reproducing on the lsaves themselves | 20:34 |
| gordc | i'll try the hashseed option | 20:34 |
| * gordc doesn't know enough about test system to know how to debug it. | 20:34 | |
| jesusaurus | jeblair: noted | 20:34 |
| *** gyee has joined #openstack-infra | 20:35 | |
| clarkb | gordc: it runs `tox -evenv setup.py build_sphinx` on trusty machines using latest tox | 20:35 |
| *** baoli has joined #openstack-infra | 20:36 | |
| clarkb | gordc: which should be functionally equivalent to `tox -edocs` if I read the tox.ini correctly | 20:36 |
| gordc | clarkb: i assume all slaves pull libs from same source? | 20:36 |
| *** fifieldt has joined #openstack-infra | 20:36 | |
| clarkb | gordc: not complteely. they all use the same pypi mirror but apt packages come from cloud specific package repos | 20:37 |
| clarkb | gordc: they also have different base images. but this failure happens on both rax and hpcloud pretty evenly so I don't think we are looking at cloud specific issues | 20:38 |
| *** ominakov has quit IRC | 20:38 | |
| *** hashar has quit IRC | 20:38 | |
| gordc | clarkb: yeah, it's pretty random. seems to pass once and then a few hours later it'll fail on same slave | 20:38 |
| *** nelsnelson has quit IRC | 20:38 | |
| *** nelsnelson has joined #openstack-infra | 20:39 | |
| *** alkari1 has quit IRC | 20:39 | |
| *** baoli has quit IRC | 20:40 | |
| clarkb | gordc: well that was easy I got it to fail on my laptop | 20:40 |
| clarkb | clearly I shoudl've tired here instaed of my desktop | 20:40 |
| gordc | clarkb: :) that's good to know | 20:41 |
| clarkb | tox -evenv python setup.py build_sphinx is what I did with master checked out | 20:41 |
| asselin | clarkb, I'm debugging the missing zuul pid. I enabled debug logging. let me know if anything stands out as a possible error: http://paste.openstack.org/show/98895/ | 20:41 |
| *** hashar has joined #openstack-infra | 20:42 | |
| gordc | clarkb: darn still passes for me with that command (i don't have a debian machine though) | 20:42 |
| *** nelsnelson has quit IRC | 20:42 | |
| *** nelsnelson has joined #openstack-infra | 20:43 | |
| clarkb | gordc: running a second time seems to pass | 20:43 |
| * clarkb tries again with clean venv | 20:44 | |
| *** nelsnelson has quit IRC | 20:44 | |
| gordc | clarkb: :( i guess the hashseed approach might be required? or is that totally unrelated. | 20:44 |
| clarkb | gordc: not sure yet. It is possible that it is required due to a bug in the wsme thing (or elsewhere) | 20:45 |
| *** nelsnelson has joined #openstack-infra | 20:45 | |
| clarkb | jeblair: in the fix cloner test change what is the else block after >=2 for? | 20:45 |
| *** marun has quit IRC | 20:45 | |
| *** alkari has joined #openstack-infra | 20:45 | |
| clarkb | jeblair: we should check change 1, then change 2, then stop looping right? | 20:45 |
| clarkb | jeblair: oh duh its for the case where change = 1 | 20:46 |
| hashar | speaking of Zuul cloner, I am slowly migrating Wikimedia jobs to use it :-] | 20:46 |
| *** ramashri has joined #openstack-infra | 20:46 | |
| hashar | still have to support the per project branch override :( | 20:46 |
| clarkb | sometimes "talking" out loud is the thing I need to do to make my brain work | 20:47 |
| clarkb | gordc: a rebuild of the venv worked. so this is very sporadic even on my laptop | 20:49 |
| clarkb | gordc: do we know what the TypeError is? is path None so join() fails? | 20:50 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Apply defaults to job-templates parameters https://review.openstack.org/100020 | 20:50 |
| jeblair | clarkb: yeah, i decided that approach wasn't scalabale and dropped it in future patchsets; i could backport that if we wanted, but probably it's not a big deal | 20:50 |
| jeblair | hashar: oh hi! | 20:50 |
| jeblair | hashar: we were just talking about the series starting here: https://review.openstack.org/#/c/116053/1 | 20:51 |
| gordc | clarkb: yep, i'm assuming its None | 20:51 |
| clarkb | gordc: ya I bet we are falling through to https://github.com/stackforge/wsme/blob/master/wsmeext/sphinxext.py#L140 | 20:51 |
| jeblair | hashar: i did that work on the cloner we talked about (it didn't need much!) | 20:51 |
| clarkb | jeblair: ya not a huge deal, just wrapping my brain around it as I go | 20:51 |
| clarkb | gordc: now to sort out why that could be | 20:52 |
| gordc | clarkb: same. something wrong with scan_services | 20:52 |
| *** weshay has quit IRC | 20:52 | |
| jeblair | clarkb: the idea that it checks the state for every change in the queue (and that things will be different for each) is kept, so that's worth groking | 20:52 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Integration tests for templates defaults https://review.openstack.org/103134 | 20:52 |
| gordc | clarkb: i've a feeling its https://github.com/stackforge/wsme/blob/master/wsmeext/sphinxext.py#L120-L122 but i'm not sure what why it'd work 10% of the time | 20:53 |
| *** mmedvede has quit IRC | 20:53 | |
| *** nelsnelson has quit IRC | 20:54 | |
| *** YorikSar has quit IRC | 20:54 | |
| *** nelsnelson has joined #openstack-infra | 20:54 | |
| jeblair | clarkb: (and yeah, this whole thing is screaming for a testscenarios refactor, but i think we can do that later) | 20:55 |
| *** annegentle has joined #openstack-infra | 20:55 | |
| *** YorikSar has joined #openstack-infra | 20:55 | |
| *** maxbit has joined #openstack-infra | 20:56 | |
| *** stevemar has quit IRC | 20:56 | |
| *** sdake_ has joined #openstack-infra | 20:57 | |
| *** Longgeek has quit IRC | 20:57 | |
| *** hogepodge_ has joined #openstack-infra | 20:59 | |
| hashar | zaro: I have rebased two of the changes, a trivial conflict with JJB doc/source/configuration.rst that was renamed | 20:59 |
| hashar | zaro: the last change ( https://review.openstack.org/#/c/105155/ ) I can't remember offhand, and I had too many ale (beer) to really think about it right now. Left a comment though. | 21:00 |
| jeblair | hashar: and the end of my zuul-cloner series adds the per-project override | 21:00 |
| hashar | you are awesome | 21:00 |
| hashar | I hope you make it to Paris in Nov && you like beer | 21:01 |
| jeblair | hashar: no, you are! i hardly had to change anything :) | 21:01 |
| hashar | cause I owe you a few gallons | 21:01 |
| hashar | great! | 21:01 |
| *** zul has quit IRC | 21:01 | |
| hashar | the whole zuul cloner idea came from a discussion with jhesketh when he presented me turbo-hipster | 21:01 |
| *** rkukura has quit IRC | 21:01 | |
| clarkb | gordc: the iteration in scan_services seems to be deterministic | 21:01 |
| hashar | he basically had to write a bunch of python around devstack gate | 21:01 |
| *** hogepodge has quit IRC | 21:02 | |
| *** hogepodge_ is now known as hogepodge | 21:02 | |
| clarkb | gordc: dir() returns a sorted list for example | 21:02 |
| clarkb | gordc: so if it is hashseed related it would need to be a result of the order the paths are called? | 21:02 |
| jeblair | hashar: i'm about to start using it for an infra job, and once i have that worked out, i'll start working on using it in devstack-gate | 21:02 |
| gordc | clarkb: hmm...possibly. first time really looking at wsme code. | 21:03 |
| gordc | clarkb: i'll try pushing a hashseed patch to see if that passes. | 21:03 |
| *** Sukhdev has quit IRC | 21:04 | |
| *** emagana has quit IRC | 21:05 | |
| clarkb | gordc: sounds good | 21:05 |
| *** ominakov has joined #openstack-infra | 21:05 | |
| gordc | clarkb: thanks for taking a look | 21:05 |
| clarkb | I am still trying to get it to fail a second time | 21:06 |
| clarkb | may start doing a complete git clean first | 21:06 |
| hashar | jeblair: not internally it uses git clone, so if the destination already exist git clone exit 128 because the directory already exist. | 21:07 |
| lifeless | clarkb: oh hi; I have invoked you for input in #openstack-nova | 21:07 |
| *** emagana_ has joined #openstack-infra | 21:07 | |
| hashar | jeblair: the trick is to have the parent repo cloned first : i.e. clone myproject before myproject/plugin1 | 21:07 |
| hashar | jeblair: might not be an issue for you though. | 21:07 |
| pballand | is there any “standard” in openstack for taggint non-release milestones? | 21:08 |
| pballand | (https://wiki.openstack.org/wiki/StableBranchRelease only talks about tags for releases) | 21:08 |
| *** ominakov has quit IRC | 21:08 | |
| pballand | *tagging | 21:08 |
| jesusaurus | nibalizer: do you have a link for the git-subtree %B bug? i cant seem to find it | 21:08 |
| clarkb | pballand: they should follow python rc/alpha/beta release scheme. we have a regex to match that | 21:09 |
| * clarkb digs that up | 21:09 | |
| *** esker has quit IRC | 21:09 | |
| *** nelsnelson has quit IRC | 21:09 | |
| clarkb | pballand: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n96 | 21:09 |
| *** paulrad has joined #openstack-infra | 21:09 | |
| *** nelsnelson has joined #openstack-infra | 21:10 | |
| *** dustins has quit IRC | 21:10 | |
| *** dangers is now known as dangers_away | 21:12 | |
| *** mfer has quit IRC | 21:12 | |
| *** nelsnelson has quit IRC | 21:13 | |
| *** rfolco has quit IRC | 21:13 | |
| pballand | clarkb: thanks - so 1.0alpha or 1.0.alpha ? (both look fine by the regex) | 21:13 |
| *** ihrachyshka has joined #openstack-infra | 21:13 | |
| harlowja | qq, maybe its better fixed or something tiny, but has/was there any adjustment to infra that made it so that custom tox environments that run can get there testr results published correctly? ex, http://logs.openstack.org/91/116191/5/check/gate-taskflow-tox-py26-sa7-mysql/a3f6b50/ (this is one of those custom environments outputs) | 21:14 |
| clarkb | harlowja: I think those tpyes of jobs should use the run-unittests runner which may require modifications to that runner | 21:15 |
| *** paulrad has quit IRC | 21:15 | |
| *** bradjones has quit IRC | 21:15 | |
| clarkb | pballand: I wnt to say one is actually less correct. lifeless ^ you are deep into versioning right now | 21:15 |
| harlowja | clarkb let me try that switch and we'll see how this goes | 21:16 |
| *** mbacchi has quit IRC | 21:16 | |
| clarkb | harlowja: I think the biggest thing is run-unittests assumes a py* prefix for the target name | 21:16 |
| pballand | clarkb: looks like projects are using the <version><dot><prerelease> version, but it isn’t always followed | 21:16 |
| clarkb | harlowja: we can stop assuming that and force everything to provide an actual complete name | 21:16 |
| harlowja | ya, hmmm | 21:16 |
| harlowja | i think taskflow can try this and see what happens, taskflow envs have that prefix | 21:17 |
| harlowja | see how it goes | 21:17 |
| *** dimsum_ has quit IRC | 21:17 | |
| *** dimsum_ has joined #openstack-infra | 21:18 | |
| clarkb | harlowja: oh ya so its cheap for you to switch | 21:18 |
| harlowja | i do like the cheap, ha | 21:18 |
| *** sabeen1 has quit IRC | 21:19 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Sort the gerrit failed event bug urls map for predictive tests https://review.openstack.org/116375 | 21:19 |
| *** datsun180b has quit IRC | 21:21 | |
| *** dprince has quit IRC | 21:21 | |
| *** e0ne has joined #openstack-infra | 21:22 | |
| *** dimtruck is now known as zz_dimtruck | 21:22 | |
| *** dimsum_ has quit IRC | 21:22 | |
| *** doug-fish has left #openstack-infra | 21:23 | |
| asselin | clarkb, I'm finding that the zuul pid is created, then deleted about 6-8 seconds later (via watch -n 1 ls -l /var/run/zuul). but the 2 zuul-server processes are still running. any ideas how to debug? | 21:23 |
| clarkb | jeblair: 116098 has a couple small cleanups I -1'd for. let me know if you would rather leave it as is | 21:23 |
| clarkb | asselin: are you running latest zuul? | 21:24 |
| asselin | yes | 21:24 |
| clarkb | asselin: that sounds like a bug in the daemonization process | 21:24 |
| *** bradjones has joined #openstack-infra | 21:24 | |
| *** bradjones has joined #openstack-infra | 21:24 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Switch taskflow to use the unittests slave script https://review.openstack.org/116388 | 21:25 |
| asselin | clarkb, operating system bug or installed library? | 21:25 |
| harlowja | clarkb if u get a sec ^ ; pretty sure thats right | 21:25 |
| clarkb | asselin: I think installed lib or zuul | 21:25 |
| zaro | hashar: doc error in https://review.openstack.org/#/c/100020/ | 21:25 |
| clarkb | asselin: the OS doesn't really have much part in it other than syscalls | 21:25 |
| clarkb | harlowja: I think that is correct. let me double checking the zuul node selection | 21:26 |
| openstackgerrit | Atin Ruia proposed a change to openstack-infra/jenkins-job-builder: Adding feature to specify the user id for jenkins user credentials https://review.openstack.org/109723 | 21:26 |
| clarkb | harlowja: yup that looks like it will be fine | 21:27 |
| *** ominakov has joined #openstack-infra | 21:28 | |
| harlowja | clarkb thx much, should be helpful in seeing the testr outputs and such | 21:29 |
| *** gondoi is now known as zz_gondoi | 21:29 | |
| clarkb | jeblair: one of your changes was hit by the double teardown thing | 21:30 |
| clarkb | jeblair: I think that is a zuul wide issue and not related to your change. I am going to see if I can debug | 21:31 |
| *** sdake_ has quit IRC | 21:31 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Switch taskflow to use the unittests slave script https://review.openstack.org/116388 | 21:31 |
| harlowja | just comment update ^ | 21:31 |
| harlowja | ^ commit message update i mean | 21:32 |
| hashar | zaro: fixing it | 21:32 |
| ianw | what does the || mean in "node: 'devstack-precise || devstack-trusty'" | 21:32 |
| ianw | i see it sets assignedNode in the jenkins xml | 21:32 |
| ianw | is that "precise first, trusty if not available" | 21:33 |
| clarkb | ianw: it means that the job can run on devstack-precise or devstack-trusty on the jenkins side. then we have zuul select which one to use | 21:33 |
| *** bradjones has quit IRC | 21:33 | |
| ianw | clarkb: how does zuul select, round-robin? | 21:33 |
| clarkb | ianw: it is a short circuiting ||. We do that as an artifact of how jenkins registers jobs with gearman | 21:33 |
| clarkb | ianw: no, we have a zuul layout function that selects them based on job | 21:33 |
| clarkb | ianw: we could register all jobs on all jobs by leaving the node specification in jenkins blank then have zuul do all the selection but I went with this to restrict how big fires could get if zuul did the wrong thing | 21:34 |
| clarkb | ianw: and it helped in at least one case so I didn't change it :) | 21:34 |
| clarkb | jeblair: confirmed zuul master is affected | 21:35 |
| ianw | clarkb: oh, default_params_trusty() and around there i guess in zuul? | 21:35 |
| clarkb | ianw: yup | 21:35 |
| *** eharney has quit IRC | 21:35 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Apply defaults to job-templates parameters https://review.openstack.org/100020 | 21:36 |
| clarkb | it basically has a set of rules based job name and job branch to select the nodes | 21:36 |
| gordc | clarkb: seems like a wsme issue. tests are sporadically failing there as well... | 21:36 |
| gordc | clarkb: down the rabbit hole i go. | 21:36 |
| *** e0ne has quit IRC | 21:36 | |
| *** baoli has joined #openstack-infra | 21:36 | |
| clarkb | gordc: have fun | 21:37 |
| clarkb | and don't let the crazy cat get you | 21:37 |
| *** ominakov has quit IRC | 21:37 | |
| hashar | zaro: the tests are badly falling because of dictionary orders :-( | 21:37 |
| *** maxbit has quit IRC | 21:37 | |
| hashar | zaro: I mean the JJB tests (random example http://logs.openstack.org/20/100020/5/check/gate-jenkins-job-builder-python27/ea07856/console.html ) | 21:38 |
| clarkb | hashar: zaro you can work around that by overriding the hashseed value | 21:38 |
| *** annegentle has quit IRC | 21:38 | |
| clarkb | hashar: but long term we should make it work with arbitrary dict order or use ordered dicts from collections. I think there is a JJB change up to start using ordereddict | 21:38 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Classify tempest lxml hashseed bug 1360034 https://review.openstack.org/116158 | 21:39 |
| uvirtbot | Launchpad bug 1360034 in tempest "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20; lxml random hashseed failure" [Critical,Fix committed] https://launchpad.net/bugs/1360034 | 21:39 |
| *** annegentle has joined #openstack-infra | 21:39 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for heat UserKeyPairMissing bug 1360415 https://review.openstack.org/116363 | 21:39 |
| uvirtbot | Launchpad bug 1360415 in heat "test_nova_keypair_resources fails with UserKeyPairMissing" [Undecided,New] https://launchpad.net/bugs/1360415 | 21:39 |
| *** laxity has quit IRC | 21:39 | |
| hashar | clarkb: how come it suddenly started to fail? Is that some python / nose version that has been bumped? | 21:39 |
| ianw | clarkb: thanks, hopefully my comments on https://review.openstack.org/#/c/115752/ make sense | 21:39 |
| clarkb | hashar: we upgraded the version of tox | 21:40 |
| *** Sukhdev has joined #openstack-infra | 21:40 | |
| clarkb | hashar: latest tox sets a random pythonhashseed value | 21:40 |
| fungi | hashar: latest versions of tox randomize the python hash seed on python 2.x to match the behavior of 3.x | 21:40 |
| *** baoli has quit IRC | 21:41 | |
| hashar | ohh | 21:41 |
| zaro | how do you fix that? | 21:41 |
| clarkb | ianw: yup lgtm | 21:41 |
| hashar | I have just graduated from 101 Python | 21:41 |
| hashar | need to look at 102 Python hash seed messing up dicts | 21:41 |
| hashar | clarkb: fungi: thanks for the explanation :] | 21:41 |
| fungi | zaro: you fix places where you didn't realize your tests (or the program itself) depend on the order of elements in an unordered data type | 21:42 |
| hashar | zaro: I guess ordereddict is the answer | 21:42 |
| *** mtreinish-dplgnr is now known as mriedem | 21:42 | |
| fungi | or you work around it by setting an explicit PYTHONHASHSEED=0 envvar and pretend it's not a problem, and then hope someone else rewrites it for you before it's ever used on python 3 ;) | 21:43 |
| *** mriedem-dplgnr is now known as mtreinish | 21:43 | |
| *** ihrachyshka has quit IRC | 21:43 | |
| hashar | zaro: | 21:43 |
| hashar | [testenv:hash] | 21:43 |
| hashar | setenv = | 21:43 |
| hashar | PYTHONHASHSEED = 100 | 21:43 |
| hashar | new in tox 1.6.2 https://testrun.org/tox/latest/example/basic.html#special-handling-of-pythonhashseed | 21:43 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Handle latest testtools tearDown checks https://review.openstack.org/116393 | 21:43 |
| zaro | would this fix it? https://review.openstack.org/#/c/107063/ | 21:43 |
| clarkb | jeblair: fungi ^ that should make zuul tests happy again | 21:43 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Error on duplicate job names being found https://review.openstack.org/56734 | 21:43 |
| clarkb | hashar: ya but set to 0 if you want the old behavior | 21:44 |
| clarkb | 0 is the default | 21:44 |
| fungi | zaro: potentially | 21:44 |
| *** _nadya_ has joined #openstack-infra | 21:44 | |
| hashar | I dont understand PYTHONHASHSEED honestly | 21:44 |
| hashar | if you have everything with PYTHONHASHSEED=42 , you should end up with ordered dict dont you? | 21:45 |
| clarkb | ianw: also note that because dsvm is in that test name it will default to trusty. explicitly saying f20 in the name will hcange that if you want f20 | 21:45 |
| clarkb | hashar: yes you would end up with consistent ordering, but that consistent ordering may be different than the consistend ordering from HASHSEED=0 | 21:45 |
| fungi | hashar: the short story is that it was discovered a few years ago that you could predict hash collisions for python dicts, and so as one of the mitigating solutions they grew a feature to allow you to specify your own seed value to alter how the hash was generated | 21:45 |
| *** mriedem has quit IRC | 21:46 | |
| clarkb | and in python3 it is random by default but python2 is still 0 by default | 21:46 |
| fungi | and for py3k this was randomized completely between invocations, to be even less predictable, yes | 21:46 |
| hashar | so in python3 hash are randomized to make sure they yield elements in random orders? | 21:47 |
| fungi | hashar: right | 21:48 |
| hashar | I should propose a pep to rename dict() to dict_randomized() | 21:48 |
| hashar | and have ordereddict renamed to dict() :D | 21:48 |
| *** eharney has joined #openstack-infra | 21:49 | |
| *** eharney has quit IRC | 21:49 | |
| *** _nadya_ has quit IRC | 21:49 | |
| fungi | hashar: you could just do that in your software if you really wanted (renaming/replacing built-in functions is actually not hard at all) | 21:49 |
| hashar | anyway,for JJB: we have to convert to ordereddict | 21:49 |
| hashar | since we want the same XML output to be rendered | 21:49 |
| hashar | fungi: yeah what puzzle me is why py3 actually randomize the hash | 21:50 |
| hashar | seems like a trick to make sure developers think twice before using a dict (i.e.: ask themselves: do I care about key order?) | 21:51 |
| clarkb | hashar: yup | 21:51 |
| fungi | hashar: because it was still fairly low-adoption so they could get away with doing it there (as opposed to python 2 where some significant percentage of the existing install base may depend on arbitrarily ordered data from what were presented as being unordered data types) | 21:51 |
| hashar | which is evil | 21:51 |
| hashar | but clever | 21:51 |
| clarkb | hashar: well not quite it is a security bug | 21:51 |
| clarkb | hashar: you don't want predictable hash collisions | 21:51 |
| clarkb | a side effect of that is that your dict keys are not ordered | 21:52 |
| hashar | you guys should host Python hackercamps | 21:52 |
| *** lascii is now known as alaski | 21:52 | |
| hashar | I basically learned python by chatting here | 21:52 |
| hashar | clarkb: got it :-] | 21:53 |
| fungi | in fact, your dict keys never were ordered per se... if you ran the same program on two different platforms with the same python release or on the same platform with two slightly different python releases, you got different ordering | 21:53 |
| hashar | fungi: but it works for me!!!! ™ | 21:53 |
| hashar | I can imagine the number of bugs it caused | 21:53 |
| fungi | this just makes it so that you get different ordering every time you run, even on the same platform with the same python release | 21:53 |
| *** signed8bit is now known as signed8bit_ZZZzz | 21:54 | |
| *** aysyd has quit IRC | 21:54 | |
| *** bhuvan has quit IRC | 21:55 | |
| *** gordc has quit IRC | 21:55 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Bump zake up to 0.1 https://review.openstack.org/116399 | 21:58 |
| *** laxity has joined #openstack-infra | 21:58 | |
| *** bhuvan has joined #openstack-infra | 21:59 | |
| *** bdpayne_ has joined #openstack-infra | 22:00 | |
| *** bdpayne has quit IRC | 22:00 | |
| hashar | fungi: yeah that make sense | 22:00 |
| hashar | fungi: clarkb: thank you two for the hash seed explanation. Much appreciated | 22:00 |
| lifeless | clarkb: hi, context ? | 22:01 |
| lifeless | clarkb: oh versions - 1.0.0.0a1 IIRC, let me double check | 22:01 |
| clarkb | jeblair: can you see my comment on 116105? | 22:02 |
| clarkb | jeblair: just making sure I understand as I go | 22:02 |
| lifeless | clarkb: The identifier MUST comprise only a, b, rc followed by non-negative | 22:03 |
| lifeless | integer value. The identifier MUST NOT be empty. | 22:03 |
| lifeless | so 1.0.0.0a1 | 22:03 |
| clarkb | pballand: ^ | 22:04 |
| asselin | clarkb, figured it out, thanks. I submitted a bug report with my investigation notes & tips to resolve. https://bugs.launchpad.net/zuul/+bug/1360456 | 22:04 |
| uvirtbot | Launchpad bug 1360456 in zuul "zuuls fails to reload/stop due to missing PID" [Undecided,New] | 22:04 |
| lifeless | jeblair: hi, I'd like to get two nova patches ninjad into stable/icehouse, and clarkb has tagged you as infra person to $approve/$assist something | 22:04 |
| lifeless | jeblair: dprince and mikal are pro doing this, and there's more context in #openstack-nova | 22:04 |
| *** armax has left #openstack-infra | 22:05 | |
| clarkb | asselin: oh good, I think we avoid that by gating on the layout check | 22:05 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add open reviews to elastic-recheck status page https://review.openstack.org/109144 | 22:06 |
| *** tgohad__ has quit IRC | 22:06 | |
| asselin | clarkb, yea, good idea. I'll see if I can add that to my script. | 22:08 |
| jesusaurus | clarkb: asselin ive seen that as well (when layout.yaml is valid yaml but with unknown options) | 22:10 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 22:11 |
| r1chardj0n3s | might make people happy: I've just got site-wide config into pip for the 1.6 release :) | 22:11 |
| jeblair | lifeless: #openstack-nova is not logged, can you repeat here? | 22:12 |
| lifeless | jeblair: sure, sec | 22:12 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Classify tempest lxml hashseed bug 1360034 https://review.openstack.org/116158 | 22:12 |
| uvirtbot | Launchpad bug 1360034 in tempest "safe_minidom_parse_string raising ExpatError in Jenkins runs since 8/20; lxml random hashseed failure" [Critical,Fix committed] https://launchpad.net/bugs/1360034 | 22:12 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Use OrderedDict to preserve key order in created dicts https://review.openstack.org/107063 | 22:13 |
| jeblair | clarkb: regarding your first comment on 116105, it's stable/havana branch tip because it's overriding the fallback branch to stable/havana. so in d-g terms it's $BRANCH | 22:14 |
| openstackgerrit | Brant Knudson proposed a change to openstack-infra/devstack-gate: Stash keystone apache httpd logs https://review.openstack.org/116404 | 22:15 |
| clarkb | jeblair: I think that is a deviation in behavior then | 22:15 |
| jroll | ttx: did you mark these started? why? they have no code proposed :/ https://blueprints.launchpad.net/ironic/+spec/drac-management-driver https://blueprints.launchpad.net/ironic/+spec/ironic-snmp-power-driver | 22:15 |
| jeblair | clarkb: no, in d-g, we set $BRANCH to stable/foo on the old side of the grenade job | 22:15 |
| jeblair | clarkb: because we want grenade to, assuming it's testing no change, check out stable/foo | 22:16 |
| clarkb | jeblair: hrm I must be missing how that works then | 22:16 |
| jeblair | clarkb: well, let's talk about the desirability first -- does it make sense that when performang an upgrade test, you want the old side to be stable/foo and the new side to be master? | 22:17 |
| clarkb | jeblair: yes | 22:17 |
| jeblair | clarkb: so the mechanics of that are: | 22:17 |
| lifeless | jeblair: http://paste.ubuntu.com/8117616/ | 22:17 |
| jeblair | GRENADE_OLD_BRANCH='stable/havana' | 22:17 |
| lifeless | jeblair: (btw you are in the channel so it should be in your scrollback :P) | 22:17 |
| jeblair | setup_project openstack/nova $GRENADE_OLD_BRANCH | 22:17 |
| jeblair | lifeless: the log isn't for me | 22:17 |
| jeblair | lifeless: it's because you're asking one of us to break protocol | 22:18 |
| lifeless | jeblair: roger | 22:18 |
| jeblair | lifeless: i'm not doing that because you asked nicely. everyone gets to be on the hook on the record for that. :) | 22:18 |
| lifeless | jeblair: thats totally fine | 22:18 |
| lifeless | I can add a bit more context too, sec | 22:18 |
| *** dimsum_ has joined #openstack-infra | 22:19 | |
| jeblair | lifeless: what is this actually blocking, a pbr change? | 22:19 |
| * persia is discovering reasons that it would be nice to have $BRANCH of stable/foo and test changes towards stable/bar, but may have been chatting with the wrong folk | 22:19 | |
| jeblair | persia: we can handle that too | 22:19 |
| lifeless | jeblair: better paste - http://paste.ubuntu.com/8117630/ | 22:19 |
| lifeless | jeblair: top of that explains | 22:19 |
| *** alkari has quit IRC | 22:20 | |
| lifeless | jeblair: we can't make semver error on bad versions until we have no bad versions in the gate branches | 22:20 |
| lifeless | jeblair: and grenade pulls in stable/icehouse for changes to trunk | 22:21 |
| *** gordc has joined #openstack-infra | 22:22 | |
| *** che-arne has quit IRC | 22:22 | |
| jeblair | lifeless: https://review.openstack.org/#/c/115815/1 is passing tests | 22:23 |
| *** che-arne has joined #openstack-infra | 22:23 | |
| *** dmsimard is now known as dmsimard_away | 22:23 | |
| *** dimsum_ has quit IRC | 22:23 | |
| *** rkukura has joined #openstack-infra | 22:24 | |
| lifeless | that was the 10th? recheck. | 22:24 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Start running bashate on config (nonvoting) https://review.openstack.org/112446 | 22:24 |
| lifeless | jeblair: unless something environment has just been fixed, I fully expect to cause a gate reset and bounce out | 22:24 |
| *** superdan is now known as dansmith | 22:27 | |
| *** hdd has quit IRC | 22:28 | |
| clarkb | jeblair: https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n293 is what I am looking at. And I agree that the behavior codified by the new test is what we want. So I guess my question may be how is d-g doing it the way we want | 22:29 |
| bknudson | is elastic-recheck working for others? http://status.openstack.org/elastic-recheck/ | 22:29 |
| clarkb | also in other news I think we should vote for Gerrit, Zuul, and Jenkins as the three most influential people in openstack | 22:29 |
| jeblair | lifeless: i don't object, but i also don't have time to actually do it right now; i'm about to run out | 22:30 |
| *** juzuluag has quit IRC | 22:30 | |
| clarkb | bknudson: no :( | 22:30 |
| lifeless | jeblair: can clarkb do it now that its approved? | 22:30 |
| clarkb | I can do it | 22:30 |
| clarkb | and it sounds like we have a form of agreement from dansmith, jeblair and others | 22:30 |
| jeblair | clarkb: because we call setup_workspace with $GRENADE_OLD_BRANCH as an argument | 22:30 |
| * clarkb prepares the ninja suit | 22:30 | |
| jeblair | clarkb: which gets set somewhere in devstack-gate | 22:31 |
| dansmith | clarkb: there's my jsonschema fix and what else? | 22:31 |
| clarkb | jeblair: right but in that set of conditions it will try the oldbranch ref, then the master ref, then old branch proper then master proper | 22:31 |
| jeblair | clarkb: yep, and it's oldbranch proper we want and get | 22:31 |
| clarkb | jeblair: right so how does it skip grabbing the master zuul ref | 22:32 |
| clarkb | jeblair: since it tries that first I would expect it to not keep falling through to get old branch proper | 22:32 |
| *** emagana_ has quit IRC | 22:32 | |
| jeblair | clarkb: i think the documentation for setup_project may be wrong | 22:33 |
| lifeless | clarkb: I need to relocate; do you need me for the ninja? If so, is in ~25m ok ? | 22:33 |
| clarkb | dansmith: https://review.openstack.org/#/c/112882/ is the other change | 22:33 |
| clarkb | lifeless: I don't need you to do this. just going to confirm with dansmith on the second change | 22:33 |
| dansmith | clarkb: and the need to ninja is that we need both at the same time? | 22:34 |
| lifeless | dansmith: no | 22:34 |
| lifeless | dansmith: its that its so fragile - 10 rechecks to get green on *check* | 22:34 |
| lifeless | dansmith: both patches are going to have the same fragility since neither affect that | 22:35 |
| dansmith | so this doesn't fix the problem, you just want it in? | 22:35 |
| dansmith | I see | 22:35 |
| jeblair | clarkb: tracing the code in this case, i think it actually tries: stable zuul ref, stable zuul ref (a second time), stable tip, master. | 22:35 |
| clarkb | dansmith: we need jsonschema first, then the second one will have a marginally better chance | 22:35 |
| clarkb | jeblair: oohhhh that would explain it | 22:35 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 22:35 |
| clarkb | jeblair: I will reread and see if I get that concluson too | 22:35 |
| jeblair | clarkb: nope that's wrong, it's actually: stable zuul ref, stable zuul ref (a second time), stable tip, master. | 22:35 |
| jeblair | clarkb: grr | 22:35 |
| lifeless | ok, back in 25m | 22:35 |
| jeblair | clarkb: nope that's wrong, it's actually: stable zuul ref, stable tip, master. | 22:36 |
| jeblair | clarkb: there, it doesn't do it twice. | 22:36 |
| jeblair | clarkb: anyway, in cloner, i reworked the documentation, i think it may be more correct there. | 22:36 |
| jeblair | okay, really going now. | 22:36 |
| clarkb | jeblair: what happens to that second git fetch || git fetch? | 22:36 |
| clarkb | jeblair: ya no rush it seems to be working as expected so I won't worry | 22:36 |
| jeblair | clarkb: i think fallback zuul ref is "" because the project _does_ have the stable branch | 22:36 |
| * fungi thought it did stable zuul ref, stable branch tip, master zuul ref, master branch tip | 22:36 | |
| *** Sukhdev has quit IRC | 22:36 | |
| clarkb | jeblair: oh thats it | 22:37 |
| jeblair | clarkb: so the path is override, (skip fallback), stable, (master does not run) | 22:37 |
| clarkb | jeblair: that is exactly what I was missing. its the hasbranch check earlier | 22:37 |
| *** bdpayne_ has quit IRC | 22:37 | |
| jeblair | yep | 22:37 |
| jeblair | clarkb: which, be careful, is logic-reversed in cloner | 22:37 |
| *** Sukhdev has joined #openstack-infra | 22:37 | |
| *** baoli has joined #openstack-infra | 22:37 | |
| clarkb | dansmith: so I will defer to you on this | 22:37 |
| *** gordc has quit IRC | 22:38 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 22:38 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use OrderedDict to preserve key order in created dicts https://review.openstack.org/107063 | 22:38 |
| *** hashar is now known as hasharUnknownWMF | 22:38 | |
| jeblair | yeah, honestly i'm pretty uncomfortable with tho whole situation because we normally only ninja merge stuff that can't get in otherwise (catch-22 on races) | 22:38 |
| *** hasharUnknownWMF is now known as hasharMaybeWMF | 22:39 | |
| jeblair | i would really prefer that people not ask us to do things like this | 22:39 |
| dansmith | jeblair: yeah, I'm kinda nervous about it | 22:39 |
| clarkb | fungi: jeblair we are running new centos6 images in the three rax AZs now. We should see hostnames in jobs that fail on those nodes now | 22:39 |
| clarkb | dansmith: jeblair: in that case maybe we should hold off | 22:39 |
| *** bdpayne_ has joined #openstack-infra | 22:39 | |
| fungi | clarkb: well, yes and no. if they fail because of the "booted from an older snapshot than we asked for" they won't have the slave script updates and additional file, but at least it'll be that much more obvious | 22:40 |
| jeblair | but since this fixes a test, (apparently to a high degree of confidence to the nova devs), i didn't feel like i should block it | 22:40 |
| jeblair | i would be quite happy if we decided not to do it though | 22:40 |
| jogo | fungi: the first iteration of the open reviews listing on http://status.openstack.org/elastic-recheck/ is up | 22:40 |
| jeblair | okay, really really leaving now :) | 22:41 |
| *** marcusvrn has quit IRC | 22:41 | |
| clarkb | fungi: right | 22:41 |
| clarkb | a node I just spotchecked has the image-hostname file and is using unbound as its resolver \o/ | 22:41 |
| fungi | jogo: what am i looking for? i'm likely staring straight at it | 22:41 |
| jogo | 'Open reviews' | 22:42 |
| *** baoli has quit IRC | 22:42 | |
| jogo | fungi: ^ search for that term | 22:42 |
| pleia2 | jogo: cool | 22:42 |
| fungi | jogo: aha! | 22:42 |
| fungi | neat-o | 22:42 |
| clarkb | jhesketh: fungi: jeblair for when weekending is not impending https://review.openstack.org/#/c/116393/ will unstick zuul tests | 22:42 |
| jogo | pleia2 fungi: thanks, I am bad with UI so if you have some ideas to make it better | 22:43 |
| clarkb | ooh | 22:43 |
| fungi | clarkb: yep, lgtm--was just waiting on it to prove itself. +2 now | 22:43 |
| fungi | jogo: you want it larger and bolder i think | 22:43 |
| fungi | jogo: blink tag, flaming text, spinning skulls | 22:44 |
| * fungi is clearly even worse at ui | 22:44 | |
| jogo | fungi: hmm yeah, I have a long plane ride ahead to figure out how to make it larger and bolder | 22:44 |
| pleia2 | fungi's first idea was good though ;) | 22:44 |
| *** annegentle has quit IRC | 22:44 | |
| clarkb | fungi: marquee | 22:44 |
| * fungi uses budugllydesign.com as his go-to site for what the web should look like | 22:45 | |
| jogo | fungi: https://review.openstack.org/#/c/115815 can use a promotion to the top of the gate | 22:45 |
| jogo | if it isn't already | 22:45 |
| *** annegentle has joined #openstack-infra | 22:45 | |
| fungi | jogo: i think that's one of the ones clarkb/dansmith are coordinating promoting or not | 22:47 |
| fungi | or were they for master and that is that the icehouse backport of one of them? | 22:47 |
| clarkb | fungi: not coordinating, ninja merging | 22:48 |
| clarkb | but I get the feeling that ninja merge is making uncomfortableness and should be avoided | 22:48 |
| *** wenlock has quit IRC | 22:48 | |
| jogo | fungi: this is the icehouse backport | 22:48 |
| jogo | fungi: don't know if its already promoted or not | 22:49 |
| fungi | clarkb: you choose to dishonor your real ultimate power? y u no ninja? | 22:49 |
| jogo | https://review.openstack.org/#/c/94480 is another gate bug patch | 22:49 |
| jogo | that is in the queue | 22:49 |
| jogo | hmm that one has some unit test failures though | 22:49 |
| asselin | is there a way to configure a zuul job to not post any comment at all? Run the job(s), upload log files, and stop, don't post results for any jobs marked as "private", but post the others. | 22:49 |
| fungi | jogo: yeah, also a stable branch fix | 22:49 |
| fungi | jogo: i think that's the other icehouse backport of the two clarkb and dansmith were discussing | 22:50 |
| fungi | oh, maybe not | 22:50 |
| fungi | the master version of that one is already landed | 22:50 |
| fungi | asselin: you could put the jobs in a "silent" pipeline | 22:51 |
| asselin | fungi, thanks will look into that | 22:52 |
| jogo | time for my flight | 22:52 |
| jogo | o/ | 22:52 |
| pleia2 | have fun | 22:53 |
| *** msabramo has quit IRC | 22:56 | |
| *** gokrokve has quit IRC | 22:57 | |
| *** hogepodge has quit IRC | 22:58 | |
| lifeless | ok back | 22:59 |
| clarkb | lifeless: so jeblair and dansmith both are uncomfortable with ninja merges so I haven't done it | 22:59 |
| *** Ryan_Lane1 has quit IRC | 22:59 | |
| dansmith | my reasoning is because it's not a ninja patch to fix an issues of circular dependencies or some other reason we ninja things in | 23:00 |
| dansmith | it's to land two things we need, despite the fragility and I think that's the wrong reason to ninja things | 23:00 |
| *** homeless has quit IRC | 23:00 | |
| lifeless | dansmith: what makes you nervous ? | 23:00 |
| dansmith | lifeless: the precedent it sets | 23:01 |
| asselin | clarkb, fungi, a quick review of this would be appreciated. some cinder ci folks setting up 3rd party ci also need it. currently cherry picking https://review.openstack.org/#/c/116373/ | 23:01 |
| openstackgerrit | Kevin L. Mitchell proposed a change to openstack-infra/jenkins-job-builder: Work around PYTHONHASHSEED-related problems https://review.openstack.org/116410 | 23:01 |
| *** Ryan_Lane1 has joined #openstack-infra | 23:02 | |
| lifeless | dansmith: ok, so the issue is - we're not going to backport things like the neutron callback on vif plugging etc | 23:02 |
| lifeless | dansmith: the jsonschema patch has to be the next patch to land on stable/icehouse | 23:03 |
| dansmith | icehouse already has the neutron callback, unless you're talking about something else | 23:03 |
| lifeless | dansmith: ah, I did have memory fail :) | 23:05 |
| *** otherwiseguy has quit IRC | 23:05 | |
| *** terryw has joined #openstack-infra | 23:05 | |
| *** annegentle has quit IRC | 23:05 | |
| *** jamespag` has quit IRC | 23:05 | |
| *** gnuoy has quit IRC | 23:05 | |
| *** reed has quit IRC | 23:06 | |
| *** eglynn-office has quit IRC | 23:07 | |
| openstackgerrit | A change was merged to openstack-infra/config: Allow zuul & nodepool to be installed on one VM https://review.openstack.org/116373 | 23:08 |
| lifeless | well, we'll see what happens | 23:09 |
| clarkb | pleia2: arg I didn't get to 2.6.0 stable install yet | 23:09 |
| clarkb | pleia2: I am going to try that really quicly now | 23:09 |
| *** hasharMaybeWMF has quit IRC | 23:09 | |
| *** eglynn-office has joined #openstack-infra | 23:09 | |
| pleia2 | clarkb: no worries | 23:10 |
| lifeless | but at the heart of this I am concerned I don't have the depth of time to fix stable/icehouse gate instability | 23:10 |
| *** msabramo has joined #openstack-infra | 23:10 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 23:10 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 23:12 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Use OrderedDict to preserve key order in created dicts https://review.openstack.org/107063 | 23:12 |
| *** mmaglana has quit IRC | 23:12 | |
| *** david-lyle has quit IRC | 23:14 | |
| *** gnuoy has joined #openstack-infra | 23:14 | |
| clarkb | pleia2: done, that was quicker than I expected | 23:14 |
| *** flaper87 is now known as flaper87|afk | 23:14 | |
| clarkb | pleia2: if you login I will superpootle you | 23:14 |
| clarkb | pleia2: and confirmed I can view my own profile when logged in | 23:16 |
| pleia2 | clarkb: lyz created | 23:16 |
| clarkb | pleia2: done | 23:17 |
| pleia2 | thanks :) | 23:17 |
| clarkb | pleia2: was the .pot upload supposed to be fixed too? | 23:18 |
| *** Sukhdev has quit IRC | 23:19 | |
| clarkb | I am going to give that a go | 23:19 |
| pleia2 | clarkb: yep | 23:19 |
| *** Sukhdev has joined #openstack-infra | 23:19 | |
| clarkb | no spinning thing so thats progress | 23:20 |
| pleia2 | \o/ | 23:20 |
| clarkb | I don't see any strings to translate though | 23:21 |
| clarkb | there are a ton of warnings in the log | 23:21 |
| clarkb | I will look next week | 23:21 |
| openstackgerrit | A change was merged to openstack/requirements: Add the tuskar and python-tuskarclient projects https://review.openstack.org/115640 | 23:21 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 23:21 |
| openstackgerrit | A change was merged to openstack/requirements: Bump oslo.db to >= 0.4.0 https://review.openstack.org/115626 | 23:21 |
| pleia2 | clarkb: ok, thanks | 23:22 |
| pleia2 | we pretty much have zanata going now (just some things to tidy up, I'm sure they'll get to it on their Monday (Sunday)) and that'll give them something to start with | 23:22 |
| clarkb | pleia2: oh now I have strings | 23:23 |
| clarkb | pleia2: I guess it is laggy? | 23:23 |
| clarkb | pleia2: so it may be ready now. clarktest is a thing with nova.pot in it and afrikaans | 23:23 |
| *** msabramo1 has joined #openstack-infra | 23:25 | |
| pleia2 | cool | 23:26 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:27 | |
| pballand | clarkb, lifeless: thanks | 23:27 |
| *** msabramo has quit IRC | 23:28 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/jeepyb: Be more careful about when to close pull requests https://review.openstack.org/116414 | 23:31 |
| *** SumitNaiksatam has quit IRC | 23:31 | |
| *** chuckC has quit IRC | 23:31 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:31 | |
| *** hogepodge has joined #openstack-infra | 23:33 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 23:34 |
| *** annegentle has joined #openstack-infra | 23:36 | |
| *** baoli has joined #openstack-infra | 23:38 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 23:38 |
| pleia2 | clarkb: btw, changed admin's default password because it was abysmal | 23:38 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Ensure dict orders are deterministic https://review.openstack.org/107063 | 23:38 |
| clarkb | pleia2 thanks | 23:39 |
| * fungi imagines that the initial login was really "admin/abysmal" | 23:41 | |
| *** annegentle has quit IRC | 23:41 | |
| *** rcarrillocruz has quit IRC | 23:42 | |
| *** baoli has quit IRC | 23:43 | |
| *** lcheng has quit IRC | 23:45 | |
| *** rcarrillocruz has joined #openstack-infra | 23:45 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Switch taskflow to use the unittests slave script https://review.openstack.org/116388 | 23:46 |
| harlowja | clarkb fungi if u can recheck ^ seems like i need (?) to add a job template? | 23:46 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Some tweaks to get closer to Python 3 compat https://review.openstack.org/87810 | 23:47 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 23:47 |
| *** terryw has quit IRC | 23:47 | |
| pleia2 | fungi: tsk, now I need to change it again | 23:49 |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Error on duplicate job names being found https://review.openstack.org/56734 | 23:49 |
| pleia2 | ;) | 23:49 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 23:49 |
| *** dimsum_ has joined #openstack-infra | 23:50 | |
| jroll | our "reverify" comments don't seem to be working, known issue? https://review.openstack.org/#/c/116103/ | 23:50 |
| JayF | jroll: I'm digging deep, but I think recheck may be used now for verifys as well | 23:51 |
| pleia2 | jroll: they work, jenkins just doesn't tell you anymore when it's starting jobs | 23:51 |
| JayF | aha | 23:51 |
| jroll | oh huh | 23:51 |
| jroll | thanks pleia2 | 23:51 |
| pleia2 | jroll: search for the change on the zuul status page if you want to confirm it's running: http://status.openstack.org/zuul/ | 23:51 |
| jroll | :) | 23:51 |
| jroll | yeah :) | 23:51 |
| pleia2 | sure thing :) | 23:51 |
| JayF | jroll: it already failed another voting check :( | 23:51 |
| jroll | goody | 23:52 |
| pleia2 | and yeah, recheck and reverify are synonyms now | 23:52 |
| *** msabramo1 has left #openstack-infra | 23:52 | |
| JayF | jroll: check-tempest-dsvm-neutron-icehouse | 23:52 |
| JayF | jroll: has already failed on that check per the great zuul | 23:52 |
| jroll | I don't care until I can recheck it :) | 23:52 |
| clarkb | harlowja: oh right because we have py26 and py27 et al as special jobs | 23:53 |
| clarkb | harlowja: so a generic template for unittests may need to be added | 23:53 |
| harlowja | clarkb ya, i think i added it, not sure if i got it right though ;) | 23:54 |
| *** dimsum_ has quit IRC | 23:54 | |
| *** tomoe_ has joined #openstack-infra | 23:54 | |
| openstackgerrit | Darragh Bailey proposed a change to openstack-infra/jenkins-job-builder: Support lazy resolving of include yaml tags https://review.openstack.org/63580 | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!