| clarkb | fungi: jogo: that logging should be a noop | 00:00 |
|---|---|---|
| lifeless | fungi: print is dropped in 3 | 00:00 |
| lifeless | fungi: | 00:00 |
| fungi | clarkb: the legacy function should be a no-op yes | 00:00 |
| lifeless | fungi: % is more complex, but print with nothing else is a clear syntax error on 3 | 00:01 |
| fungi | clarkb: i think that the reason jogo is adding that is to confirm it doesn't actually get run on icehouse, but that whole section should be able to come out now that havana is gone | 00:01 |
| fungi | lifeless: oh, you're right, no idea why i didn't spot the missing () | 00:02 |
| fungi | i cannot for the life of me figure out why https://review.openstack.org/130001 is consistently failing the apply-precise job. i've even gone so far as to manually test the vhost config on static.o.o temporarily so apache shouldn't be barfing... http://paste.openstack.org/show/123348/ | 00:03 |
| clarkb | fungi: jogo: imo we should just run the script to check that | 00:03 |
| clarkb | no need to go through an entire image build | 00:03 |
| *** tsg has joined #openstack-infra | 00:03 | |
| *** zz_dimtruck is now known as dimtruck | 00:03 | |
| fungi | clarkb: good point, it's basically atomic | 00:03 |
| clarkb | I will give it a go on my test vm | 00:04 |
| *** asettle has quit IRC | 00:04 | |
| jogo | fungi: yes the legacy print is a sanity check | 00:04 |
| jogo | clarkb: run which script? | 00:04 |
| clarkb | jogo: that script you are editing | 00:04 |
| clarkb | we don't need a change to debug this. ou can just run the script | 00:05 |
| clarkb | I am going to do this now | 00:05 |
| jogo | clarkb: cool | 00:05 |
| jogo | clarkb: wasn't sure what env to run it in etc | 00:05 |
| fungi | yeah, the prep scripts just run that and it dances on a devstack checkout to download things | 00:05 |
| clarkb | jogo: you should just need the two dirs that are listed in that script | 00:06 |
| *** marun has quit IRC | 00:06 | |
| clarkb | jogo: it may need root but I am not sure yet | 00:06 |
| *** unicell has joined #openstack-infra | 00:07 | |
| fungi | worst case you can change them in the script to something you can write in | 00:07 |
| fungi | and replace the whet calls with print statements | 00:07 |
| fungi | s/and/or/ | 00:07 |
| fungi | i give up on typing | 00:07 |
| fungi | http://logs.openstack.org/01/130001/1/check/gate-infra-puppet-apply-precise/b5d3ac2/console.html#_2014-10-21_19_31_32_524 really looks to me like it's just failed to install apache2 at all | 00:08 |
| *** gokrokve has quit IRC | 00:09 | |
| clarkb | jogo: what is the name of the image we are looking for? | 00:09 |
| fungi | clarkb: Fedora-i386-20-20131211.1-sda.qcow2 | 00:09 |
| *** gokrokve has joined #openstack-infra | 00:09 | |
| fungi | that's the one it uses in the stable/icehouse branch | 00:10 |
| fungi | specifically https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/i386/Fedora-i386-20-20131211.1-sda.qcow2 | 00:11 |
| clarkb | ty | 00:12 |
| clarkb | hrm do we use the trove mysql image | 00:12 |
| clarkb | SlickNik: ^ | 00:12 |
| clarkb | (it seems to be big) | 00:12 |
| clarkb | and i am not sure if we are actually able to boot a nested vm capable or running mysql | 00:12 |
| *** ChuckC___ has quit IRC | 00:13 | |
| clarkb | I am also working on upgrading 05 and 04 concurrently | 00:13 |
| *** ChuckC___ has joined #openstack-infra | 00:13 | |
| clarkb | Running: ['wget', '-nv', '-c', 'https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/i386/Fedora-i386-20-20131211.1-sda.qcow2', '-O', '/home/ubuntu/cache/files/Fedora-i386-20-20131211.1-sda.qcow2'] | 00:15 |
| fungi | oh, hah. found my issue. the puppetlabs-apache module demands a docroot even if you don't plan to use one in your custom vhost template | 00:15 |
| clarkb | so I think it is working | 00:15 |
| clarkb | going to let it finish before I declare success (maybe it manages to delete that image somehow?) | 00:15 |
| clarkb | Fedora-i386-20-20131211.1-sda.qcow2: QEMU QCOW Image (v2), 2147483648 bytes | 00:15 |
| *** msabramo2 has quit IRC | 00:16 | |
| clarkb | so I dunno should be working | 00:16 |
| jogo | that isn't what the nodepool logs show | 00:16 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/system-config: Host a devstack.org redirect on static https://review.openstack.org/130001 | 00:16 |
| clarkb | ya I have to go look there next and probably look on a node too | 00:16 |
| clarkb | note this was a trusty test vm not precise which is wactually where this needs to be iirc | 00:17 |
| *** tsg has quit IRC | 00:17 | |
| SlickNik | clarkb: Yup, we use the trove mysql image and can spin the guest vm using it. | 00:17 |
| jogo | ohh good point | 00:17 |
| jogo | I was doing the wrong test locally too | 00:17 |
| SlickNik | clarkb: Sorry, just jumped on so don't have much context. Reading scrollback.... | 00:17 |
| clarkb | SlickNik: on the gate slaves? | 00:18 |
| SlickNik | clarkb: We're not using it in the gate (yet). | 00:19 |
| SlickNik | specifically in the d-g tests | 00:19 |
| clarkb | ok so maybe w eshouldn't be caching that everywhere now | 00:19 |
| clarkb | fungi: jeblair doing 05 and 04 right now | 00:19 |
| clarkb | should be up in a minute or two | 00:19 |
| jogo | clarkb: the theory I am working under is image_list doesn't work under precise for some reason | 00:21 |
| fungi | clarkb: excellent | 00:22 |
| *** gokrokve has quit IRC | 00:22 | |
| jogo | clarkb: yup | 00:23 |
| jogo | double checking | 00:23 |
| jogo | [ERROR] /home/devstack/functions-common:329 Support for -. is incomplete: no support for finding if a package is installed | 00:23 |
| jogo | http://pastebin.com/wMTA7WUu | 00:23 |
| *** gokrokve has joined #openstack-infra | 00:24 | |
| jogo | from a 12.04.5 LTS docker container | 00:24 |
| clarkb | cool its a devstack bug :) | 00:24 |
| *** tomoe has joined #openstack-infra | 00:24 | |
| jogo | hmm hold on | 00:24 |
| jogo | false alarm | 00:24 |
| jogo | I think | 00:24 |
| jogo | as that was on master branch | 00:25 |
| clarkb | 04 and 05 are both back up and idle because zuul really has nothing to do? | 00:25 |
| jogo | clarkb: yeah, something is funny because that was master branch of devstack | 00:25 |
| jogo | and it looks like those images are working | 00:25 |
| clarkb | jogo: it would still bail out cache_devstack early | 00:25 |
| clarkb | I don't think its anything funny | 00:25 |
| clarkb | devstack provides a public interface that is broken | 00:25 |
| clarkb | wow zuul really is just happily doing a small amount of work | 00:26 |
| *** lbiasi has joined #openstack-infra | 00:26 | |
| jogo | clarkb: then how come we are able to cache some images? | 00:26 |
| ianw | fungi: i hit that docroot thing in the puppet apache too | 00:27 |
| fungi | ianw: heh | 00:27 |
| *** bknudson has joined #openstack-infra | 00:27 | |
| ianw | fungi: which is why i ended up at https://review.openstack.org/#/c/129496/ | 00:27 |
| clarkb | jogo: are we caching some images? | 00:28 |
| fungi | ianw: do you happen to know what's up with epel on the hpcloud centos6 images? error i posted about an hour ago | 00:28 |
| fungi | 2014-10-22 22:53:57,369 INFO nodepool.image.build.hpcloud-b1.bare-centos6: Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again | 00:28 |
| ianw | fungi: sorry, my /tmp filled up and everything went to heck, i've just finished rebooting ... what was wrong? | 00:28 |
| jogo | grep nodepool.image.build.rax-ord.devstack-precise image.log.2014-10-21 | grep "\-nv" | pastebinit | 00:29 |
| fungi | ianw: we've been unable to update nodepool images for bare-centos6 in hpcloud for about a week and they're dying with that error | 00:29 |
| ianw | fungi: hmm, that looks like what happens when there's a new epel release | 00:29 |
| clarkb | jogo: /home/jenkins/cache/files is empty | 00:29 |
| jogo | http://paste.debian.net/128227/ | 00:29 |
| clarkb | do we mv files out of there when running? that may explain it being empty | 00:29 |
| *** david_lyle__ has joined #openstack-infra | 00:30 | |
| * clarkb spins up a precise VM | 00:30 | |
| fungi | where was the change that started installing zuul on all the workers? | 00:31 |
| *** david_lyle__ is now known as david_lyle | 00:31 | |
| fungi | i'm struggling to figure out where we added it | 00:31 |
| clarkb | fungi: it would be in nodepool/scripts | 00:31 |
| jogo | clarkb: I didn't see anything | 00:31 |
| clarkb | fungi: now living under project-config | 00:31 |
| ianw | fungi / clarkb : ok downloading the logs to see what's up with bare-centos6 ... https://review.openstack.org/129096 or https://review.openstack.org/130064 would help :) | 00:31 |
| clarkb | jogo: didn't see anything where? | 00:31 |
| *** bknudson has quit IRC | 00:31 | |
| jogo | clarkb: any 'mv' from cache/jenkins | 00:32 |
| fungi | clarkb: aha, found. i had a stale git checkout | 00:32 |
| *** msabramo has joined #openstack-infra | 00:33 | |
| *** david-lyle_ has quit IRC | 00:33 | |
| *** rmcall_ has joined #openstack-infra | 00:34 | |
| clarkb | jogo: I think line 8 of your paste captures the problem | 00:34 |
| *** rmcall has quit IRC | 00:35 | |
| *** rmcall_ is now known as rmcall | 00:35 | |
| clarkb | jogo: or is a side effect of what is going on | 00:35 |
| jogo | clarkb: that line is because there is a blank line at the end of what devstack returns | 00:36 |
| *** shashankhegde has quit IRC | 00:36 | |
| *** krotscheck has quit IRC | 00:36 | |
| clarkb | hrm I get that on trusty too and it works ok. odd | 00:36 |
| clarkb | we should guard against that in cache_devstack I guess | 00:36 |
| openstackgerrit | Benjamin Staffin proposed a change to openstack-infra/jenkins-job-builder: Add support for the S3 Publisher plugin https://review.openstack.org/128167 | 00:37 |
| *** krotscheck has joined #openstack-infra | 00:37 | |
| clarkb | its a good thing comcast doesn't have quotas | 00:37 |
| clarkb | oh wait | 00:37 |
| jogo | clarkb: so clarkb on most runs the images are cached | 00:39 |
| jogo | http://logs.openstack.org/94/130394/1/check/check-tempest-dsvm-postgres-full/d8b0b55/logs/devstacklog.txt.gz#_2014-10-22_23_39_25_179 | 00:39 |
| jogo | note no wgets for images | 00:39 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/project-config: Use Python 2.7 when installing zuul on workers https://review.openstack.org/130406 | 00:39 |
| fungi | that ^ should fix the py3k-precise nodepool image updates | 00:39 |
| clarkb | jogo: that is a trusty box | 00:39 |
| clarkb | jogo: I am 99.99% sure it works fine on trusty | 00:39 |
| jogo | clarkb: yeah, that shows its working on trusty | 00:40 |
| clarkb | fungi: that will break on centos6 | 00:40 |
| fungi | oh crappity | 00:40 |
| fungi | you're right | 00:40 |
| clarkb | fungi: you may be able to do python2 | 00:40 |
| fungi | checking | 00:40 |
| clarkb | at least on ubuntu that is valid | 00:40 |
| fungi | clarkb: yep! works on py3k-precise and bare-centos6 so should be fine. amending now | 00:42 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/project-config: Use Python 2 when installing zuul on workers https://review.openstack.org/130406 | 00:43 |
| fungi | clarkb: thanks! | 00:43 |
| patrickeast | hey everyone, i figure I’ll ask in here since you guys probably know… is there an irc channel or something specific to devstack? I’m hoping to collect some reviews for a blocking issue for my third party ci but am not sure where/who to ask | 00:43 |
| *** asettle has joined #openstack-infra | 00:43 | |
| fungi | patrickeast: #openstack-qa is what you want | 00:43 |
| patrickeast | fungi: sweet, thx | 00:43 |
| clarkb | if at first you don't succeed: sudo | 00:45 |
| clarkb | yay for VMs | 00:45 |
| fungi | yay for being able to delete your horrible security mistakes as soon as you are done ;) | 00:45 |
| clarkb | I am practicing this art | 00:46 |
| clarkb | since jenkins/zuul are so quiet and I am going to put 01 02 and 03 all in shutdown mode now | 00:46 |
| *** rmcall has quit IRC | 00:47 | |
| *** bknudson has joined #openstack-infra | 00:48 | |
| clarkb | (note the jenkins plugin updates are not related to my inappropriate use of sudo) | 00:48 |
| fungi | great idea | 00:48 |
| fungi | on both fronts ;) | 00:48 |
| ianw | fungi: ok, centos6 issue comes down to not being able to make a ssl connection | 00:49 |
| ianw | anyone here able to get a bug report to people in hpcloud not via support robots? | 00:49 |
| clarkb | mordred: ^ | 00:50 |
| *** MarkAtwood has joined #openstack-infra | 00:50 | |
| ianw | updating the image is one solution, we can do a work-around of updating certificates | 00:50 |
| fungi | that's kind of sad if it ends up being necessary | 00:50 |
| *** dkliban is now known as dkliban_afk | 00:51 | |
| fungi | maybe they've "fixed" it by uploading an even newer centos 6 image in their catalog and we just need to configure nodepool to start using it? | 00:51 |
| ianw | fungi: no, it still happens with their latest image, just tried it | 00:51 |
| *** gokrokve_ has joined #openstack-infra | 00:51 | |
| *** gokrokve_ has quit IRC | 00:51 | |
| fungi | ick | 00:51 |
| *** gokrokve_ has joined #openstack-infra | 00:52 | |
| jogo | clarkb: heh was missing lsb-release | 00:53 |
| clarkb | virsh hates me | 00:53 |
| *** gokrokve has quit IRC | 00:53 | |
| jogo | clarkb: well no luck, now that I instaleld lsb-release, tools/image_list.sh works on stable/icehouse | 00:54 |
| clarkb | jogo: so it downloads the missing image? | 00:54 |
| clarkb | jogo: try running cache_devstack.py precise | 00:54 |
| tchaypo | anteaya: i always interpret "virsh" as being a version of vi that runs in rsh | 00:54 |
| tchaypo | I'm not sure why that was addressed to anteaya | 00:56 |
| clarkb | of course there is no file or directly virsh that storage pool was delted | 00:56 |
| jogo | clarkb: checking now | 00:56 |
| *** mfer has joined #openstack-infra | 00:58 | |
| *** ChuckC has quit IRC | 00:58 | |
| clarkb | apparently virsh will let you kill a storage pool that belongs to a domain then complain about it later | 00:59 |
| clarkb | seriously if this was going to be a problem why didn't you tell me before? | 00:59 |
| *** teran has joined #openstack-infra | 00:59 | |
| clarkb | I might need to rage quit on virhs right now. no wonder docker is so popular | 01:00 |
| *** ChuckC___ has quit IRC | 01:01 | |
| jogo | http://blog.docker.com/2014/04/docker-in-ubuntu-ubuntu-in-docker/ | 01:01 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/project-config: Update ca-certificates for HP Cloud centos6 images https://review.openstack.org/130410 | 01:02 |
| ianw | fungi: ^ | 01:02 |
| *** mfer has quit IRC | 01:03 | |
| *** patrickeast has left #openstack-infra | 01:03 | |
| fungi | ianw: looks safe to me--thanks! | 01:03 |
| fungi | between that one and my py3k fix, we should have most of our regularly-used images successfully updating again | 01:04 |
| jogo | fungi clarkb: have you thought about making run_local fail on non 0 exit status? | 01:04 |
| jogo | because that seems to be hiding things | 01:04 |
| *** teran has quit IRC | 01:05 | |
| jogo | clarkb: got things running | 01:05 |
| *** ChuckC has joined #openstack-infra | 01:06 | |
| jogo | the apt-get part is going a bit slow | 01:06 |
| clarkb | jogo: not really. we want to use dib instead | 01:06 |
| fungi | jogo: i'm generally a fan of aborting on any error unless we can identify and whitelist the specific error condition. perhaps run_local needs to be passed an allowed list of exit codes which defaults to just 0 | 01:06 |
| fungi | but yeah, dib it the future | 01:06 |
| fungi | and the future is now | 01:06 |
| fungi | or tomorrow. sometimes it's yesterday... but mostly nowish | 01:06 |
| *** gokrokve_ has quit IRC | 01:07 | |
| *** MarkAtwood has quit IRC | 01:09 | |
| jogo | clarkb: output from my test | 01:09 |
| jogo | http://paste.openstack.org/show/123363 | 01:09 |
| clarkb | jogo: ls ~/cache/files? | 01:10 |
| *** MarkAtwood has joined #openstack-infra | 01:11 | |
| jogo | directory doesn't exist | 01:11 |
| clarkb | you should create it. that is where everything is cached to | 01:11 |
| jogo | running again | 01:12 |
| jogo | but what I saw is different then what the nodepool logs had | 01:12 |
| jogo | which is strange by itself | 01:12 |
| *** signed8bit has joined #openstack-infra | 01:13 | |
| *** MarkAtwood has quit IRC | 01:14 | |
| fungi | it is that time of the evening when i finally get to hopefully unwind most of the not-yet-completed browser tabs i accumulated over the course of the day | 01:14 |
| jogo | heh we install docker | 01:14 |
| clarkb | jogo: some jobs use it | 01:15 |
| clarkb | jogo: and it should only download it with the cache script | 01:15 |
| clarkb | then devstack intsalls it | 01:15 |
| jogo | clarkb: right | 01:15 |
| jogo | its just sorta funny but makes sense | 01:15 |
| *** yaguang has joined #openstack-infra | 01:16 | |
| *** jergerber has quit IRC | 01:16 | |
| clarkb | I am running in docker now too | 01:16 |
| clarkb | while simultaneously trying to rebuild my virsh stuff | 01:16 |
| clarkb | when the new NUCs arrive I am just going to do what sdad does with the NUC | 01:17 |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 01:18 | |
| *** lbiasi has quit IRC | 01:18 | |
| clarkb | I had to install lsb-release, git, wget, and sudo | 01:18 |
| *** asettle has quit IRC | 01:19 | |
| jogo | clarkb: that sounds right | 01:19 |
| clarkb | ok back shortly. getting beer | 01:20 |
| *** gokrokve has joined #openstack-infra | 01:20 | |
| *** arnaud__ has quit IRC | 01:21 | |
| *** arnaud has quit IRC | 01:21 | |
| clarkb | ianw: any chance you can review https://review.openstack.org/130406 I am positive it will work on ubuntu releases and fungi checked centos6. What about 7 and f20? | 01:21 |
| clarkb | fungi: if ianw sas it should work I will approve your change | 01:21 |
| fungi | clarkb: cool. i suspect it _should_ be fine as those are both newer than centos 6 and the python2 link is supposedly the disambiguation way forward in pythonland, decided for a while now | 01:22 |
| jogo | clarkb: http://paste.ubuntu.com/8634373/ | 01:22 |
| clarkb | jogo: cool so it worked there | 01:22 |
| ianw | clarkb: checking... | 01:23 |
| jogo | clarkb: locally | 01:23 |
| clarkb | jogo: ya | 01:23 |
| jogo | clarkb: but not on the nodepool thinggy | 01:23 |
| clarkb | jogo: right so it could be an issue of the base image we start with | 01:23 |
| clarkb | or the underlying VM eg maybe we run out of disk? | 01:23 |
| jogo | clarkb: also docker logs | 01:23 |
| jogo | to dump your logs | 01:23 |
| *** gokrokve has quit IRC | 01:24 | |
| anteaya | tchaypo: me either, i have no opinion on virsh | 01:25 |
| clarkb | ok really going to rest eyes for a few minutes | 01:25 |
| clarkb | back when stuff is done churning | 01:25 |
| clarkb | fungi: I take it no opposition from your for me finishing up jenkins plugin upgrades tonight? | 01:25 |
| *** stevemar has joined #openstack-infra | 01:26 | |
| fungi | clarkb: no opposition at all | 01:26 |
| *** gokrokve_ has joined #openstack-infra | 01:27 | |
| jogo | clarkb: when forcing _legacy_find_images to run, it still finds Fedora-x86_64-20-20131211.1-sda.qcow2 | 01:27 |
| *** jerryz has joined #openstack-infra | 01:27 | |
| *** Longgeek has joined #openstack-infra | 01:28 | |
| clarkb | jogo: ok it just ended here I have reproduced successful run on precise docker container | 01:29 |
| jogo | clarkb: same result as me? | 01:30 |
| clarkb | ya it appears to have worked fine | 01:30 |
| jogo | I am confused | 01:30 |
| clarkb | me too | 01:30 |
| clarkb | I am going to poke around on devstack-precise nodes a bit more | 01:31 |
| jogo | clarkb: checking older logs to see if this was just a bad build | 01:31 |
| clarkb | ok I just confirmed its not in ~/cache/files on a node that hasn't really gotten into d-g yet | 01:32 |
| clarkb | so I don't think its disappearing post boot | 01:32 |
| fungi | clarkb: any guess what dream mordred meant "stdout" to represent in https://review.openstack.org/#/c/129983/1/launch/utils.py ? there's no local stdout var and i seriously dount it was sys.stdout | 01:32 |
| clarkb | whcih lines up with your log greps | 01:32 |
| fungi | s/dount/doubt/ | 01:33 |
| *** Longgeek has quit IRC | 01:33 | |
| clarkb | fungi: I think that must be expected to be a parameter to that function | 01:33 |
| fungi | aha, i bet he meant to pass output in from the caller | 01:33 |
| clarkb | so you run puppet and get a tuple of return code and output | 01:34 |
| fungi | yep | 01:34 |
| clarkb | then pass that tuple to function | 01:34 |
| jogo | clarkb: how odd | 01:34 |
| jogo | clarkb: and cache/files isn't empty | 01:34 |
| clarkb | no there are other images there | 01:34 |
| clarkb | and then it goes empt when we we move them into devstack | 01:34 |
| clarkb | but prior to that that image isn't there | 01:34 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/system-config: Add detailed-exitcodes to launch_node https://review.openstack.org/129983 | 01:35 |
| tchaypo | anteaya: remember me asking about the check-tripleo-jobs template yesterday? | 01:35 |
| fungi | sometimes there's more puzzle to fixing those reviews than i can handle this late at night | 01:35 |
| tchaypo | I was getting really confused becaues it wasn't in the file anwhere.. but then I rebased and it appeared :) | 01:35 |
| *** dane_leblanc has joined #openstack-infra | 01:38 | |
| *** baoli has joined #openstack-infra | 01:38 | |
| clarkb | ok I am really stumped as to why this caceh_devstack.py thing isn't working | 01:38 |
| clarkb | jogo: fungi: any chance we know if it is a rax vs hpcloud thing? | 01:38 |
| fungi | clarkb: seems to be an everywhere thing | 01:39 |
| fungi | clarkb: jogo grepped the nodepool image log and none of the image updates seem to have pulled that file | 01:39 |
| clarkb | oh right | 01:39 |
| clarkb | though I wonder how far back that log went? I will dig in logs more | 01:40 |
| fungi | looked like it was just from today | 01:40 |
| jogo | I checked as far back as 9-22 | 01:40 |
| fungi | oh, you pulled older logs too? | 01:40 |
| jogo | pulled a few logs | 01:40 |
| clarkb | oh nice | 01:40 |
| jogo | spot checked some | 01:40 |
| fungi | that helps | 01:40 |
| clarkb | ok so its been broken for a month ish? | 01:40 |
| jogo | was hoping it was a one time thing | 01:40 |
| jogo | at least | 01:40 |
| clarkb | (at least) | 01:40 |
| clarkb | jinx | 01:40 |
| fungi | okay, i think i'm going to call it a night. later everyone | 01:41 |
| clarkb | goodnight | 01:41 |
| anteaya | tchaypo: good work | 01:42 |
| openstackgerrit | James Polley proposed a change to openstack-infra/project-config: Add the prep-source-repos project https://review.openstack.org/129533 | 01:42 |
| *** Ryan_Lane has quit IRC | 01:42 | |
| *** xchu has joined #openstack-infra | 01:43 | |
| *** yamahata has joined #openstack-infra | 01:43 | |
| tchaypo | SpamapS: ^^^ | 01:44 |
| anteaya | tchaypo: for upstream url you have: git@github.com:jamezpolley/prep-source-repos.git | 01:44 |
| anteaya | tchaypo: we need git:// or http:// or https:// | 01:44 |
| tchaypo | oh | 01:44 |
| anteaya | and thanks for adding the .git extention | 01:44 |
| jogo | clarkb: I am going to call it a day very soon too, getting close to dinner time | 01:45 |
| anteaya | look at the examples fore and aft of your addition in the file | 01:45 |
| clarkb | jogo: no problem we can pick this up tomorrow | 01:45 |
| clarkb | it is a very odd one | 01:45 |
| tchaypo | any preference between git:// and https:// ? | 01:45 |
| anteaya | not from me | 01:46 |
| jogo | very | 01:46 |
| jogo | night | 01:46 |
| anteaya | night jogo | 01:46 |
| *** mmedvede has quit IRC | 01:48 | |
| openstackgerrit | James Polley proposed a change to openstack-infra/project-config: Add the prep-source-repos project https://review.openstack.org/129533 | 01:49 |
| *** xchu has quit IRC | 01:49 | |
| *** bradjones has quit IRC | 01:50 | |
| anteaya | tchaypo: sorry doing this a line at a time | 01:50 |
| *** bradjones has joined #openstack-infra | 01:50 | |
| tchaypo | heh | 01:50 |
| anteaya | you supply an acl file with the patch as well as point to acl-config: /home/gerrit2/acls/openstack/os-apply-config.config in the patch | 01:50 |
| anteaya | so I'm confused | 01:50 |
| *** signed8bit has quit IRC | 01:51 | |
| anteaya | which acl file do you want to use with this repo | 01:51 |
| tchaypo | not as confused as me. | 01:51 |
| anteaya | if you want to us os-apply-config.config you don't need the new .config file in the patch | 01:51 |
| anteaya | it is one or the other | 01:51 |
| tchaypo | I think I want to use the one I added, I probably forgot to change ther ference when I coped configs from o-a-c | 01:52 |
| *** groknix has quit IRC | 01:52 | |
| *** tomoe has quit IRC | 01:52 | |
| tchaypo | ther ference == the configs | 01:52 |
| tchaypo | but that reminds me | 01:53 |
| anteaya | okay so let me finish my review | 01:53 |
| clarkb | jenkins01 has just beenrestarted to upgrade gearman plugin | 01:53 |
| anteaya | but if you want to use the config you provided the acl-config: line will have to go | 01:53 |
| anteaya | clarkb: cool | 01:54 |
| clarkb | 2 and 3 will happen shortly | 01:54 |
| clarkb | I will leave jenkins.o.o for tomorrow I think | 01:54 |
| clarkb | since it is the singular jenkins better to do that one when other rooters are around | 01:54 |
| anteaya | good idea | 01:56 |
| anteaya | need more rooters | 01:56 |
| anteaya | and you should be resting your eyes I think | 01:56 |
| anteaya | should you not? | 01:56 |
| clarkb | anteaya: I did for a short bit. And will be having dinner as soonas this is done | 01:56 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Use Python 2 when installing zuul on workers https://review.openstack.org/130406 | 01:57 |
| *** dims has quit IRC | 01:58 | |
| *** dims has joined #openstack-infra | 01:58 | |
| *** teran has joined #openstack-infra | 02:00 | |
| *** gokrokve has joined #openstack-infra | 02:01 | |
| *** arnaud__ has joined #openstack-infra | 02:01 | |
| anteaya | clarkb: good | 02:01 |
| *** dmsimard_away is now known as dmsimard | 02:01 | |
| anteaya | eyes heal fast but they need to rest | 02:01 |
| *** gokrokve has quit IRC | 02:01 | |
| *** nosnos has joined #openstack-infra | 02:02 | |
| *** dims has quit IRC | 02:03 | |
| *** gokrokve_ has quit IRC | 02:04 | |
| clarkb | I am running the cache devstack script in a proper precise virsh domain now | 02:04 |
| clarkb | which should be more like our hpcloud nodes than a docker container | 02:04 |
| clarkb | let us see if that makes a difference | 02:04 |
| *** teran has quit IRC | 02:05 | |
| clarkb | and 01 02 and 03 are upgraded | 02:05 |
| clarkb | nwo time for dinner | 02:05 |
| tchaypo | that acls file references groups like pre-source-repose-core | 02:05 |
| tchaypo | I think that's probably wrong, I think I just want to re-use the acls from tripleo-incubator | 02:05 |
| anteaya | clarkb: enjoy dinner | 02:08 |
| anteaya | tchaypo: your choice | 02:08 |
| anteaya | tripleo has a number of acl choices | 02:08 |
| anteaya | would make sense to use one of them | 02:09 |
| *** yaguang has quit IRC | 02:11 | |
| tchaypo | I think to do that I ca remove the acl file and jjust point at whatever file the tripleo-incubator project uses? | 02:12 |
| anteaya | if you like the permissions the -incubator config uses | 02:13 |
| anteaya | good idea | 02:13 |
| *** dimtruck is now known as zz_dimtruck | 02:19 | |
| *** david_lyle has quit IRC | 02:24 | |
| *** david-lyle has joined #openstack-infra | 02:25 | |
| *** MarkAtwood has joined #openstack-infra | 02:25 | |
| *** liusheng has quit IRC | 02:27 | |
| *** asettle has joined #openstack-infra | 02:29 | |
| *** dane_leblanc has quit IRC | 02:31 | |
| *** yamamoto_ has joined #openstack-infra | 02:33 | |
| *** zaphod42 has quit IRC | 02:35 | |
| *** zaphod42 has joined #openstack-infra | 02:35 | |
| *** baoli has quit IRC | 02:37 | |
| *** zaphod42 has quit IRC | 02:40 | |
| *** esker has joined #openstack-infra | 02:44 | |
| *** esker has quit IRC | 02:46 | |
| *** ChuckC___ has joined #openstack-infra | 02:47 | |
| *** ChuckC___ has quit IRC | 02:50 | |
| *** frank_zhou has joined #openstack-infra | 02:50 | |
| *** rmcall has joined #openstack-infra | 02:51 | |
| *** bdpayne has joined #openstack-infra | 02:54 | |
| *** Longgeek has joined #openstack-infra | 03:00 | |
| *** zz_dimtruck is now known as dimtruck | 03:01 | |
| *** teran has joined #openstack-infra | 03:01 | |
| *** ivar-laz_ has joined #openstack-infra | 03:01 | |
| *** bdpayne has quit IRC | 03:02 | |
| *** Longgeek has quit IRC | 03:04 | |
| *** ivar-lazzaro has quit IRC | 03:05 | |
| *** homeless has quit IRC | 03:05 | |
| *** ivar-laz_ has quit IRC | 03:05 | |
| *** teran has quit IRC | 03:07 | |
| *** dimtruck is now known as zz_dimtruck | 03:10 | |
| *** julim has quit IRC | 03:12 | |
| *** tsg has joined #openstack-infra | 03:15 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 03:16 | |
| *** yamahata has quit IRC | 03:20 | |
| *** tsg has quit IRC | 03:21 | |
| *** r-daneel has joined #openstack-infra | 03:22 | |
| mordred | clarkb: cache_devstack.py hasn't worked for a month??? | 03:28 |
| *** r-daneel has quit IRC | 03:30 | |
| *** winston-d_zZZ is now known as winston-d | 03:31 | |
| *** freemanbrandon has quit IRC | 03:33 | |
| clarkb | not on precise nodes | 03:37 |
| clarkb | its not getting a fedora image for icehouse for some reason | 03:38 |
| clarkb | but running locally it works | 03:38 |
| *** harlowja is now known as harlowja_away | 03:39 | |
| *** arnaud__ has quit IRC | 03:39 | |
| mordred | clarkb: FANTASTIC | 03:40 |
| mordred | clarkb: that'll be eversomuch to track down | 03:40 |
| mordred | clarkb: also, cody-somerville just showed me this: http://wiki.baserock.org/overview/ | 03:44 |
| mordred | clarkb: I haven't touched it at all, but I think it's my new favorite thing | 03:44 |
| *** zaphod42 has joined #openstack-infra | 03:44 | |
| *** tomoe has joined #openstack-infra | 03:45 | |
| *** MarkAtwood has quit IRC | 03:47 | |
| *** hdd has joined #openstack-infra | 03:48 | |
| *** gokrokve has joined #openstack-infra | 03:49 | |
| *** harlowja_at_home has joined #openstack-infra | 03:55 | |
| clarkb | neat | 03:56 |
| *** nosnos has quit IRC | 04:00 | |
| *** nosnos has joined #openstack-infra | 04:01 | |
| *** gokrokve has quit IRC | 04:02 | |
| *** gokrokve has joined #openstack-infra | 04:02 | |
| *** teran has joined #openstack-infra | 04:02 | |
| *** freemanbrandon has joined #openstack-infra | 04:03 | |
| *** nosnos has quit IRC | 04:05 | |
| *** zaphod42 has quit IRC | 04:06 | |
| *** zaphod42 has joined #openstack-infra | 04:07 | |
| *** gokrokve has quit IRC | 04:07 | |
| *** teran has quit IRC | 04:07 | |
| viglesias | anybody have experience creating zuul configs? | 04:09 |
| viglesias | im new and trying to get my first hook to kick but not sure if my layout.yml is structured properly | 04:09 |
| *** garyh has quit IRC | 04:09 | |
| viglesias | http://hastebin.com/oyatulowir.sm | 04:10 |
| *** jergerber has joined #openstack-infra | 04:11 | |
| *** zaphod42 has quit IRC | 04:11 | |
| *** freemanbrandon has quit IRC | 04:12 | |
| *** talluri has joined #openstack-infra | 04:13 | |
| clarkb | that looks about right | 04:13 |
| *** talluri has quit IRC | 04:13 | |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml is ours | 04:14 |
| viglesias | thanks clarkb so i would need a job in jenkins called eutester-test right? | 04:17 |
| viglesias | thats where my understanding breaks down a bit | 04:17 |
| *** SumitNaiksatam has quit IRC | 04:18 | |
| * viglesias goes back to read more docs | 04:18 | |
| *** dmsimard is now known as dmsimard_away | 04:18 | |
| *** shashankhegde has joined #openstack-infra | 04:19 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:19 | |
| *** arnaud has joined #openstack-infra | 04:21 | |
| mordred | viglesias: yes. you would need a jenkins job called eutester-test | 04:21 |
| viglesias | ack thanks mordred! | 04:22 |
| viglesias | so far i think i have everybody talking to each other, just not seeing the jobs kick yet | 04:23 |
| viglesias | i got this far: | 04:23 |
| viglesias | http://hastebin.com/jeveqijapo.hs | 04:23 |
| viglesias | ah it has no workers maybe | 04:23 |
| mordred | viglesias: yah - was just about to say - it looks like jenkins may not have registered with zuul | 04:24 |
| mordred | viglesias: have you clicked the enable button on the gearman plugin config? | 04:24 |
| viglesias | yep | 04:24 |
| mordred | kk | 04:24 |
| viglesias | and did the connection test | 04:24 |
| viglesias | any way to check worker status in gearman/zuul? | 04:24 |
| mordred | yes - but I do not remember how - I normally let jeblair and clarkb tae care of things liek that :) | 04:25 |
| mordred | clarkb: jeblair and I just spent the day sitting with viglesias and talking the wonders of zuul - so he may have questions and stuff :) | 04:25 |
| viglesias | scary thought :) | 04:26 |
| mordred | hehe | 04:26 |
| viglesias | that was great though mordred really learned a lot | 04:26 |
| mordred | likewise | 04:26 |
| mordred | viglesias: btw - that test case management thing you use - is taht open source? | 04:26 |
| viglesias | for sure, testlink | 04:26 |
| mordred | thanks | 04:26 |
| * mordred goes to look | 04:26 | |
| clarkb | viglesias: connect to gearman on port 4730 with telnet or nc | 04:29 |
| *** Ryan_Lane has joined #openstack-infra | 04:29 | |
| viglesias | clarkb: ack, and zuul starts its own gearman right? | 04:29 |
| clarkb | then enter 'status' to get job listings the three numbers per job are queued, running and workers that can run job | 04:30 |
| clarkb | yup | 04:30 |
| viglesias | thanks clarkb i had my own gearman running which im guessing makes some sadness start to occur | 04:30 |
| *** Longgeek has joined #openstack-infra | 04:30 | |
| mordred | sad gearman | 04:31 |
| *** amotoki has joined #openstack-infra | 04:32 | |
| *** Longgeek has quit IRC | 04:35 | |
| *** asettle has quit IRC | 04:36 | |
| viglesias | ah found the way to check for workers | 04:38 |
| *** pcrews has quit IRC | 04:38 | |
| *** nosnos has joined #openstack-infra | 04:40 | |
| viglesias | so i have workers, ie i see 1 worker per executor in jenkins | 04:40 |
| *** shashankhegde has quit IRC | 04:44 | |
| *** yamamoto_ has quit IRC | 04:45 | |
| *** stevebaker has quit IRC | 04:45 | |
| *** stevebaker has joined #openstack-infra | 04:45 | |
| *** stevebaker has quit IRC | 04:45 | |
| *** stevebaker has joined #openstack-infra | 04:45 | |
| *** stevebaker has quit IRC | 04:45 | |
| *** stevebaker has joined #openstack-infra | 04:45 | |
| *** stevebaker has quit IRC | 04:45 | |
| harlowja_at_home | hmmm, when u guys gonna replace gearmean with taskflow ;) | 04:45 |
| *** stevebaker has joined #openstack-infra | 04:45 | |
| *** yamamoto_ has joined #openstack-infra | 04:47 | |
| *** otter768 has joined #openstack-infra | 04:49 | |
| *** pcrews has joined #openstack-infra | 04:51 | |
| *** yamahata has joined #openstack-infra | 04:55 | |
| *** stevemar has quit IRC | 04:55 | |
| *** stevemar has joined #openstack-infra | 04:56 | |
| *** harlowja_at_home has quit IRC | 04:56 | |
| *** yamahata has quit IRC | 04:57 | |
| *** yamahata has joined #openstack-infra | 04:57 | |
| viglesias | no luck getting the job to kick, i wonder if its related to trying to use the master vs an actual slave | 04:59 |
| viglesias | i would doubt it though since the workers are simply assigned to executors in jenkins | 04:59 |
| *** katyafervent_awa is now known as katyafervent | 05:01 | |
| *** talluri has joined #openstack-infra | 05:02 | |
| *** yfried_ has joined #openstack-infra | 05:02 | |
| *** gokrokve has joined #openstack-infra | 05:03 | |
| *** teran has joined #openstack-infra | 05:03 | |
| *** shashankhegde has joined #openstack-infra | 05:05 | |
| *** gokrokve has quit IRC | 05:08 | |
| *** teran has quit IRC | 05:08 | |
| *** jcoufal has joined #openstack-infra | 05:12 | |
| zaro | viglesias: you still running your own gearman or using the python gear implementation? | 05:12 |
| *** stevemar has quit IRC | 05:13 | |
| *** asettle has joined #openstack-infra | 05:14 | |
| viglesias | python gear | 05:15 |
| *** yamamoto_ has quit IRC | 05:15 | |
| viglesias | so right now i can see my workers in gearman (via telnet) and can see that zuul has jobs queued | 05:15 |
| viglesias | but they arent kicking jenkins | 05:16 |
| *** yamamoto_ has joined #openstack-infra | 05:16 | |
| *** tsg has joined #openstack-infra | 05:19 | |
| *** shashankhegde has quit IRC | 05:21 | |
| *** yfried_ has quit IRC | 05:22 | |
| viglesias | jenkins is definitely talking to gearman: http://hastebin.com/wuneqojura.avrasm | 05:23 |
| *** SumitNaiksatam has quit IRC | 05:24 | |
| *** bdpayne has joined #openstack-infra | 05:25 | |
| *** dims has joined #openstack-infra | 05:25 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:26 | |
| *** arnaud has quit IRC | 05:27 | |
| *** yfried_ has joined #openstack-infra | 05:28 | |
| *** dims has quit IRC | 05:30 | |
| openstackgerrit | Mehdi Abaakouk proposed a change to openstack-infra/project-config: oslo.messaging: Add experimental functional tests https://review.openstack.org/130370 | 05:31 |
| *** mbacchi has quit IRC | 05:35 | |
| *** katyafervent is now known as katyafervent_awa | 05:38 | |
| *** katyafervent_awa is now known as katyafervent | 05:39 | |
| *** hdd has quit IRC | 05:41 | |
| *** yfried_ has quit IRC | 05:43 | |
| *** jergerber has quit IRC | 05:44 | |
| *** Longgeek has joined #openstack-infra | 05:46 | |
| *** talluri has quit IRC | 05:49 | |
| *** MarkAtwood has joined #openstack-infra | 05:49 | |
| *** katyafervent is now known as katyafervent_awa | 05:50 | |
| *** Longgeek has quit IRC | 05:51 | |
| *** otter768 has quit IRC | 05:52 | |
| *** HeOS has quit IRC | 05:52 | |
| *** jcooley_ has joined #openstack-infra | 05:52 | |
| *** katyafervent_awa is now known as katyafervent | 05:53 | |
| *** mrmartin has joined #openstack-infra | 05:53 | |
| *** jcooley_ has quit IRC | 05:54 | |
| *** bdpayne has quit IRC | 05:58 | |
| *** talluri has joined #openstack-infra | 05:59 | |
| *** asettle has quit IRC | 06:01 | |
| *** e0ne has joined #openstack-infra | 06:01 | |
| *** gokrokve has joined #openstack-infra | 06:04 | |
| *** teran has joined #openstack-infra | 06:05 | |
| *** ildikov has quit IRC | 06:05 | |
| *** pcrews has quit IRC | 06:09 | |
| *** dmsimard_away has quit IRC | 06:11 | |
| *** mika has quit IRC | 06:11 | |
| *** openstack has joined #openstack-infra | 06:13 | |
| *** gokrokve has quit IRC | 06:15 | |
| *** teran has quit IRC | 06:15 | |
| *** dmsimardz has joined #openstack-infra | 06:15 | |
| *** dmsimardz is now known as dmsimard | 06:15 | |
| *** rushiagr_away has quit IRC | 06:15 | |
| *** toabctl has quit IRC | 06:15 | |
| *** toabctl_ has joined #openstack-infra | 06:15 | |
| *** mikap has joined #openstack-infra | 06:15 | |
| *** toabctl_ is now known as toabctl | 06:15 | |
| *** mikap is now known as mika | 06:15 | |
| *** grue_pm has joined #openstack-infra | 06:15 | |
| *** lifeless has quit IRC | 06:15 | |
| *** bookwar_ has joined #openstack-infra | 06:15 | |
| *** lifeless has joined #openstack-infra | 06:15 | |
| *** amcrn has quit IRC | 06:15 | |
| *** armax has quit IRC | 06:15 | |
| sileht | Hi infra teams, I have a issue with a missing tarball in our pypi mirror, if someone can take a look, it blocks the gate for gnocchi: https://bugs.launchpad.net/openstack-ci/+bug/1384553 | 06:21 |
| uvirtbot | Launchpad bug 1384553 in openstack-ci "SQLAlchemy-Utils 0.27.3 tarball is missing from many mirror" [Undecided,New] | 06:21 |
| *** tsg has quit IRC | 06:21 | |
| *** arnaud has joined #openstack-infra | 06:22 | |
| *** yfried_ has joined #openstack-infra | 06:25 | |
| *** Longgeek has joined #openstack-infra | 06:28 | |
| *** sdake_ has joined #openstack-infra | 06:29 | |
| *** e0ne has quit IRC | 06:30 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Adds homepage for API WG project https://review.openstack.org/130377 | 06:31 |
| *** yfried_ is now known as yfried|afk | 06:36 | |
| *** mrmartin has quit IRC | 06:36 | |
| *** yfried|afk is now known as yfried_ | 06:36 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Update ca-certificates for HP Cloud centos6 images https://review.openstack.org/130410 | 06:40 |
| *** dmsimard has quit IRC | 06:43 | |
| *** dmsimard_away has joined #openstack-infra | 06:44 | |
| *** sdake_ has quit IRC | 06:44 | |
| *** dmsimard_away is now known as dmsimard | 06:44 | |
| *** bo_sh has joined #openstack-infra | 06:45 | |
| *** ianw has quit IRC | 06:46 | |
| *** shashankhegde has joined #openstack-infra | 06:47 | |
| *** med_ has quit IRC | 06:47 | |
| *** ryanpetrello has quit IRC | 06:49 | |
| *** ryanpetrello has joined #openstack-infra | 06:49 | |
| *** sdake_ has joined #openstack-infra | 06:49 | |
| *** med_ has joined #openstack-infra | 06:50 | |
| *** med_ has quit IRC | 06:51 | |
| *** med_ has joined #openstack-infra | 06:51 | |
| *** shashankhegde has quit IRC | 06:51 | |
| *** coolsvap has quit IRC | 06:53 | |
| *** changbl has quit IRC | 06:53 | |
| *** krotscheck has quit IRC | 06:59 | |
| *** openstack has quit IRC | 06:59 | |
| *** openstack has joined #openstack-infra | 14:18 | |
| fungi | dougwig: the meetbot and statusbot are connected again. due to a ssl_ipv6 incompatibility in twisted we have to pin them to a specific irc server and the one we were using was apparently taken out of the chat.freenode.net rotation this morning. i've pinned them to a different freenode server now | 14:19 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Adds ironic grenade forward testing https://review.openstack.org/127321 | 14:19 |
| dougwig | fungi: ty | 14:19 |
| isviridov | fungi : great news. Thank yout | 14:19 |
| BobBall | Ah - actually it seems that we just inherited use of pypi.openstack.org with the config fork we're using? | 14:20 |
| mtreinish | fungi: heh, I feel like you end up spending a lot of time tracking down irc bots... :) | 14:21 |
| fungi | BobBall: possibly | 14:21 |
| *** marun has joined #openstack-infra | 14:21 | |
| fungi | mtreinish: yes, only slightly less time than it would take to rewrite them though | 14:21 |
| *** kgiusti has joined #openstack-infra | 14:22 | |
| fungi | bo_sh: to keep 4 months of compressed job logs on disk we use upwards of 12tb space, which is the maximum amount of cinder block storage rackspace supports attaching to a server instance | 14:22 |
| *** hdd has joined #openstack-infra | 14:23 | |
| *** cpowell has joined #openstack-infra | 14:23 | |
| *** cpowell has quit IRC | 14:23 | |
| fungi | bo_sh: by comparison, elasticsearch indexing of the same logs is many orders of magnitude larger than their compressed versions | 14:23 |
| *** cpowell has joined #openstack-infra | 14:24 | |
| fungi | bo_sh: we're working on new features in zuul and our log decorator frontend which will allow us to stick logs in a swift object store instead, which we intend to use to be able to retain at least 6 months (one development cycle's worth) of test logs | 14:25 |
| marun | lifeless: ping | 14:26 |
| marun | lifeless: we have our stacktrace for the subunit bug: http://logs.openstack.org/20/129720/1/check/check-neutron-dsvm-functional/3d02088/console.html#_2014-10-23_00_07_56_991 | 14:26 |
| fungi | bo_sh: instead of using block storage for the browsable logs i mean, not instead of logstash/elasticsearch | 14:26 |
| bo_sh | fungi: i probably didn't explain myself well. why not compressing the entire log directories after data sent in logs.openstack.org. | 14:28 |
| *** mmaglana has joined #openstack-infra | 14:28 | |
| *** gokrokve has quit IRC | 14:28 | |
| *** jergerber has joined #openstack-infra | 14:29 | |
| *** zeus has joined #openstack-infra | 14:29 | |
| fungi | bo_sh: it is compressed on logs.openstack.org | 14:30 |
| *** zeus is now known as Guest75691 | 14:30 | |
| *** tonytan4ever has joined #openstack-infra | 14:30 | |
| jeblair | fungi: i'm here for a few mins, then away again for most of today; anything you need from me? | 14:30 |
| *** dizquierdo has joined #openstack-infra | 14:30 | |
| fungi | jeblair: nothing exciting has come up yet | 14:30 |
| fungi | jeblair: just working through minor breaks of the usual stuff | 14:30 |
| *** lttrl has quit IRC | 14:30 | |
| *** lttrl has joined #openstack-infra | 14:31 | |
| bo_sh | fungi: not all of them. | 14:31 |
| fungi | jeblair: though the discussion at https://github.com/smontanaro/pylockfile/issues/11 might interest you | 14:31 |
| *** luqas has joined #openstack-infra | 14:31 | |
| fungi | bo_sh: most of the larger logs we compress prior to transferring them to logs.openstack.org | 14:31 |
| *** mfer has quit IRC | 14:31 | |
| *** otherwiseguy has joined #openstack-infra | 14:32 | |
| *** mfer has joined #openstack-infra | 14:32 | |
| *** mmaglana has quit IRC | 14:32 | |
| fungi | bo_sh: the rest we catch with that maintenance script running from cron. it compresses anything which needs compressing and then deletes anything older than our defined retention period | 14:32 |
| bo_sh | fungi: ok. :). thanks for making that clear. | 14:33 |
| *** lttrl has quit IRC | 14:33 | |
| *** dkliban_afk is now known as dkliban | 14:34 | |
| *** lttrl has joined #openstack-infra | 14:34 | |
| fungi | bo_sh: from the perspective of the logstash workers it seems like they're not compressed because apache serves them up in text/plain (with a filtering cgi we use there to strip out lines for loglevels below our interest threshold) | 14:34 |
| jeblair | fungi: gah! that's insane. maybe someone could ask github to just do it. | 14:35 |
| mtreinish | fungi: looking at that github link you posted could you do something like a user->you transfer, then add it to the openstack organization? | 14:36 |
| fungi | jeblair: yeah, i think it's pretty intractable without an intermediary | 14:36 |
| jeblair | mtreinish: oh good idea! | 14:36 |
| fungi | mtreinish: that's one option i considered, though i suspect github winds up creating two sets of redirects | 14:36 |
| fungi | curious whether they optimize that out, or whether the second transfer breaks the original entirely | 14:37 |
| *** Guest75691 is now known as zeus | 14:37 | |
| *** gokrokve has joined #openstack-infra | 14:37 | |
| *** zeus has quit IRC | 14:37 | |
| *** zeus has joined #openstack-infra | 14:37 | |
| jeblair | fungi: could test it (at least the breakage part) | 14:37 |
| fungi | or if it just ends up being chained through two redirects before getting to the target | 14:37 |
| fungi | yeah | 14:37 |
| *** wenlock has joined #openstack-infra | 14:38 | |
| jeblair | fungi: make 200 redirects and see if it takes 200x as long and also whether github sends you a nice email. <jk> ;) | 14:38 |
| *** lttrl has quit IRC | 14:38 | |
| fungi | maybe clarkb and i can ping-pong a repo between us and into stackforge temporarily later today | 14:38 |
| *** lttrl has joined #openstack-infra | 14:39 | |
| *** gokrokve has quit IRC | 14:39 | |
| *** mfer has quit IRC | 14:39 | |
| *** mfer has joined #openstack-infra | 14:40 | |
| *** sdake has quit IRC | 14:41 | |
| *** mfer is now known as mattfarina | 14:42 | |
| *** ZZelle has quit IRC | 14:42 | |
| *** ZZelle has joined #openstack-infra | 14:43 | |
| fungi | in good news, this means that if we want to use that feature to transfer projects from stackforge to openstack or openstack to openstack-attic or whatever, should be easy since it's the same admin teams in each org | 14:44 |
| jeblair | ya | 14:44 |
| jeblair | though i'd like us to just get out of that business; i'm thinking the big-tent discussion may provide an opportunity to do so | 14:45 |
| *** jpeeler has quit IRC | 14:45 | |
| *** david-lyle has joined #openstack-infra | 14:47 | |
| *** ihrachyshka has quit IRC | 14:48 | |
| bo_sh | hoo fungi, another question that i would be grateful if you answer on it. why not sending to logstash the logs in real-time? | 14:48 |
| bo_sh | * answer it | 14:49 |
| fungi | jeblair: i'm all for big tent discussion telling githubbers to manage their own forks if they want our code on there | 14:49 |
| *** tdasilva has joined #openstack-infra | 14:49 | |
| fungi | (or bitbucket or whatever) | 14:49 |
| *** lttrl has quit IRC | 14:50 | |
| *** marun_ has joined #openstack-infra | 14:50 | |
| *** armax has joined #openstack-infra | 14:50 | |
| tdasilva | clarkb, fungi: hello, was wondering if you could help me setup openstackstatus on #swiftonfile | 14:51 |
| *** david-lyle has quit IRC | 14:51 | |
| fungi | bo_sh: we need the logs in two places and were putting them on the browseable logs site already, so logstash integration was hooking into that. also we store a lot more log data than we want indexed into logstash, so we need somewhere to filter it down first (which admittedly could probably be done inline with the workers). further, job artifacts/logs are handled by jenkins plugins, so trying to | 14:51 |
| fungi | split them to logstash and a webserver from there would be rough | 14:51 |
| notmyname | tdasilva: Ii'm not sure if it's the status bot or not. but you just want the channel logs :-) | 14:52 |
| clarkb | bo_sh: fungi also authenticating slaves with logstash is hard and we dont want job results tied to indexing | 14:52 |
| *** marun has quit IRC | 14:52 | |
| *** changbl has joined #openstack-infra | 14:53 | |
| *** david-lyle has joined #openstack-infra | 14:53 | |
| fungi | tdasilva: yeah, if you just want the channel logged, that's http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp#n39 | 14:54 |
| tdasilva | fungi: thanks! | 14:54 |
| tdasilva | fungi: i'll check it out | 14:54 |
| *** bswartz has quit IRC | 14:54 | |
| clarkb | fungi tdasilva but we wont log without the openstackinfra bot too | 14:55 |
| *** bswartz has joined #openstack-infra | 14:56 | |
| clarkb | so two pieces. one does chanserv management the other logging | 14:56 |
| fungi | clarkb: it's already set up | 14:56 |
| clarkb | thats easy then | 14:56 |
| bo_sh | clarkb: fungi: thanks! | 14:56 |
| *** jlibosva has quit IRC | 14:56 | |
| fungi | http://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml#n143 has " - name: swiftonfile" | 14:57 |
| *** zz_jgrimm is now known as jgrimm | 14:57 | |
| *** thedodd has joined #openstack-infra | 14:57 | |
| fungi | i was helping them with chanops setup yesterday and my access levels had already been set by the accessbot | 14:58 |
| *** atiwari has joined #openstack-infra | 14:58 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Increase build-timeout for rally-cli job https://review.openstack.org/130525 | 14:59 |
| *** afazekas has quit IRC | 14:59 | |
| *** yaguang has quit IRC | 14:59 | |
| *** eglynn-office is now known as eglynn | 15:00 | |
| *** oramraz has quit IRC | 15:01 | |
| *** dustins_ is now known as dustins | 15:02 | |
| *** jlibosva has joined #openstack-infra | 15:04 | |
| *** pradk has joined #openstack-infra | 15:04 | |
| *** baoli has quit IRC | 15:04 | |
| *** baoli has joined #openstack-infra | 15:05 | |
| *** gokrokve has joined #openstack-infra | 15:05 | |
| *** yamahata has quit IRC | 15:06 | |
| *** yamahata has joined #openstack-infra | 15:06 | |
| *** mrmartin has quit IRC | 15:06 | |
| *** homeless has quit IRC | 15:06 | |
| *** dane_leblanc has quit IRC | 15:07 | |
| *** openstackgerrit has quit IRC | 15:07 | |
| *** dane_leblanc has joined #openstack-infra | 15:08 | |
| *** zeus has quit IRC | 15:08 | |
| tdasilva | fungi, clarkb: thanks for all the help and info... | 15:11 |
| *** ociuhandu has joined #openstack-infra | 15:11 | |
| fungi | tdasilva: you're welcome | 15:11 |
| mriedem | fungi: was there a planned date for when py26 jobs would be dropped? | 15:12 |
| fungi | mriedem: as soon as we can get the corresponding changes merged | 15:12 |
| fungi | mriedem: i think maybe clarkb was going to send a follow-up announcement indicating it happened | 15:12 |
| mriedem | fungi: ok, saw the thread on removing it after juno was released, wasn't sure if there were changes up yet | 15:13 |
| mriedem | thanks | 15:13 |
| clarkb | fungi: I need to send that email today since ajaegers changes are readuly | 15:14 |
| AJaeger | clarkb, you have not send the py26 email out yet... | 15:14 |
| clarkb | No I havent | 15:14 |
| AJaeger | clarkb: you beat me by one second ;) | 15:14 |
| clarkb | then give stackforge the weekend to keep jobs and merge mondayish? | 15:15 |
| AJaeger | mriedem: see https://review.openstack.org/#/c/129433 for the stack | 15:15 |
| fungi | i'm +2 on the updated state of https://review.openstack.org/129433 now | 15:15 |
| AJaeger | clarkb: I'm fine with whatever you propose. | 15:16 |
| fungi | i'm still unconvinced by 129434 | 15:16 |
| AJaeger | clarkb, fungi: I'll have a long weekend off (friday to monday) with little internet, so if changes need to be done to the patches, somebody else needs to update... | 15:16 |
| *** afazekas has joined #openstack-infra | 15:17 | |
| fungi | the assumption that all stackforge projects accidentally added python 2.6 testing and have to hit a dead-man switch to get them back doesn't sit right with me | 15:17 |
| AJaeger | fungi: many added py26 testing with adding the python-jobs template | 15:17 |
| clarkb | fungi ok. my conceen is if we hit them on the head in a year when we have no slaves that is worse | 15:17 |
| clarkb | if we do it now everyone knows and acknowledges this is happening and can prepare | 15:18 |
| fungi | for some reason i missed that suggestion when it was brought up in the infra meeting or i would have objected then | 15:18 |
| AJaeger | fungi: we can give them a longer "grace period" - and wait a week | 15:18 |
| clarkb | ya I dont mind waiting longer for that change | 15:19 |
| clarkb | but it should happen well in advance of juno eol | 15:19 |
| AJaeger | if I should reorder the stack somehow, please suggest it as well... | 15:19 |
| AJaeger | clarkb: Let's give a fixed date and with Paris summit starting in 10 days, I suggest we do it before. | 15:20 |
| AJaeger | So, that's next week Thursday. | 15:20 |
| fungi | if we do approve 129434 it should probably be the last one in sequence. the others in that series are not contentious | 15:20 |
| AJaeger | Or we do it well after it - in three weeks earliest | 15:20 |
| AJaeger | fungi: so, there's agreement on https://review.openstack.org/#/c/129435/ as well? | 15:21 |
| clarkb | fungi the problem we have is we will turn it off for everyone in ~12-18 months | 15:21 |
| AJaeger | I'm fine to put 129434 after the olso patch and can do so later today | 15:21 |
| fungi | well, and jeblair still needs to weigh in on 129435 since that means we stop testing zuul, nodepool, et cetera against 2.6 | 15:22 |
| AJaeger | fungi: yeah | 15:22 |
| clarkb | and we can either do it will fallback or really drop it on everyone | 15:22 |
| clarkb | imo doing it early with options is a far better solution | 15:23 |
| mtreinish | fungi, clarkb: can you add me to the qa-specs-core gerrit group which was newly created (I missed when that patch landed) | 15:23 |
| fungi | mtreinish: sure thing | 15:24 |
| *** arnaud has joined #openstack-infra | 15:24 | |
| AJaeger | time to prepare dinner, I'll be back later... | 15:24 |
| mtreinish | fungi: thanks | 15:24 |
| fungi | mtreinish: done | 15:24 |
| clarkb | fungi did you want to leave notes in the etherpad too? | 15:25 |
| fungi | clarkb: which etherpad was that? | 15:26 |
| clarkb | (I didnt realize this was objected to as it has been brought up in multiple places) | 15:26 |
| *** zaphod42 has joined #openstack-infra | 15:26 | |
| anteaya | I have a problem, according to these two patches there should be a file technical-committee-repos.yaml in the reference dir in the governance repo: https://review.openstack.org/#/c/129188/ https://review.openstack.org/#/c/125509/ | 15:26 |
| anteaya | I can't see it on git.o.o/openstack/governance | 15:26 |
| clarkb | I have to find it | 15:27 |
| anteaya | thanks | 15:27 |
| clarkb | fungi: https://etherpad.openstack.org/p/py26-deprecation there it is | 15:27 |
| clarkb | anteaya: oh sorry I was responding to fungi | 15:27 |
| anteaya | ah sorry you weren't replying to me | 15:27 |
| anteaya | s'okay | 15:27 |
| anteaya | I kinda jumped into the middle there | 15:27 |
| fungi | anteaya: i see http://git.openstack.org/cgit/openstack/governance/tree/reference/technical-committee-repos.yaml so maybe one of the mirrors is behind | 15:27 |
| *** pcrews has joined #openstack-infra | 15:27 | |
| fungi | checking now | 15:28 |
| anteaya | thanks | 15:28 |
| anteaya | I can see it by specifically feeding in the url | 15:28 |
| fungi | no hung processes in the gerrit queue | 15:28 |
| fungi | oh, well that's odd | 15:28 |
| *** mmaglana has joined #openstack-infra | 15:29 | |
| fungi | anteaya: you don't see it listed at http://git.openstack.org/cgit/openstack/governance/tree/reference/ then? | 15:29 |
| anteaya | I do now | 15:29 |
| anteaya | I didn't before | 15:29 |
| fungi | maybe your browser needed to refresh, but i'll double check that commit is on all the mirrors anyway just in case | 15:29 |
| clarkb | cgit does do a 5 minute cache too | 15:30 |
| clarkb | so git may update while you get rednered data that is up to 5 minutes old | 15:30 |
| *** hdd has quit IRC | 15:30 | |
| anteaya | but a brand new chrome instance can't find it: http://imgur.com/gTCx16J | 15:31 |
| anteaya | well the file was created on tuesday | 15:31 |
| *** tsg has joined #openstack-infra | 15:32 | |
| anteaya | I think my firefox knows about it now since I gave it the specific url for the file | 15:32 |
| anteaya | but chrome doesn't | 15:32 |
| clarkb | the cache may not be clearing properly too | 15:32 |
| anteaya | refreshed chrome still doesn't | 15:32 |
| clarkb | I want to say I saw that once before and killing the cache made it happy | 15:32 |
| fungi | all 5 git mirrors have that bdce59b ref in their /var/lib/git/openstack/governance.git so i don't think the issue is infra-side | 15:33 |
| anteaya | that happened before last week, was it not? | 15:33 |
| anteaya | interesting | 15:33 |
| anteaya | clarkb: that is what happened yes, I couldn't see the project-config dir | 15:33 |
| *** mmaglana has quit IRC | 15:33 | |
| fungi | just to be sure it's not cgit's server-side cache i tested http://git01.openstack.org:8080/cgit/openstack/governance/tree/reference/ and its four siblings. it lists that file on every one of them | 15:35 |
| clarkb | client side cache then? | 15:36 |
| anteaya | good | 15:36 |
| anteaya | meaning me? | 15:36 |
| clarkb | would be odd if chrom* decided to cahe a file longer for than what cgit says the ttl would be but I Havne't looked at what cgit says | 15:36 |
| clarkb | anteaya: ya | 15:36 |
| anteaya | how much cache would need to be cleared on my chrome that I just fired up and never use to navigate to git.o.o | 15:37 |
| anteaya | and I shut off my laptop last night | 15:37 |
| clarkb | well it persists to disk | 15:37 |
| *** salv-orlando has quit IRC | 15:37 | |
| clarkb | a simpler test may be to just wget the file | 15:37 |
| anteaya | will closing my chrome and opening it again clear the cache? | 15:37 |
| clarkb | and look at what you get via wget as it won't do anything smart | 15:37 |
| clarkb | anteaya: typically no | 15:37 |
| fungi | also modern browsers pre-load pages from urls linked to pages you visit so they'll be cached in case you decide to go to them | 15:38 |
| clarkb | if you run in incognito mode it will otherwise no | 15:38 |
| anteaya | what am I wgetting? | 15:38 |
| clarkb | anteaya: the url to the index page without the file listed | 15:38 |
| anteaya | the git.o.o/governance repo | 15:38 |
| anteaya | kk | 15:38 |
| clarkb | then you can open that file and see if it is listed | 15:38 |
| anteaya | kk | 15:38 |
| *** bdpayne has joined #openstack-infra | 15:38 | |
| fungi | another possibility is that your isp has some sort of caching proxy between you and the internet and that's misbehaving for some reason | 15:38 |
| fungi | for it to affect one browser and not the other would certainly be odd, though not impossible | 15:39 |
| anteaya | it isn't there | 15:41 |
| *** bdpayne_ has joined #openstack-infra | 15:42 | |
| clarkb | anteaya: cool so that at least means that chrome is likely not at fault | 15:42 |
| anteaya | http://imgur.com/3TCoLvM | 15:42 |
| clarkb | pin the tail on the ISP? | 15:42 |
| anteaya | I can call my isp | 15:42 |
| *** vhoward has left #openstack-infra | 15:42 | |
| clarkb | they likely won't do anything but maybe canadian isps are nicer | 15:43 |
| anteaya | my isp is awesome | 15:43 |
| *** lbiasi has joined #openstack-infra | 15:43 | |
| anteaya | I know about 6 of them on a first name basis | 15:43 |
| *** cpowell has quit IRC | 15:43 | |
| clarkb | ok I need to finish morning things | 15:44 |
| anteaya | kk | 15:44 |
| clarkb | when I get back I will finish jenkins.o.o then I guess debug precise image caching weirdness? | 15:44 |
| clarkb | fungi: did you want to do the github hot potato asap? | 15:44 |
| *** jp_at_hp1 has quit IRC | 15:45 | |
| *** bdpayne has quit IRC | 15:45 | |
| fungi | clarkb: nah, i think that's not really a solution we're going to want to support (transferring ownership of imported github repos) and dhellmann has proposed a pull request against the github repo in question to just replace it with a very small readme | 15:46 |
| *** dprince has quit IRC | 15:46 | |
| fungi | often the simplest solutions are the best solutions | 15:46 |
| *** rkukura has quit IRC | 15:47 | |
| clarkb | ++ | 15:47 |
| *** rkukura has joined #openstack-infra | 15:48 | |
| *** annegent_ has joined #openstack-infra | 15:48 | |
| *** eharney has quit IRC | 15:49 | |
| *** changbl has quit IRC | 15:50 | |
| *** sdake_ has quit IRC | 15:51 | |
| anteaya | isp can see the missing file | 15:52 |
| anteaya | he is going to try another computer | 15:52 |
| *** dustins has quit IRC | 15:52 | |
| *** mpaolino has quit IRC | 15:53 | |
| mordred | morning | 15:54 |
| *** mpaolino has joined #openstack-infra | 15:54 | |
| mordred | what's the github issue? (I couldn't figure out from the backscroll because ENOCOFFEE | 15:55 |
| *** ihrachyshka has joined #openstack-infra | 15:55 | |
| fungi | mordred: some user of pylockfile was urging us to get redirects in place on github to point from the original owner to our copy of the repo | 15:56 |
| anteaya | morning | 15:56 |
| jedimike | timezone appropriate greetings :) | 15:56 |
| anteaya | my isp can see it and doesn't know why I can't, so they have filed a ticket | 15:56 |
| anteaya | hey jedimike | 15:57 |
| *** changbl has joined #openstack-infra | 15:57 | |
| *** salv-orlando has joined #openstack-infra | 15:57 | |
| DinaBelova | SergeyLukjanov, it looks like openstack bot is dead in the #openstack-meeting channel | 15:58 |
| *** krtaylor has quit IRC | 15:58 | |
| jd__ | who fired the bot? | 15:58 |
| *** arnaud__ has joined #openstack-infra | 15:58 | |
| DinaBelova | infra team, help! | 15:58 |
| *** gyee has joined #openstack-infra | 15:58 | |
| DinaBelova | unlogged meeting is the hell.. | 15:58 |
| DinaBelova | :D | 15:58 |
| fungi | again?!? | 15:59 |
| * fungi shakes his fist at freenode | 15:59 | |
| *** dustins has joined #openstack-infra | 15:59 | |
| *** arxcruz has quit IRC | 15:59 | |
| DinaBelova | fungi, please make it alive :D | 15:59 |
| fungi | ping timeout at 15:07:36 utc | 15:59 |
| mordred | fungi: ah | 15:59 |
| DinaBelova | (facepalm) | 15:59 |
| DinaBelova | fungi, 7 minutes of meeting is nice, heh, | 16:00 |
| mordred | fungi: does github _do_ redirects? | 16:00 |
| mordred | oh! they do. how interesting | 16:00 |
| SpamapS | jedimike: I tend to use "greetings earthlings" if the timezone is mixed. :) | 16:00 |
| jedimike | haha :) | 16:00 |
| *** openstack has joined #openstack-infra | 16:02 | |
| fungi | DinaBelova: nevermind, it was the gerritbot i was noticing had exited (openstackgerrit) not the meetbot (openstack) | 16:02 |
| clarkb | except that you have to be admin of both orgs | 16:02 |
| clarkb | and thats where github breaks | 16:03 |
| mordred | oh. wow | 16:03 |
| mordred | clarkb: both orgs or would being admin of the repo to be moved be good enough? | 16:03 |
| * mordred is now mostly just curious | 16:03 | |
| clarkb | both orgs | 16:04 |
| clarkb | my reading of bug is we would need to be admin of that persons personal guthub or they would have to be admin of openstack org | 16:05 |
| clarkb | so that an account overlapped both | 16:05 |
| *** changbl has quit IRC | 16:05 | |
| *** todd_dsm has joined #openstack-infra | 16:06 | |
| mordred | that's fantastic | 16:06 |
| mordred | well, that's clearly not going to happen | 16:06 |
| mordred | also - it doesn't look like there is an api call for it anyway | 16:06 |
| jpich | II could use some help to debug this: http://logs.openstack.org/74/102274/5/experimental/gate-horizon-dsvm-integration/6c415f5/console.html . A package with that name *should* be available for Trusty: http://packages.ubuntu.com/trusty/xvfb , also for some reason the other devstack logs don't seem available either | 16:07 |
| *** eglynn is now known as eglynn-office | 16:08 | |
| *** mmaglana has joined #openstack-infra | 16:09 | |
| *** mwagner_lap has quit IRC | 16:09 | |
| *** annegent_ has quit IRC | 16:10 | |
| fungi | jpich: interesting. it does look like maybe the package index cached on that machine was incomplete/corrupt | 16:11 |
| *** krtaylor has joined #openstack-infra | 16:11 | |
| *** thedodd has quit IRC | 16:11 | |
| fungi | i'll sample another booted from the same image and see if there are issues with it | 16:11 |
| jpich | fungi: Thanks! Should I attempt another recheck, or better figure out a proper diagnostic first? | 16:12 |
| *** MaxV has quit IRC | 16:15 | |
| *** IvanBerezovskiy has left #openstack-infra | 16:16 | |
| fungi | jpich: it doesn't look to me like anything caused by your change... i was able to 'sudo apt-get install -y xvfb' from another machine of that type booted in the same provider/network | 16:16 |
| fungi | jpich: diagnostic in this case is "no idea at all" | 16:16 |
| jpich | fungi: :-) | 16:16 |
| fungi | can't seem to reproduce it | 16:16 |
| *** jpeeler has joined #openstack-infra | 16:16 | |
| clarkb | does it update before installing? | 16:17 |
| clarkb | if you dont then you can really confuse apt | 16:17 |
| fungi | that's a very good point | 16:17 |
| fungi | doesn't look like it did, so perhaps that explains it | 16:17 |
| fungi | though generally in those situations it would try to retrieve the package and then 404 | 16:18 |
| clarkb | ya | 16:18 |
| fungi | (if the rev in the package list did not match the rev on the apt repo) | 16:18 |
| *** ildikov_ has quit IRC | 16:19 | |
| fungi | since there's no link to the change in that log it's hard to tell | 16:19 |
| jpich | I'm going to cautiously attempt a recheck for now | 16:22 |
| *** yamahata has quit IRC | 16:22 | |
| *** luqas has quit IRC | 16:22 | |
| *** Longgeek has quit IRC | 16:23 | |
| fungi | jpich: what's the link to your change? | 16:24 |
| jpich | It seems to fail the same way already https://jenkins07.openstack.org/job/gate-horizon-dsvm-integration/1/console | 16:24 |
| jpich | fungi: https://review.openstack.org/#/c/102274/ | 16:24 |
| *** eharney has joined #openstack-infra | 16:25 | |
| fungi | it doesn't seem to be introducing xvfb. i'll have a look at what the functional test job tries to do | 16:25 |
| clarkb | I am going to manually build a devstack-precise image now so that I can see it cache things without noise from a bunch of otherbuilds (hoping that this semi interactiveness might help debug) | 16:25 |
| mtreinish | jogo: it happened again: http://logs.openstack.org/84/127384/3/check/check-tempest-dsvm-neutron-heat-slow-icehouse/a8eae34/logs/devstacklog.txt.gz#_2014-10-23_15_27_09_004 | 16:26 |
| *** xyang0 has quit IRC | 16:27 | |
| jpich | fungi: That's the latest patch that merged in infra related to this job, for reference https://review.openstack.org/#/c/128325/ | 16:27 |
| *** markmcclain has quit IRC | 16:28 | |
| anteaya | fungi: should the builder for that job have sudo apt-get update before installing the lib? | 16:29 |
| fungi | jpich: yep, as clarkb predicted, the xvfb-install job macro is not updating the package list first | 16:29 |
| fungi | the npm-install macro looks similarly broken | 16:30 |
| jpich | fungi: Fair enough. I'd modelled it on the npm macro above | 16:30 |
| fungi | yep | 16:30 |
| fungi | chef-bundler-prep looks like it does the right thing though | 16:30 |
| anteaya | yes I see it there | 16:30 |
| fungi | of the two in there using apt-get install, you got unlucky and copied from the broken one ;) | 16:31 |
| anteaya | but the job just above the xvfb job does'nt either | 16:31 |
| anteaya | jpich: mix up a patch and I'll +2 | 16:31 |
| anteaya | sorry, I didn't know you needed to update first | 16:31 |
| jpich | fungi: Ah well. I could have sworn the jshint horizon job uses the npm macro for running the tests though, and it works | 16:32 |
| *** dangers_away is now known as dangers | 16:32 | |
| jpich | anteaya: No worries! Getting that job definition to a good state has been and continues to be a learning experience :-) | 16:32 |
| *** zz_gondoi is now known as gondoi | 16:32 | |
| anteaya | for both of us | 16:32 |
| jpich | anteaya: And thanks, I'll prepare a patch to see if apt-get update helps here | 16:32 |
| anteaya | great | 16:32 |
| fungi | jpich: yep... it could be something different about the xvfb package vs the nodejs and npm packages in the package repository | 16:33 |
| *** MaxV has joined #openstack-infra | 16:33 | |
| *** jgallard has quit IRC | 16:34 | |
| jpich | Fair enough! I won't fix what's not broken for now then, only the horizon job | 16:34 |
| *** cdent has quit IRC | 16:34 | |
| mordred | jpich, clarkb: even if you do update first, you can still get broken links because apt repo updates are not atomic | 16:34 |
| fungi | jpich: ahh, yep. those are time bombs. xvfb has a version in trusty and a newer one in trusty-updates, while for the moment npm and nodejs only have versions in trusty | 16:34 |
| mordred | at least, until we roll out vos release based apt mirros ... | 16:34 |
| * mordred shuts up now | 16:35 | |
| clarkb | mordred: ya but not updating when you can have images 3 weeks old is really a bad situation | 16:35 |
| mordred | yes. 100% | 16:35 |
| *** mpaolino has quit IRC | 16:35 | |
| fungi | when a new version of npm and/or nodejs find their way into trusty-updates or trusty-security then the jobs using the install-npm macro will likely break similarly | 16:36 |
| clarkb | waiting for your key to be put on an image build server by puppet is really boring | 16:36 |
| mordred | yah | 16:37 |
| *** xyang0 has joined #openstack-infra | 16:37 | |
| clarkb | I am putting jenkins.o.o in shutdown mode now | 16:37 |
| *** dimtruck is now known as zz_dimtruck | 16:37 | |
| clarkb | mordred: you see this UNC stuff? | 16:37 |
| clarkb | and its idle so I am upgrading the plugin now | 16:38 |
| *** MaxV has quit IRC | 16:38 | |
| fungi | clarkb: academic fraud? in north carolina? no way... | 16:38 |
| fungi | </sarcasm> | 16:38 |
| *** _nadya_ has quit IRC | 16:38 | |
| *** MaxV has joined #openstack-infra | 16:39 | |
| *** pelix has quit IRC | 16:39 | |
| mordred | clarkb: I did not! should I go read espn? | 16:39 |
| clarkb | mordred: it is apparently really bad | 16:39 |
| clarkb | and jenkins.o.o is coming back now | 16:39 |
| fungi | mordred: it's just unc, so nothing to care about unless you just want to gloat | 16:39 |
| clarkb | fungi: that is what I figured he might want to do | 16:40 |
| clarkb | there is talk of vacating championships | 16:40 |
| * fungi attended unc-asheville instead, where the sports teams were so bad (except womens' soccer) that there was no question they all took classes like everyone else | 16:40 | |
| *** Guest53198 has quit IRC | 16:41 | |
| mordred | clarkb: I mean, I'm a die-hard duke fan ... so if UNC has been cheating, it makes my world good | 16:41 |
| jogo | mtreinish: clarkb fungi and I dug into that bug yesterday but came up short | 16:41 |
| *** pblaho has quit IRC | 16:41 | |
| clarkb | jogo: mtreinish I have a shell on an image build host now trying to see if anything looks off | 16:42 |
| clarkb | so far it doesn't | 16:42 |
| mordred | clarkb: so, basically, that guy from a few months ago who came out and said he was taking fake classes was not lying? | 16:42 |
| clarkb | mordred: apparently not. there were entire shadow curriculms | 16:42 |
| *** bcourt has joined #openstack-infra | 16:42 | |
| *** mancdaz is now known as mancdaz_away | 16:43 | |
| *** MaxV has quit IRC | 16:43 | |
| clarkb | at this point all our jenkins gearman plugins should be happy | 16:44 |
| mordred | WOW | 16:44 |
| clarkb | zaro: ^ | 16:44 |
| *** yfried|afk has joined #openstack-infra | 16:44 | |
| *** derekh has quit IRC | 16:44 | |
| *** doug-fish has quit IRC | 16:45 | |
| *** pmoosh has joined #openstack-infra | 16:45 | |
| *** dolphm has left #openstack-infra | 16:46 | |
| *** SumitNaiksatam has quit IRC | 16:47 | |
| clarkb | mordred: fungi: I do see no ~jenkins/cache/files dir | 16:47 |
| *** dane_leblanc has quit IRC | 16:47 | |
| jpich | anteaya, fungi: Here we go -> https://review.openstack.org/#/c/130594/ | 16:47 |
| clarkb | does wget mkdir for us? | 16:47 |
| fungi | i guess openstackgerrit abandoned us | 16:47 |
| fungi | clarkb: not afaik, no | 16:47 |
| clarkb | hrm I wonder if this is related | 16:47 |
| clarkb | maybe puppet failed to puppet something like that for some reason | 16:48 |
| fungi | pretty sure wget assumes the destination directory already exists | 16:48 |
| clarkb | ya that is what I thought too | 16:48 |
| *** AJaeger has quit IRC | 16:48 | |
| clarkb | I will keep an eye on this to see what happens | 16:48 |
| *** baoli has quit IRC | 16:49 | |
| *** rmcall has joined #openstack-infra | 16:49 | |
| *** AJaeger has joined #openstack-infra | 16:49 | |
| *** openstackgerrit has joined #openstack-infra | 16:50 | |
| *** mjturek has quit IRC | 16:50 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from all stackforge projects https://review.openstack.org/129434 | 16:51 |
| fungi | okay, i brought the openstackgerrit gerritbot back online | 16:51 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Remove python26 jobs from various projects https://review.openstack.org/129435 | 16:52 |
| clarkb | oh I see install_devstack_dependencies mkdirs | 16:52 |
| *** baoli has joined #openstack-infra | 16:52 | |
| *** rmcall has quit IRC | 16:52 | |
| AJaeger | I'm rebasing all the python26 patches now - they are independ of each other, no need to stack them... | 16:52 |
| clarkb | AJaeger: oh good point | 16:52 |
| clarkb | ++ to that | 16:52 |
| anteaya | jpich: +A, let's see what this does | 16:53 |
| *** afazekas is now known as _afazekas | 16:53 | |
| *** rfolco has quit IRC | 16:53 | |
| AJaeger | when I created them first, they were dependent on krotscheck's large change but since that one is merged, I can unstack | 16:53 |
| jpich | anteaya: Cheers! | 16:53 |
| *** doude_ has quit IRC | 16:53 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Adjust python 2.6 jobs for oslo libraries https://review.openstack.org/130444 | 16:53 |
| fungi | AJaeger: sounds great | 16:54 |
| *** dizquierdo has quit IRC | 16:54 | |
| AJaeger | I also gave them all a topic "python26" - clarkb something you might want to suggest in your email | 16:55 |
| anteaya | fungi: you are spending today babysitting bots | 16:55 |
| *** stevemar has joined #openstack-infra | 16:55 | |
| AJaeger | clarkb: Add https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:python26,n,z as reference and ask to submit patches with that topic for any changes | 16:55 |
| *** garyh has quit IRC | 16:55 | |
| anteaya | among the normal fixing all the things | 16:55 |
| *** rmcall has joined #openstack-infra | 16:56 | |
| *** garyh has joined #openstack-infra | 16:56 | |
| mtreinish | clarkb: hmm, I've seen it on a couple of runs this morning | 16:56 |
| AJaeger | dhellmann: could you review https://review.openstack.org/130444 , please? | 16:57 |
| *** jistr has quit IRC | 16:57 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 16:57 | |
| clarkb | fungi: is a reasonable compromise on the python26 stackforge thing to sit on that particular change for a while? or do you think it is also problematic that we are asking for changes to explicitly say "we want python26" | 16:57 |
| *** tonytan4ever has quit IRC | 16:57 | |
| clarkb | mtreinish: this is like the slowest image download ever | 16:57 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Xvfb install: Update repo before installing the package https://review.openstack.org/130594 | 16:57 |
| *** amcrn has joined #openstack-infra | 16:57 | |
| anteaya | taking a poll on the name puppet-openstackci_apache | 16:57 |
| clarkb | its not wonder the jobs fail when its not cached | 16:57 |
| *** changbl has joined #openstack-infra | 16:57 | |
| *** afazekas has joined #openstack-infra | 16:57 | |
| anteaya | I'm not a fan of hyphens and underscores in a repo name | 16:57 |
| anteaya | am I alone? | 16:57 |
| clarkb | no | 16:58 |
| mtreinish | clarkb: heh, yeah I know it's hysterically slow. | 16:58 |
| AJaeger | anteaya: yeah, either or but not both | 16:58 |
| jogo | clarkb: the image cache logic is very fast and loose with error codes etc | 16:58 |
| *** jpich has quit IRC | 16:58 | |
| anteaya | okay thanks | 16:58 |
| clarkb | jogo: yes I know | 16:58 |
| clarkb | jogo: I think the idea is its a cache | 16:58 |
| *** alexpilotti has joined #openstack-infra | 16:58 | |
| clarkb | if you fail to cache something ohwell | 16:58 |
| anteaya | ianw: can we get either puppet-openstackci-apache or puppet_openstackci_apache, please? | 16:58 |
| jogo | clarkb: sure, we can be fast and loose on some things but not everything in that script | 16:59 |
| clarkb | jogo: that is apaprently the wrong attitude | 16:59 |
| *** baoli has quit IRC | 16:59 | |
| clarkb | jogo: no really everything in that script should be able to be fast and loose | 16:59 |
| *** amcrn has quit IRC | 16:59 | |
| anteaya | I'm leaning towards hyphens personally but can live with the underscores if I have to | 16:59 |
| clarkb | reality is computers are bad | 16:59 |
| clarkb | and we need to be more careful | 16:59 |
| fungi | clarkb: given that there was some tacit approval of the plan in the meeting and i was somehow so engrossed in working out the oslo details that i missed the stackforge comments, i'm fine with it. but a slightly longer warning before we remove those might go a long way to avoiding a clash on the ml (at least we'll have more ammunition to show we were being considerate) | 16:59 |
| clarkb | ya I have no problem with leaving the stackforge removal up for a while | 17:00 |
| clarkb | maybe give them a couple weeks post summit just so everyone has a chance to look | 17:00 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/project-config: Pass stress arguments from job yaml to allow customization https://review.openstack.org/130599 | 17:00 |
| jogo | clarkb: at the very least we should log when something goes wrong | 17:01 |
| AJaeger | clarkb, so, let's say 30th of November? | 17:01 |
| clarkb | AJaeger: ya that works | 17:01 |
| fungi | wfm | 17:01 |
| AJaeger | That's a Sunday... but still good. | 17:02 |
| clarkb | jogo: keep in mind we don't actually try downloading that image at all | 17:02 |
| clarkb | jogo: so the error code stuff is likely not related in any way | 17:02 |
| AJaeger | We can still approve on Monday, 1st of December but 30th of November is easier to remember... | 17:02 |
| *** david-lyle has quit IRC | 17:03 | |
| *** david-lyle has joined #openstack-infra | 17:03 | |
| *** BobBall is now known as BobBall_awol | 17:03 | |
| clarkb | the download speed for this first fedora image is so slow I am seriously considering pushing a patch to devstack that rips it and the other one out | 17:05 |
| *** baoli has joined #openstack-infra | 17:05 | |
| *** rmcall_ has joined #openstack-infra | 17:07 | |
| rainya | good regional time of day, infra | 17:07 |
| jogo | clarkb: good point, I do have a set of patches up to make icehouse use the master version of the fedora image | 17:08 |
| jogo | which would side step this | 17:08 |
| *** doug-fish has joined #openstack-infra | 17:08 | |
| *** rmcall has quit IRC | 17:08 | |
| *** rmcall_ is now known as rmcall | 17:08 | |
| clarkb | jogo: no the master image is super slow too | 17:08 |
| clarkb | or do you mean the other thing I said about errors? | 17:08 |
| clarkb | in any case I think we need to find better hosted images | 17:08 |
| *** jcoufal has quit IRC | 17:08 | |
| *** mjturek has joined #openstack-infra | 17:09 | |
| *** rfolco has joined #openstack-infra | 17:09 | |
| *** jlibosva1 has joined #openstack-infra | 17:09 | |
| jogo | clarkb: the master image may be slow but its cached right? | 17:09 |
| *** mwagner_lap has joined #openstack-infra | 17:09 | |
| clarkb | jogo: its cached for the jobs but not for you and me running devstack locally | 17:09 |
| clarkb | and its not cached for my image build | 17:09 |
| clarkb | we are up to 61MB after about 15 minutes \o/ | 17:10 |
| *** cnesa has joined #openstack-infra | 17:10 | |
| *** dane_leblanc has joined #openstack-infra | 17:10 | |
| clarkb | (17really but who is counting) | 17:10 |
| *** HeOS has quit IRC | 17:10 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Adjust python 2.6 jobs for oslo libraries https://review.openstack.org/130444 | 17:10 |
| *** thedodd has joined #openstack-infra | 17:11 | |
| jogo | clarkb: true enough | 17:11 |
| *** tonytan4ever has joined #openstack-infra | 17:11 | |
| AJaeger | clarkb: sneakernet would be faster ;) | 17:11 |
| jogo | do we know anyone who runs that site? | 17:11 |
| jogo | it is redhat after all | 17:12 |
| *** jlibosva has quit IRC | 17:12 | |
| *** davidhadas has joined #openstack-infra | 17:12 | |
| jogo | eglynn-office: why is dl.fedoraproject.org so slow | 17:12 |
| *** SumitNaiksatam has joined #openstack-infra | 17:14 | |
| *** patrickeast has joined #openstack-infra | 17:14 | |
| *** jlibosva1 has quit IRC | 17:15 | |
| *** reed has quit IRC | 17:15 | |
| *** packet has joined #openstack-infra | 17:16 | |
| *** tsg has quit IRC | 17:18 | |
| *** zul has quit IRC | 17:18 | |
| jogo | dansmith: ^ maybe you know who we can ask ^ | 17:18 |
| jogo | russellb: ^ | 17:19 |
| dansmith | heh, I dunno, no | 17:19 |
| *** Ryan_Lane has joined #openstack-infra | 17:20 | |
| openstackgerrit | David Lyle proposed a change to openstack-infra/project-config: Adding Horizon repos https://review.openstack.org/130603 | 17:20 |
| *** dprince has joined #openstack-infra | 17:21 | |
| anteaya | well it was me then, my browser cache apparently is to blame | 17:21 |
| anteaya | not sure how it confounded wget though | 17:21 |
| russellb | i dunno either ... | 17:21 |
| anteaya | my isp doesn't cache | 17:21 |
| russellb | fedora is on irc ... not sure what channel is best | 17:21 |
| russellb | err, freenode i mean | 17:21 |
| russellb | #fedora-admin perhaps | 17:22 |
| russellb | https://fedoraproject.org/wiki/Infrastructure#IRC | 17:22 |
| russellb | jogo: ^ | 17:22 |
| *** markmcclain has joined #openstack-infra | 17:23 | |
| mtreinish | jogo: I'm inclined to just stop running that job, I think it only has 1 test run as part of it... (although it's been a while since I've seen passing results) | 17:23 |
| jogo | russellb: oh thanks | 17:23 |
| fungi | yeah, seems they have a wiki page of irc channels... quite a popular pattern! https://fedoraproject.org/wiki/Communicating_and_getting_help?rd=Communicate#IRC | 17:23 |
| russellb | np | 17:23 |
| jogo | mtreinish: that is one way to fix that | 17:23 |
| anteaya | fungi: should I wait for check jobs to return before approving 130603 david-lyle's patch? | 17:24 |
| jogo | mtreinish: strange when I run wget -c https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Images/i386/Fedora-i386-20-20131211.1-sda.qcow2 locally I get great speeds | 17:25 |
| jogo | clarkb: ^ | 17:25 |
| jogo | what do you get | 17:25 |
| jogo | 3.4 MB/s | 17:25 |
| mtreinish | jogo: 9.55 MB/s | 17:26 |
| jogo | damn | 17:26 |
| anteaya | and jenkins is happy | 17:27 |
| jogo | I get 70K on my RAX VM | 17:27 |
| *** rmcall has quit IRC | 17:27 | |
| *** bcourt has quit IRC | 17:27 | |
| *** dprince has quit IRC | 17:28 | |
| *** dprince has joined #openstack-infra | 17:28 | |
| *** tsg has joined #openstack-infra | 17:28 | |
| *** rmcall has joined #openstack-infra | 17:29 | |
| *** cnesa has quit IRC | 17:29 | |
| jogo | clarkb: ^ maybe they are rate limiting downloads to RAX | 17:29 |
| *** stevemar has quit IRC | 17:29 | |
| clarkb | possible | 17:29 |
| clarkb | maybe because the test slaves dosed them | 17:29 |
| *** koolhead17 has joined #openstack-infra | 17:30 | |
| clarkb | the build I started is still downloading that first image | 17:30 |
| *** amcrn has joined #openstack-infra | 17:30 | |
| clarkb | I really vote we kill these images | 17:30 |
| jogo | clarkb: I think they are used by ceilometer or something | 17:31 |
| jogo | but I am with you on this | 17:31 |
| clarkb | its heat | 17:31 |
| clarkb | but we are basically incapable ofbuilding images with them right now | 17:31 |
| fungi | my home broadband seems to be able to sustain no better than 30-40KB/s pulling that | 17:31 |
| clarkb | regardless of any other issues | 17:31 |
| fungi | jogo: used by heat | 17:32 |
| *** thedodd has quit IRC | 17:32 | |
| *** koolhead17 has quit IRC | 17:32 | |
| fungi | jogo: apparently their tests need something more full-fledged than a cirros vm | 17:32 |
| jogo | fungi: you should get mtreinish's internet | 17:32 |
| *** koolhead17 has joined #openstack-infra | 17:32 | |
| jogo | fungi: ahh right | 17:32 |
| jogo | got a bite on fedora-admin | 17:32 |
| fungi | they run an agent on it or some such | 17:32 |
| *** koolhead17 has quit IRC | 17:32 | |
| *** koolhead17 has joined #openstack-infra | 17:32 | |
| mtreinish | fungi: yeah it's the heat agent | 17:33 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Adding Horizon repos https://review.openstack.org/130603 | 17:33 |
| *** dkranz has quit IRC | 17:33 | |
| mtreinish | jogo: it was bouncing up against the max down that I pay for to be fair | 17:33 |
| fungi | at home: 3% downloaded, 87min eta | 17:33 |
| *** zul has joined #openstack-infra | 17:34 | |
| fungi | in rax-iad 4% downloaded, 61min eta | 17:34 |
| jogo | fungi: want to join #fedora-admin | 17:34 |
| *** dkranz has joined #openstack-infra | 17:34 | |
| fungi | yeah, this is _not_ a developer friendly choice. an hour to download an image used by a handful of tests which take at most minutes to run | 17:34 |
| jogo | taking to nirik | 17:34 |
| *** mrmartin has joined #openstack-infra | 17:35 | |
| fungi | nah, i'm more thinking in terms of our choice to add this as a requirement to devstack/tempest | 17:35 |
| fungi | if fedoraproject.org can't get fast mirrors, that's nothing i'm going to pester them over | 17:35 |
| mtreinish | fungi: I actually think because of some issues it's only like 1 test on that job | 17:35 |
| anteaya | in case anyone cares this was the opening of Canadian Parlament today: http://www.cbc.ca/player/News/Canada/Ottawa/Ottawa%20Events/ID/2568028829/ | 17:35 |
| anteaya | Parliament | 17:36 |
| jogo | https://fedorahosted.org/fedora-infrastructure/ticket/4458 | 17:36 |
| *** cpowell has joined #openstack-infra | 17:38 | |
| *** cpowell has joined #openstack-infra | 17:38 | |
| *** Hal has joined #openstack-infra | 17:39 | |
| *** Hal is now known as Guest1921 | 17:39 | |
| *** teran has quit IRC | 17:40 | |
| *** harlowja_away is now known as harlowja | 17:40 | |
| kashyap | jogo, Jumping in randomly, previously these kind of speed issues were tracked down to the cloud provider | 17:40 |
| kashyap | e.g. -- https://lists.fedoraproject.org/pipermail/infrastructure/2014-June/014445.html | 17:41 |
| jogo | kashyap: I don't really care where teh problem is, as long as it gets fixed | 17:41 |
| *** amitgandhinz has joined #openstack-infra | 17:42 | |
| amitgandhinz | hi, can someone help me with stackalytics weirdness around launchpad id's used and company affiliation? | 17:43 |
| *** _nadya_ has joined #openstack-infra | 17:43 | |
| kashyap | jogo, Sure. Pinging the said cloud providers (HP/Rackspace/whoever) with this data might be useful too. But I see you're also talking to the Fedora admin folks to see if they can help. . . | 17:44 |
| *** annegent_ has joined #openstack-infra | 17:44 | |
| anteaya | amitgandhinz: there is a stackalytics channel | 17:44 |
| anteaya | I think it is #openstack-stackalytics | 17:44 |
| anteaya | you might try there | 17:44 |
| amitgandhinz | anteaya: thanks, i;ll check there =) | 17:44 |
| anteaya | np | 17:44 |
| jogo | kashyap: I don't want to be hunting down providers, ISPs etc. it should just work | 17:45 |
| jogo | if it doesn't we shouldn't use it | 17:45 |
| kashyap | Well, if they are the ones who can fix it, e.g. see here, an HP support person who could reproduce the intermittent network failures -- http://lists.openstack.org/pipermail/openstack-infra/2014-June/001299.html | 17:46 |
| *** ivar-lazzaro has joined #openstack-infra | 17:47 | |
| *** ivar-lazzaro has quit IRC | 17:47 | |
| jogo | clarkb: try using download.fedora* instead of dl.* | 17:47 |
| jogo | that gets the mirror system | 17:47 |
| fungi | in this case i can reproduce it in hpcloud east, three regions in rackspace north america, and from my cable broadband provider in the usa | 17:47 |
| jogo | worked way better for me | 17:47 |
| jogo | fungi: ^ | 17:47 |
| *** ivar-lazzaro has joined #openstack-infra | 17:48 | |
| fungi | jogo: agreed, orders of magnitude faster | 17:48 |
| *** openstackgerrit has quit IRC | 17:48 | |
| clarkb | so we should probably update devstack with that different hostname | 17:49 |
| jogo | clarkb: doing that now | 17:49 |
| *** signed8bit has quit IRC | 17:49 | |
| *** signed8bit has joined #openstack-infra | 17:49 | |
| jogo | looks like https://review.openstack.org/#/c/88481/1 moved us off of that domain | 17:49 |
| *** openstackgerrit has joined #openstack-infra | 17:49 | |
| *** baoli has quit IRC | 17:49 | |
| AJaeger | 34 K from SUSE ;( | 17:49 |
| AJaeger | and 18K at home ;( | 17:50 |
| jogo | AJaeger: with download.fedora or dl.fedora? | 17:50 |
| *** amitgandhinz has left #openstack-infra | 17:50 | |
| fungi | jogo: clarkb: 96s to download at home, several minutes (still underway) in rax-iad, but considerably faster than an hour or more | 17:50 |
| AJaeger | with dl.fedora - download is fast (up to 4 MB/s from SUSE) | 17:51 |
| *** sdake_ has joined #openstack-infra | 17:51 | |
| AJaeger | but going down to 100K and then up again.. | 17:51 |
| jogo | AJaeger: can you try download.fedora | 17:51 |
| fungi | sdad's commit message there doesn't mention why the hostname changed | 17:52 |
| AJaeger | jogo: I tried - "download is fast" | 17:52 |
| jogo | AJaeger: good | 17:52 |
| *** marun_ has quit IRC | 17:52 | |
| AJaeger | another 14s | 17:52 |
| AJaeger | since 10s ;) | 17:52 |
| fungi | dl.f.o seems to all be in one network, while download.f.o is aliased to wildcard.f.o which is a round-robin across mirrors in multiple locations | 17:53 |
| fungi | including some ipv6 | 17:53 |
| AJaeger | finished " 210,894,848 1.71MB/s in 2m 51s" | 17:53 |
| AJaeger | via download.f.o | 17:53 |
| *** dkranz has quit IRC | 17:53 | |
| *** reed has joined #openstack-infra | 17:54 | |
| fungi | and when i say in the same network, dl.f.o is in fact a round robin across 5 immediately adjacent ipv4 addresses | 17:54 |
| *** dkranz has joined #openstack-infra | 17:54 | |
| *** pradk has quit IRC | 17:56 | |
| jogo | mtreinish: what is the bug number for this again? | 17:56 |
| jogo | patch: https://review.openstack.org/130611 | 17:56 |
| mtreinish | jogo: bug 1383928 | 17:56 |
| uvirtbot | Launchpad bug 1383928 in openstack-ci "Heat fedora image download times out job" [Undecided,New] https://launchpad.net/bugs/1383928 | 17:56 |
| fungi | something in redhat.com. as previously mentioned via a cogentco.com peer in phoenix az regardless of where i trace from, so likely they are not simultaneously multi-homed there | 17:56 |
| *** zz_dimtruck is now known as dimtruck | 17:57 | |
| fungi | though i'm now bored with this exercise, so not bothering to analyze bgp announcements from various providers | 17:57 |
| clarkb | I need to run off to meet with hogepodge to show how to spin up CI things | 17:59 |
| *** _nadya_ has quit IRC | 17:59 | |
| clarkb | so I may be effectively afk for the middle of the day | 17:59 |
| hogepodge | o/ | 17:59 |
| fungi | i'll be here if anything breaks | 17:59 |
| clarkb | good news is grenade and jenkins are all done now | 17:59 |
| clarkb | which makes me happy because I wanted that done on monday :) | 18:00 |
| fungi | productive week | 18:00 |
| fungi | it's still monday for some definitions of monday | 18:00 |
| * mtreinish is considering poking clarkb again about subunit2sql now that grenade and jenkins are done | 18:02 | |
| AJaeger | mtreinish: next monday ;) | 18:02 |
| jogo | https://review.openstack.org/130612 https://review.openstack.org/130612 https://review.openstack.org/130611 for all devstack branches | 18:03 |
| jogo | mtreinish: ^ | 18:03 |
| mtreinish | jogo: I thought I remember seeing a change which switched it from download -> dl | 18:04 |
| jogo | mtreinish: yeah sean did that he didn't say why | 18:04 |
| jogo | I assume it was a mistake | 18:04 |
| mtreinish | typical, I wouldn't assume that. It could have been something like the round robin pointing us to down mirrors and he just didn't put it in the commit msg | 18:05 |
| jogo | mtreinish: https://review.openstack.org/#/c/88481/1 | 18:05 |
| jogo | mtreinish: asking fedora-admins what happens in that case | 18:06 |
| mtreinish | jogo: hmm, ok it looks like it was a unrelated break the world | 18:07 |
| mtreinish | not a network related one | 18:07 |
| mtreinish | I'll +2 all 3 | 18:07 |
| jogo | mtreinish: thanks | 18:07 |
| *** arnaud__ has quit IRC | 18:07 | |
| *** arnaud has quit IRC | 18:07 | |
| jogo | mtreinish: mtreinish https://fedorahosted.org/mirrormanager/ | 18:09 |
| *** markmcclain has quit IRC | 18:09 | |
| *** teran has joined #openstack-infra | 18:11 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/system-config: Update RH1 to use net-label instead of net-id https://review.openstack.org/119154 | 18:13 |
| *** bswartz has quit IRC | 18:14 | |
| *** thedodd has joined #openstack-infra | 18:16 | |
| ihrachyshka | hey, it seems we get pylint job running for neutron in stable branches. obviously, it doesn't work... | 18:16 |
| ihrachyshka | https://review.openstack.org/#/c/124026/ | 18:16 |
| ihrachyshka | since the tox env is new in kilo | 18:16 |
| fungi | ihrachyshka: in that case the job should be pinned in layout.yaml to not run on stable/icehouse or stable/juno | 18:17 |
| AJaeger | ihrachyshka: do you want to send a patch or shall I cook one quickly for you? | 18:17 |
| ihrachyshka | AJaeger: I'm about to go to sleep, so if you'll cook it till my morning tomorrow, that would be great ;) | 18:18 |
| ihrachyshka | AJaeger: thanks | 18:18 |
| AJaeger | ihrachyshka: good night ;) | 18:18 |
| AJaeger | ihrachyshka: will do | 18:18 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: Add documentation and example on the usage of yaml anchors and aliases. https://review.openstack.org/130614 | 18:20 |
| *** mfink has quit IRC | 18:22 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/project-config: Don't run pylint on old neutron branches https://review.openstack.org/130616 | 18:22 |
| *** mfink has joined #openstack-infra | 18:23 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: [WIP] Email & Subscription Plugin https://review.openstack.org/129091 | 18:24 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: Add documentation and example on the usage of yaml anchors and aliases. https://review.openstack.org/130614 | 18:24 |
| mmedvede | nibalizer / jesusaurus: I am ready to submit a patch for puppet module split I am working on. What a logistics of freezing the changes to the module while the split is in progress? | 18:28 |
| mmedvede | *what are logistics | 18:28 |
| *** mfink has quit IRC | 18:28 | |
| nibalizer | mmedvede: right now there aren't really any logistics in place | 18:29 |
| *** tsg has quit IRC | 18:29 | |
| nibalizer | i think we're shy to add process if we don't think we really need it | 18:29 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Added description for node attribute in documentation. https://review.openstack.org/129287 | 18:30 |
| fungi | ugh... we really don't make the client lib sphinx docs discoverable at all | 18:30 |
| mmedvede | nibalizer, should I ask not to propose new changes to the module in a mailing list? | 18:31 |
| nibalizer | i would ask jeblair for direction on that | 18:31 |
| nibalizer | i've been doing low-turnover modules (not jenkins) so it hasn't really been a problem | 18:32 |
| *** gokrokve has quit IRC | 18:32 | |
| clarkb | no need just propose the split in gerrit and we can discuss logistics there | 18:32 |
| mmedvede | nibalizer, the first I am doing seems to be low traffic too, so I could just submit the patch to project-config | 18:32 |
| *** rodrigods has left #openstack-infra | 18:32 | |
| nibalizer | mmedvede: ya do it | 18:33 |
| mmedvede | clarkb, ok, thx | 18:33 |
| *** gokrokve has joined #openstack-infra | 18:33 | |
| anteaya | mmedvede: which module are you splitting? | 18:34 |
| mmedvede | anteaya, kibana first, elasticsearch second | 18:34 |
| *** arnaud has joined #openstack-infra | 18:34 | |
| *** arnaud__ has joined #openstack-infra | 18:35 | |
| *** ChuckC_ has quit IRC | 18:35 | |
| *** cpowell has quit IRC | 18:36 | |
| *** HeOS has joined #openstack-infra | 18:37 | |
| openstackgerrit | Mikhail S Medvedev proposed a change to openstack-infra/project-config: Adding puppet-kibana as split out module https://review.openstack.org/130619 | 18:38 |
| *** todd_dsm has quit IRC | 18:40 | |
| *** cpowell has joined #openstack-infra | 18:40 | |
| *** e0ne has quit IRC | 18:40 | |
| *** annegent_ has quit IRC | 18:43 | |
| *** alexpilotti has quit IRC | 18:43 | |
| anteaya | mmedvede: yeah I don't think kibana is undergoing a lot of patches proposed right now | 18:43 |
| anteaya | we might not need a formal freeze | 18:44 |
| *** alexpilotti has joined #openstack-infra | 18:44 | |
| *** _nadya_ has joined #openstack-infra | 18:45 | |
| mmedvede | anteaya, sounds good. So if something still gets merged while the split is in process, I should be able to update the new module upstream with the change, if I understand it. | 18:45 |
| *** alexpilotti has quit IRC | 18:45 | |
| anteaya | yeah that is bascially how it would work | 18:46 |
| anteaya | but let's get your patch up to project-config and go from there | 18:46 |
| *** alexpilotti has joined #openstack-infra | 18:46 | |
| anteaya | ah ha, you did 130619 | 18:47 |
| mmedvede | anteaya: yes :) | 18:47 |
| *** _nadya_ has quit IRC | 18:47 | |
| *** alexpilotti has quit IRC | 18:47 | |
| *** thedodd has quit IRC | 18:48 | |
| *** alexpilotti has joined #openstack-infra | 18:49 | |
| *** alexpilotti has quit IRC | 18:49 | |
| mmedvede | anteaya, I have based it off of https://review.openstack.org/#/c/126096/ | 18:49 |
| *** otherwiseguy has quit IRC | 18:50 | |
| *** cnesa has joined #openstack-infra | 18:51 | |
| *** alexpilotti has joined #openstack-infra | 18:51 | |
| anteaya | mmedvede: you did well | 18:51 |
| *** bswartz has joined #openstack-infra | 18:51 | |
| anteaya | so that patch looks good to me | 18:51 |
| anteaya | the next step is to offer a patch to system-config with the split out part removed | 18:51 |
| *** alexpilotti has quit IRC | 18:52 | |
| anteaya | and then co-ordinate with someone who can evaluate puppet and kibana (I nominate clarkb) to co-ordinate the merges happen in the right order | 18:52 |
| anteaya | and stand by to rebase the patches and your seed repo | 18:52 |
| mmedvede | anteaya: should I mark them as WIP then? | 18:52 |
| anteaya | I don't think so | 18:52 |
| anteaya | you need reviews | 18:53 |
| anteaya | I have added a comment to the project-config one | 18:53 |
| *** alexpilotti has joined #openstack-infra | 18:53 | |
| anteaya | which hopefully will prevent it getting approved prematurely | 18:53 |
| *** alazarev has left #openstack-infra | 18:53 | |
| anteaya | mention the url of the project-config patch in the commit message to system-config | 18:53 |
| *** alexpilotti has quit IRC | 18:54 | |
| mmedvede | anteaya: thank you. It is all very useful information, saving it for later :) | 18:54 |
| *** todd_dsm has joined #openstack-infra | 18:55 | |
| *** alexpilotti has joined #openstack-infra | 18:55 | |
| *** alexpilotti has quit IRC | 18:56 | |
| *** sarob has joined #openstack-infra | 18:56 | |
| *** alexpilotti has joined #openstack-infra | 18:56 | |
| *** cnesa has quit IRC | 18:57 | |
| *** alexpilotti has quit IRC | 18:57 | |
| *** alexpilotti has joined #openstack-infra | 18:57 | |
| *** alexpilotti has quit IRC | 18:58 | |
| *** jamespage_ has joined #openstack-infra | 18:59 | |
| *** alexpilotti has joined #openstack-infra | 19:00 | |
| *** alexpilotti has quit IRC | 19:00 | |
| *** alexpilotti has joined #openstack-infra | 19:02 | |
| *** sdake_ has quit IRC | 19:02 | |
| openstackgerrit | Ian Cordasco proposed a change to openstack-infra/devstack-gate: Fix broken links to the renamed openstack-infra/config project https://review.openstack.org/130285 | 19:02 |
| *** alexpilotti has quit IRC | 19:03 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/jenkins-job-builder: Add documentation and example on the usage of yaml anchors and aliases. https://review.openstack.org/130614 | 19:03 |
| *** alexpilotti has joined #openstack-infra | 19:04 | |
| *** bknudson has quit IRC | 19:05 | |
| *** alexpilotti has quit IRC | 19:05 | |
| *** alexpilotti has joined #openstack-infra | 19:06 | |
| *** ayoung is now known as ayoung-afk | 19:07 | |
| *** _nadya_ has joined #openstack-infra | 19:07 | |
| *** alexpilotti has quit IRC | 19:07 | |
| *** alexpilotti has joined #openstack-infra | 19:07 | |
| *** mfink has joined #openstack-infra | 19:08 | |
| *** alexpilotti has quit IRC | 19:08 | |
| *** Guest1921 has quit IRC | 19:09 | |
| *** mfink has quit IRC | 19:09 | |
| *** oramraz has joined #openstack-infra | 19:09 | |
| *** freemanbrandon has quit IRC | 19:09 | |
| *** Guest1921 has joined #openstack-infra | 19:09 | |
| *** mfink has joined #openstack-infra | 19:09 | |
| *** freemanbrandon has joined #openstack-infra | 19:09 | |
| *** mfink has quit IRC | 19:10 | |
| *** thedodd has joined #openstack-infra | 19:10 | |
| *** alexpilotti has joined #openstack-infra | 19:11 | |
| *** andreaf has quit IRC | 19:11 | |
| *** alexpilotti has quit IRC | 19:12 | |
| *** andreaf has joined #openstack-infra | 19:12 | |
| *** __TheDodd__ has joined #openstack-infra | 19:12 | |
| *** mfink has joined #openstack-infra | 19:12 | |
| *** bdpayne_ has quit IRC | 19:12 | |
| *** mfink has quit IRC | 19:13 | |
| *** alexpilotti has joined #openstack-infra | 19:13 | |
| *** david-lyle has quit IRC | 19:13 | |
| *** ihrachyshka has quit IRC | 19:14 | |
| *** signed8bit has quit IRC | 19:14 | |
| *** freemanbrandon has quit IRC | 19:14 | |
| *** alexpilotti has quit IRC | 19:14 | |
| *** freemanbrandon has joined #openstack-infra | 19:14 | |
| *** signed8bit has joined #openstack-infra | 19:15 | |
| *** jamespage_ has quit IRC | 19:15 | |
| *** alexpilotti has joined #openstack-infra | 19:15 | |
| *** mfink has joined #openstack-infra | 19:15 | |
| *** otherwiseguy has joined #openstack-infra | 19:16 | |
| *** thedodd has quit IRC | 19:16 | |
| *** alexpilotti has quit IRC | 19:16 | |
| *** alexpilotti has joined #openstack-infra | 19:18 | |
| *** alexpilotti has quit IRC | 19:19 | |
| *** freemanbrandon has quit IRC | 19:19 | |
| *** mrmartin has quit IRC | 19:22 | |
| *** freemanbrandon has joined #openstack-infra | 19:23 | |
| *** bknudson has joined #openstack-infra | 19:25 | |
| *** freemanbrandon has quit IRC | 19:27 | |
| *** ZZelle_ has joined #openstack-infra | 19:28 | |
| *** bknudson has quit IRC | 19:29 | |
| *** _nadya_ has quit IRC | 19:29 | |
| *** SumitNaiksatam has quit IRC | 19:29 | |
| *** ayoung-afk is now known as ayoung | 19:31 | |
| *** imcsk8 has quit IRC | 19:31 | |
| openstackgerrit | A change was merged to openstack-infra/system-config: Improve network support in launch-node.py https://review.openstack.org/129013 | 19:31 |
| *** imcsk8 has joined #openstack-infra | 19:31 | |
| *** markmcclain has joined #openstack-infra | 19:32 | |
| mwhudson | mtreinish: thanks for fixing up the message on my merge proposal | 19:34 |
| mwhudson | mtreinish: ... is it usually this hard to get things to pass in jenkins? | 19:34 |
| mwhudson | (https://review.openstack.org/#/c/127384/) | 19:34 |
| *** sdake_ has joined #openstack-infra | 19:35 | |
| *** dustins_ has joined #openstack-infra | 19:35 | |
| *** dustins has quit IRC | 19:35 | |
| *** sandywalsh_ has joined #openstack-infra | 19:37 | |
| fungi | mwhudson: bugs in openstack determine the number of false negative results you get in devstack/tempest jobs. the more of them you run, the more likely some will fail. tempest master has... 24 | 19:38 |
| fungi | most projects have 2-3 | 19:38 |
| *** ChuckC_ has joined #openstack-infra | 19:39 | |
| mwhudson | fungi: ah | 19:39 |
| *** sandywalsh has quit IRC | 19:39 | |
| fungi | so on any given day tempest is an order of magnitude harder to land changes in than most integrated openstack server projects | 19:39 |
| mwhudson | >>> 0.98**24 | 19:39 |
| mwhudson | 0.6157803365090783 | 19:39 |
| fungi | because you spin up two dozen clouds in very constrained environments (on other clouds!), beat on them mercilessly, and then hope none fall over | 19:40 |
| fungi | i'm amazed it works at all ;) | 19:40 |
| *** SumitNaiksatam has joined #openstack-infra | 19:40 | |
| mwhudson | heh | 19:40 |
| mwhudson | one of my failures seemed to be isci blowing up | 19:41 |
| fungi | at least you're not an operator running openstack at a large public cloud provider with iscsi blowing up on you... chances are there's a bug which caused your test failure and us simultaneously keeping some poor sysadmin up all night trying to put the broken pieces of his cloud back together again | 19:43 |
| mwhudson | heh | 19:44 |
| fungi | which is why we do try to take those sorts of failures seriously. they're a sign our software needs to improve | 19:44 |
| *** bknudson has joined #openstack-infra | 19:45 | |
| *** ChuckC_ has quit IRC | 19:45 | |
| *** ChuckC has joined #openstack-infra | 19:45 | |
| *** rmcall has quit IRC | 19:46 | |
| *** dimtruck is now known as zz_dimtruck | 19:49 | |
| fungi | so in good news... we're down to just four bugs blocking mass transition of python 3.3 tests to 3.4. all other outstanding unblocked changes have been merged | 19:50 |
| fungi | now it's just a race between ubuntu's utopic freeze and the glanceclient and heatclient devs working on fixing their mock apis | 19:51 |
| anteaya | go go utopic | 19:53 |
| *** freemanbrandon has joined #openstack-infra | 19:53 | |
| *** gokrokve has quit IRC | 19:54 | |
| fungi | yeah, pending trusty sru requests are waiting to be backports from utopic, which is approaching release | 19:54 |
| mtreinish | mwhudson: on that particular patch it's bounced off an image cache issue at least once | 19:55 |
| fungi | i guess utopic released today | 19:55 |
| fungi | so any time now | 19:55 |
| mwhudson | utopic released about an hour ago i think | 19:55 |
| fungi | indeed | 19:55 |
| mtreinish | mwhudson: the failures on the stable icehouse heat job are caused by an image download being slow | 19:56 |
| *** Nikerabbit has left #openstack-infra | 19:56 | |
| mtreinish | it never even gets to the test run | 19:56 |
| fungi | (very, very slow) | 19:56 |
| mwhudson | mtreinish: ugh | 19:56 |
| *** gokrokve has joined #openstack-infra | 19:56 | |
| *** gokrokve has quit IRC | 19:57 | |
| fungi | though your most recent failure on that patch was a grenade job ending on a test_snapshot_pattern failure | 19:57 |
| mwhudson | mtreinish: i'm glad i'm particularly unlucky i guess, it would be a bit discouraging if this happened to everyone's first patch | 19:57 |
| *** gokrokve has joined #openstack-infra | 19:57 | |
| mtreinish | give me a sec I'm going to psuh the change pull the job it arguably doesn't have any value | 19:57 |
| ianw | anteaya: re https://review.openstack.org/#/c/129496/ -- you don't want "-" in puppet module names, and everything else is already puppet-*, so i think it's about the only reasonable choice | 19:57 |
| *** _buzz has joined #openstack-infra | 19:57 | |
| fungi | mwhudson: oh, congrats on your first patch and welcome to our pain factory ;) | 19:58 |
| *** oramraz has quit IRC | 19:58 | |
| anteaya | ianw: expand on why I don't want hyphens in puppet module names | 19:58 |
| *** e0ne has joined #openstack-infra | 19:59 | |
| *** unicell1 has quit IRC | 20:01 | |
| *** freemanbrandon has quit IRC | 20:02 | |
| *** Ryan_Lane1 has joined #openstack-infra | 20:02 | |
| *** ociuhandu has quit IRC | 20:04 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/project-config: Remove icehouse heat slow job from tempest check https://review.openstack.org/130633 | 20:04 |
| mtreinish | fungi: ^^^ | 20:04 |
| * mtreinish thinks it's pretty indicative that a longer stable maint window won't work | 20:05 | |
| *** Ryan_Lane has quit IRC | 20:05 | |
| *** sdake_ has quit IRC | 20:06 | |
| AJaeger | mtreinish: the same job is run for tempest and tempest-lib. Shouldn't you remove it for both? | 20:06 |
| *** david-lyle has joined #openstack-infra | 20:06 | |
| fungi | AJaeger: makes an excellent point | 20:06 |
| anteaya | I just approved | 20:07 |
| AJaeger | anteaya: remove your votes again ;) | 20:07 |
| mtreinish | AJaeger: oops yeah | 20:07 |
| anteaya | okay | 20:07 |
| anteaya | removed | 20:07 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/project-config: Remove icehouse heat slow job from tempest check https://review.openstack.org/130633 | 20:08 |
| mtreinish | AJaeger: ^^^ | 20:08 |
| AJaeger | anteaya: another chance for you to approve - let me review quickly ;) | 20:08 |
| *** JayJ has joined #openstack-infra | 20:08 | |
| * AJaeger is happy ;) | 20:08 | |
| anteaya | review review | 20:08 |
| anteaya | mtreinish: the commit message doesn't say the job is removed from tempest-lib | 20:10 |
| * fungi considers tempest-lib part of tempest | 20:10 | |
| fungi | commit messages are better when they say why they do, less what they do | 20:10 |
| *** koolhead17 has quit IRC | 20:10 | |
| *** tristanC has quit IRC | 20:10 | |
| anteaya | okay | 20:10 |
| mtreinish | also, honestly in the past month I think we've seen maybe 5 tempest-lib patches which >50% were mine | 20:10 |
| anteaya | I will go ahead | 20:10 |
| mtreinish | oh, what fungi said | 20:11 |
| anteaya | kk | 20:11 |
| *** baoli has joined #openstack-infra | 20:11 | |
| anteaya | approved again | 20:12 |
| *** aysyd has quit IRC | 20:12 | |
| mtreinish | anteaya: thanks | 20:12 |
| openstackgerrit | Mikhail S Medvedev proposed a change to openstack-infra/system-config: Split out kibana module https://review.openstack.org/130634 | 20:13 |
| * AJaeger wishes everybody a great weekend, I'll be back Monday or Tuesday! | 20:13 | |
| anteaya | AJaeger: have a good weeekend | 20:13 |
| anteaya | AJaeger: I'll be around monday and gone the rest of the week | 20:14 |
| AJaeger | enjoy your vacation, anteaya ! | 20:14 |
| *** AJaeger has quit IRC | 20:14 | |
| *** bdpayne has joined #openstack-infra | 20:15 | |
| *** freemanbrandon has joined #openstack-infra | 20:15 | |
| anteaya | thanks | 20:15 |
| *** tristanC has joined #openstack-infra | 20:15 | |
| fungi | ianw: you're aware devstack-centos7 nodepool images aren't building successfully in hpcloud, right? | 20:17 |
| *** freemanbrandon has quit IRC | 20:17 | |
| *** freemanbrandon has joined #openstack-infra | 20:17 | |
| *** andreaf has quit IRC | 20:17 | |
| *** julim has quit IRC | 20:17 | |
| *** andreaf has joined #openstack-infra | 20:17 | |
| ianw | fungi: no | 20:18 |
| ianw | fungi: i'll look into it | 20:18 |
| fungi | looks like they built successfully in rax a couple days ago at least | 20:18 |
| fungi | but hpcloud has been spinning indefinitely trying to build any | 20:18 |
| ianw | fungi: i know i keep asking, but it would be great if we could get the size of the logs down | 20:18 |
| *** changbl has quit IRC | 20:19 | |
| ianw | anteaya: hyphens aren't allowed in class names | 20:19 |
| mmedvede | clarkb: Are you ok to coordinate/evaluate the kibana module split merges, as anteaya proposed? I would add you to these if so: https://review.openstack.org/#/q/topic:module-split,n,z | 20:20 |
| mtreinish | oh I just realized that I probably should have put related bug in that commit, oh well | 20:20 |
| *** gyee has quit IRC | 20:21 | |
| anteaya | ianw: puppet-openstackciapache | 20:21 |
| anteaya | why do we need openstackci as part of the name? | 20:21 |
| *** markmcclain has quit IRC | 20:21 | |
| *** markmcclain1 has joined #openstack-infra | 20:21 | |
| fungi | ianw: i have a feeling the runaway image update loops are what has actually exploded the image logs | 20:22 |
| ianw | anteaya: to not conflict with puppet-apache, the upstream version. openstackci is called out to avoid confusion that it might be the module you use to deploy apache for keystone, etc (did mention that in the comment, iirc) | 20:22 |
| ianw | (i.e. not calling it openstack_apache) | 20:23 |
| anteaya | I'm suggesting puppet-openstackciapache | 20:24 |
| anteaya | lose the underscore entirely | 20:24 |
| openstackgerrit | A change was merged to openstack-infra/project-config: Remove icehouse heat slow job from tempest check https://review.openstack.org/130633 | 20:24 |
| anteaya | mmedvede: that on job in the system-config patch is failing | 20:24 |
| anteaya | one | 20:24 |
| *** oramraz has joined #openstack-infra | 20:24 | |
| anteaya | mmedvede: and I'm not entirely sure why | 20:25 |
| *** markmcclain1 has quit IRC | 20:25 | |
| *** markmcclain has joined #openstack-infra | 20:25 | |
| mmedvede | anteaya: I thought it is normal, because it can not pull in the new module, it is not there yet? | 20:25 |
| mtreinish | anteaya: it probably needs the project-config patch to land and create the repo before it'll pass | 20:25 |
| anteaya | mmedvede: good point | 20:25 |
| anteaya | mtreinish: seems to be the cause | 20:25 |
| anteaya | mmedvede: can you make that comment on the patch? | 20:26 |
| mmedvede | anteaya: yes | 20:26 |
| anteaya | thanks | 20:26 |
| *** markmcclain1 has joined #openstack-infra | 20:26 | |
| *** markmcclain has quit IRC | 20:27 | |
| clarkb | zaro: can you tag new gearman plugin release? 0.1.1 maybe | 20:27 |
| ianw | anteaya: i dunno, i think that's harder to parse than with a _ ... i guess leave it up to the votes | 20:28 |
| *** markmcclain has joined #openstack-infra | 20:28 | |
| *** markmcclain has quit IRC | 20:29 | |
| *** markmcclain has joined #openstack-infra | 20:29 | |
| *** markmcclain1 has quit IRC | 20:29 | |
| ianw | fungi: yeah, this is true. it seems we do need a better notification method. maybe a dashboard type thing with red/green and the last time it was built | 20:29 |
| *** markmcclain has quit IRC | 20:29 | |
| *** wenlock has quit IRC | 20:29 | |
| *** markmcclain has joined #openstack-infra | 20:29 | |
| *** ryanpetrello has quit IRC | 20:29 | |
| *** markmcclain has quit IRC | 20:29 | |
| dougwig | hi infra. over in neutron, we have a feature branch (feature/lbaasv2), and jenkins now always fails because it needs a change from master. so far, i have attempted to 'git review' a vanilla merge commit (rejected), and attempted to 'git rebase master', then 'git commit --allow-empty', and then 'git review', also rejected. how do i update the feature | 20:29 |
| dougwig | branch from master, or is there something else that is should be doing? | 20:29 |
| *** ryanpetrello has joined #openstack-infra | 20:30 | |
| openstackgerrit | A change was merged to openstack-infra/system-config: Deploy vhost template for nodepool.o.o logs https://review.openstack.org/129096 | 20:30 |
| dougwig | fungi or jeblair or anyone else: ^^ :) | 20:30 |
| *** ChuckC has quit IRC | 20:30 | |
| *** markmcclain has joined #openstack-infra | 20:31 | |
| fungi | dougwig: the neutron release managers (possibly just mestery) have the ability to propose a merge of the master branch onto the feature branch, which can then be reviewed and approved as usual | 20:32 |
| fungi | dougwig: https://wiki.openstack.org/wiki/GerritJenkinsGit#Merge_Commits | 20:32 |
| mestery | fungi: Looking | 20:32 |
| dougwig | perfect, ty | 20:32 |
| mestery | fungi: Thanks, dougwig, reading now and will see what I can get you folks ASAP. | 20:33 |
| *** lbiasi has quit IRC | 20:33 | |
| fungi | dougwig: mestery: it looks like when the feature branch was created, acl permissions for neutron were never updated to accommodate it | 20:33 |
| mestery | fungi: So no one can do this now (not even me)? | 20:33 |
| fungi | digging into details now and will get a patch up to fix it in moments | 20:34 |
| fungi | i only just noticed myself | 20:34 |
| dougwig | mestery: thanks, that first example block looks very close to what i did; i'll wager it was rejected because i'm not you. | 20:34 |
| dougwig | fungi: many thanks | 20:34 |
| mestery | dougwig: :) | 20:34 |
| ashp | I just presented the most redundent set of slides on how CI testing of openstack works at my job ever | 20:37 |
| ashp | "well, here's a bunch of things you know about" | 20:37 |
| anteaya | ashp: did the managers and marketing folks applaude? | 20:37 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/project-config: Give neutron-milestone access to push merge refs https://review.openstack.org/130636 | 20:37 |
| ashp | it's tech people :D | 20:37 |
| fungi | dougwig: mestery: ^ | 20:37 |
| ashp | "here is a screenshot of jenkins, think of this as a padding slide" | 20:37 |
| *** tonytan4ever has quit IRC | 20:38 | |
| dougwig | slide 1: it integrates. slide 2: continuously. slide 3: goto slide 1. | 20:38 |
| mestery | fungi: Thanks! | 20:38 |
| fungi | dougwig: mestery: that's duplicating how keystone and swift did their access for merge refs | 20:38 |
| ashp | slide 4: sometimes things catch on fire | 20:38 |
| *** stevemar has joined #openstack-infra | 20:38 | |
| fungi | there's no cult like a cargo cul | 20:38 |
| fungi | t | 20:38 |
| anteaya | ha ha ha | 20:38 |
| dougwig | fungi: thanks! | 20:39 |
| *** otherwiseguy has quit IRC | 20:39 | |
| *** baoli has quit IRC | 20:39 | |
| fungi | ashp: slide presentations, released under a free license... http://docs.openstack.org/infra/publications/ | 20:39 |
| *** terryw has joined #openstack-infra | 20:39 | |
| ashp | next time i'll just use those shamelessly ;) | 20:39 |
| fungi | ashp: that is why we publish them! | 20:39 |
| fungi | also, so that it makes them easy to reuse when we need to present at conferences on short notice, and so it's easy to refer attendees back to | 20:40 |
| *** kgiusti has quit IRC | 20:40 | |
| fungi | ashp: particularly, when you want to present an overview of our ci... http://docs.openstack.org/infra/publications/overview/ is an excellent one to start from | 20:41 |
| ashp | sadly i was presenting our cut down version of upstream, (until I figure out how to leapfrog you, muahah) | 20:42 |
| *** hashar has joined #openstack-infra | 20:42 | |
| ashp | and then release it all as a giant ball of horror | 20:43 |
| dhellmann | fungi, mordred, lifeless : here's a fun one: https://bugs.launchpad.net/pbr/+bug/1384919 | 20:43 |
| fungi | ashp: forj.io beat you to it i think | 20:43 |
| uvirtbot | Launchpad bug 1384919 in pbr "installing pbr during doc build breaks markupsafe module under some conditions" [Undecided,New] | 20:43 |
| *** sdake_ has joined #openstack-infra | 20:43 | |
| *** tonytan4ever has joined #openstack-infra | 20:44 | |
| *** markmcclain has quit IRC | 20:44 | |
| *** markmcclain1 has joined #openstack-infra | 20:44 | |
| ashp | fungi: oh neat | 20:44 |
| *** david-lyle has quit IRC | 20:44 | |
| fungi | ashp: those are your coworkers | 20:44 |
| *** atiwari has quit IRC | 20:44 | |
| ashp | fungi: mostly I'm working on the "reinvent the wheel but in chef or ansible" because we don't use puppet piece, first | 20:44 |
| *** cpowell has quit IRC | 20:44 | |
| ashp | sadly first I'm working on the "play catchup from a few years old prod env" first | 20:45 |
| *** kgiusti has joined #openstack-infra | 20:45 | |
| *** cpowell has joined #openstack-infra | 20:45 | |
| *** pmoosh has quit IRC | 20:45 | |
| fungi | gozer, forj.io and your group... hp has a lot of clones of our upstream infrastructure which seem not to know about one another at all ;) | 20:46 |
| *** oramraz has quit IRC | 20:46 | |
| ashp | well, I'm not at hp so I'm not guilty on that front :) | 20:46 |
| fungi | hah! whoops. for some reason i had it in my head you were also working at hp. might have been your nick... sorry! | 20:47 |
| ashp | i joined a tiny openstack startup because my previous job had too many people, at 300 people | 20:47 |
| ashp | and then on day eight cisco swooped in and bought us | 20:48 |
| *** david-lyle has joined #openstack-infra | 20:48 | |
| fungi | i understand that. not a fan of working at big companies myself either | 20:48 |
| ashp | i was at puppetlabs before, writing some of the modules you rely on :D | 20:48 |
| ashp | like when the mysql fuss happened and everyone was made it was rewritten - i did that :/ | 20:48 |
| ashp | *was mad | 20:49 |
| *** zz_dimtruck is now known as dimtruck | 20:49 | |
| fungi | indeed, you're one of the puppetlabs refugees. thanks for your work! no doubt we benefitted from quite a lot of it | 20:49 |
| ashp | I hope so! otherwise I had a wasted year :) | 20:49 |
| *** bitblt has joined #openstack-infra | 20:50 | |
| fungi | grrr... this is hung https://jenkins03.openstack.org/job/gate-infra-puppet-apply-centos6/181/console | 20:50 |
| *** wenlock has joined #openstack-infra | 20:50 | |
| fungi | pretty sure it's going to fail. i'm temptex to abuse my access to the abort button so i can dislodge the change behind it | 20:51 |
| fungi | temptex. word of the day | 20:51 |
| *** andreykurilin_ has joined #openstack-infra | 20:52 | |
| *** sarob has quit IRC | 20:53 | |
| *** sarob has joined #openstack-infra | 20:53 | |
| *** gyee has joined #openstack-infra | 20:54 | |
| *** otter768 has joined #openstack-infra | 20:55 | |
| *** sarob has quit IRC | 20:58 | |
| anteaya | I do'nt want a definition | 20:58 |
| *** oramraz has joined #openstack-infra | 20:58 | |
| openstackgerrit | A change was merged to openstack-infra/project-config: Give neutron-milestone access to push merge refs https://review.openstack.org/130636 | 20:59 |
| *** ChuckC has joined #openstack-infra | 20:59 | |
| *** cpowell has quit IRC | 21:00 | |
| fungi | nor do i. it's one of those typos best left to its own devices | 21:02 |
| *** packet has quit IRC | 21:02 | |
| *** gyee has quit IRC | 21:06 | |
| *** sdake_ has quit IRC | 21:06 | |
| fungi | mestery: dougwig: the acl is updated now so neutron-milestone members should be able to propose a merge of master into that feature branch | 21:08 |
| fungi | e.g. these people https://review.openstack.org/#/admin/groups/55,members | 21:08 |
| mestery | fungi: Thank you! | 21:08 |
| fungi | of course! | 21:09 |
| jogo | got a fun python question: how do I check how much free space is in in disk partition that holds a specific directory | 21:10 |
| jogo | os.statvfs does the filesystem not partition | 21:10 |
| *** HenryG has quit IRC | 21:11 | |
| *** JayJ has quit IRC | 21:12 | |
| fungi | jogo: i don't think free space in a partition is something you can expect to query in general. the kernel can figure out in many cases the offset between the end of the filesystem and the end of the partition (for fixed-length filesystems) but that's not always going to be the case | 21:12 |
| *** JayJ has joined #openstack-infra | 21:13 | |
| fungi | jogo: do you have a more specific use case/scenario you're trying to explore? | 21:13 |
| fungi | depending on your layering a filesystem can span multiple partitions of a block device, or of multiple block devices | 21:14 |
| *** esker has quit IRC | 21:15 | |
| fungi | through use of block device management subsystems like lvm | 21:15 |
| fungi | some newer filesystems even take care of spanning/aggregation on their own | 21:16 |
| jogo | https://bugs.launchpad.net/nova/+bug/1384546 | 21:16 |
| uvirtbot | Launchpad bug 1384546 in nova "Disk filter is not worked correct when the instance directory are shared" [Undecided,New] | 21:16 |
| jogo | fungi: I want what df says basically | 21:16 |
| fungi | oh, df | 21:16 |
| fungi | df is showing you the filesystem, not the partition | 21:16 |
| jogo | fungi: hmm | 21:17 |
| jogo | well because os.statvfs('home/jogo') doesn't allign with what df says | 21:17 |
| jogo | but I have an encrypted root directory | 21:17 |
| fungi | there is not necessarily a direct mapping between filesystems and block device partitions (nor even a guarantee that you have a partition table on the underlying block device at all) | 21:17 |
| *** rlandy has quit IRC | 21:18 | |
| jogo | and the bug above may be saying it doesn't work with NFS either | 21:18 |
| fungi | gimme a sec and i'll see what os.statvfs() is doing | 21:18 |
| jogo | but not sure | 21:18 |
| *** mmedvede has quit IRC | 21:18 | |
| openstackgerrit | A change was merged to openstack-infra/system-config: Rotate out nodepool logs every 8-hours https://review.openstack.org/130064 | 21:19 |
| fungi | jogo: f_bsize is the block size in bytes looks like, f_blocks is the number of blocks the filesystem is in total, f_bavail is the number of available block out of that | 21:22 |
| fungi | jogo: it matches what df reports for me, as long as you multiply by f_bsize | 21:22 |
| jogo | fungi: do you have access to a nfs partition? | 21:23 |
| jogo | fungi: it doesn't align for my root partition | 21:23 |
| jogo | err home | 21:24 |
| jogo | but my home is encrypted | 21:24 |
| fungi | oh, hrm. no not at the moment. i finally tore down all my nfs-based infrastructure at home | 21:24 |
| *** e0ne has quit IRC | 21:25 | |
| jogo | fungi: thanks anyway. I updated the bug and am waiting for more info | 21:25 |
| *** kgiusti has left #openstack-infra | 21:26 | |
| *** dprince has quit IRC | 21:27 | |
| *** sarob has joined #openstack-infra | 21:27 | |
| lifeless | dhellmann: nice | 21:27 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/system-config: Login with 'centos' user for node launch https://review.openstack.org/130639 | 21:27 |
| *** SumitNaiksatam has quit IRC | 21:27 | |
| ianw | fungi / clarkb : ^ should hopefully fix centos7 hpcloud nodepool images | 21:28 |
| dhellmann | lifeless: adding pbr to test-requirements.txt lets me work around it for now, because I have no idea at all what's going on there or why cliff is special | 21:28 |
| fungi | ianw: lgtm--thanks! | 21:29 |
| lifeless | dhellmann: try fixing the stevedore requirements.txt to include pbr | 21:30 |
| dhellmann | lifeless: stevedore doesn't depend on pbr, though | 21:30 |
| lifeless | right | 21:30 |
| lifeless | but it uses it | 21:30 |
| *** harlowja has quit IRC | 21:31 | |
| dhellmann | no, it uses setuptools | 21:31 |
| lifeless | ./setup.py: setup_requires=['pbr'], | 21:31 |
| lifeless | ./setup.py: pbr=True) | 21:31 |
| lifeless | from the stevedore setup.py | 21:31 |
| dhellmann | yes, for installation, like everything else | 21:31 |
| dhellmann | do you mean stevedore, or cliff? | 21:31 |
| lifeless | stevedore | 21:31 |
| lifeless | of cliffs dependencies | 21:31 |
| lifeless | stevedore is the only use using pbr | 21:32 |
| dhellmann | you're going to have to walk me through the thought process | 21:32 |
| dhellmann | do things that use pbr in their setup.py also need it in their dependency list? | 21:33 |
| lifeless | so I have no idea whats messing around within markupsafe | 21:33 |
| lifeless | dhellmann: they do yes, otherwise when pip installing non-wheels pbr will be installed via easy_install which is evil | 21:33 |
| *** terryw has quit IRC | 21:33 | |
| dhellmann | sigh, ok, we should probably have pbr throw an error or something because I don't think I knew that was the case | 21:34 |
| lifeless | dhellmann: and I don't put *anything* past side effects of easy_install | 21:34 |
| lifeless | dhellmann: that said, you might try python -v to get import logging | 21:34 |
| lifeless | dhellmann: or replacing the markupsafe module with a smart object to track attribute assignments | 21:34 |
| lifeless | dhellmann: as a way to figureout whats up. I'm not aware of anything in pbr that should have the side effect you're reporting | 21:35 |
| *** mmedvede has joined #openstack-infra | 21:35 | |
| *** oramraz has quit IRC | 21:35 | |
| mestery | fungi: When following the instructions here (https://wiki.openstack.org/wiki/GerritJenkinsGit#Merge_Commits) to update our master branch, when I get to "git review" it looks like it's going to submit about 100 commits. | 21:36 |
| * mestery is worried because that seems like a lot. | 21:36 | |
| *** harlowja has joined #openstack-infra | 21:36 | |
| mestery | Will it submit each one indivirually, or as one large "merge from master into feature branch" commit? | 21:37 |
| mestery | dougwig: ^^^^ | 21:37 |
| *** wenlock has quit IRC | 21:37 | |
| anteaya | indivirtually | 21:37 |
| anteaya | the words just keep arriving today | 21:37 |
| *** markmcclain1 has quit IRC | 21:37 | |
| *** markmcclain has joined #openstack-infra | 21:37 | |
| mestery | anteaya: lol | 21:37 |
| dhellmann | lifeless: ok, I'll experiment more tomorrow and see what I can find; thanks for the suggestions! | 21:38 |
| dougwig | i was expecting to just see the merge commit in gerrit, with the rest of the chain as a ff-merge. but i couldn't experiment. | 21:38 |
| lifeless | dhellmann: added some related thoughts to the bug | 21:38 |
| mestery | dougwig: Me too | 21:38 |
| dhellmann | lifeless: thanks | 21:39 |
| *** todd_dsm has quit IRC | 21:39 | |
| *** mattfarina has quit IRC | 21:39 | |
| *** mriedem is now known as mriedem_away | 21:40 | |
| tchaypo | I know how to register prep-source-repos with pypi | 21:41 |
| tchaypo | but how do I do it such that whatever does the publishing has the rights to push it up? | 21:41 |
| lifeless | you add the infra service account bot account | 21:42 |
| ashp | did zuul's appearance change at zuul.openstack.org this week? | 21:42 |
| ashp | i swear it looked different on tuesday | 21:42 |
| *** markmcclain has quit IRC | 21:42 | |
| *** sdake_ has joined #openstack-infra | 21:42 | |
| lifeless | tchaypo: 'openstackci' | 21:42 |
| tchaypo | mayhap i hsould be reading http://ci.openstack.org/stackforge.html | 21:42 |
| *** jaypipes has quit IRC | 21:43 | |
| *** markmcclain has joined #openstack-infra | 21:44 | |
| tchaypo | nope, no mention of pypi. asking here was a good choice. | 21:44 |
| tchaypo | thanks lifeless | 21:44 |
| *** ildikov has joined #openstack-infra | 21:45 | |
| viglesias | jeblair: im able to get jenkins executors registered in gearman, i see gerrit triggers adding actions to zuul but i dont see gearman actually kicking off the jenkins job | 21:46 |
| viglesias | jeblair: fyi the telnet interface is quite helpful for debugging :) | 21:46 |
| *** dustins_ has quit IRC | 21:47 | |
| *** kingia has joined #openstack-infra | 21:47 | |
| *** julim has joined #openstack-infra | 21:47 | |
| *** kingia is now known as IaKing | 21:47 | |
| ashp | viglesias: I was having that problem yesterday, but it wasn't registering in jenkins, I had to enable/disable the checkbox in global jenkins settings for "use gearman" and then suddenly it remembered how to work | 21:50 |
| ashp | viglesias: it's probably not that for you but... it helped me... | 21:50 |
| *** markmcclain has quit IRC | 21:51 | |
| *** stevemar has quit IRC | 21:51 | |
| jhesketh | Morning | 21:52 |
| tchaypo | I keep misreading "gearman" as "german" and wondering why we want a multi-lingual jenkins | 21:53 |
| armax | jogo: ping | 21:53 |
| fungi | mestery: sorry, had stepped away... merge commits look weird and scary in gerrit because they show the merge conflicts being resolved... do you have a link to the review url? | 21:54 |
| openstackgerrit | James Polley proposed a change to openstack-infra/project-config: Add the prep-source-repos project https://review.openstack.org/129533 | 21:54 |
| mestery | fungi: I panice and backed out :) | 21:54 |
| *** sdake_ has quit IRC | 21:54 | |
| mestery | fungi: But it had a LOT of commits it was claiming it was going to submit. | 21:54 |
| fungi | mestery: also, it sounded like you wanted to update the feature branch from master, not the other way around | 21:54 |
| mestery | Correct: master into feature | 21:54 |
| jogo | armax: pong | 21:56 |
| mestery | fungi: Now I'm going to step out, I'll find you later before I proceed with this push, I just don't want it to create 100+ gerrit reviews as it appared like it wa going to do. | 21:56 |
| mestery | fungi: Back later. | 21:56 |
| armax | jogo: I spent some more time looking into https://bugs.launchpad.net/neutron/+bug/1381617 | 21:56 |
| uvirtbot | Launchpad bug 1381617 in neutron "Check for FIP status in scenario tests cause instability in neutron gate jobs" [Critical,In progress] | 21:56 |
| fungi | the idea being that the feature branch gets updated with a single merge commit bringing in all the commits which have accumulated so far on master since it diverged | 21:57 |
| *** carl_baldwin has joined #openstack-infra | 21:57 | |
| armax | jogo: I tried a couple of things, both promising…but carl_baldwin and I reached the conclusion if it would be sensible to relax the Tempest test a bit | 21:57 |
| armax | jogo: and make it wait for the state transition to occur. | 21:57 |
| armax | jogo: I just wanted to let you know that we’re actively looking at this | 21:58 |
| *** andreykurilin_ has quit IRC | 21:58 | |
| jogo | armax: awesome | 21:59 |
| armax | jogo: there’s more details on the bug report, so take your time to digest it :) | 21:59 |
| jogo | armax: do you have a tempest patch up to relax the test? | 22:00 |
| armax | jogo: we can put it up reasonably fast | 22:00 |
| fungi | sorry, just caught up on scrollback | 22:00 |
| *** yjiang5 has quit IRC | 22:00 | |
| armax | jogo: do you think that the Tempest patch is a viable fix? | 22:00 |
| jogo | armax: if you guys think relaxing the tempest test to wait for a state transition is reasonable then its worth a shot | 22:01 |
| *** pc_m_ has quit IRC | 22:02 | |
| armax | jogo: that’s how the backend implementation works | 22:03 |
| armax | jogo: between the time the FIP becomes unreachable and the states goes from ACTIVE to DOWN can take some time | 22:03 |
| armax | jogo: the agent may even go down and the state can be stuck in ACTIVE | 22:03 |
| armax | jogo: not pretty; ) | 22:03 |
| armax | :) | 22:03 |
| jkt | hmmm, when pip installing zuul, that pip piece of *** installs apscheduler 3.0.0rc1 despite the fact that zuul's requiremens are correct | 22:05 |
| *** yjiang5 has joined #openstack-infra | 22:05 | |
| *** mgagne has quit IRC | 22:05 | |
| *** mgagne has joined #openstack-infra | 22:05 | |
| armax | jogo: we’ll put a patch for review to tempest soon, we can take the discussion there if needed | 22:06 |
| mtreinish | armax: ping me when you push it up for review | 22:08 |
| jogo | armax: thanks | 22:08 |
| armax | mtreinish: will do | 22:08 |
| carl_baldwin | jogo: armax: It boils down to the fact that the agent has to bring the fip down and report status somewhat independently. They can’t be done atomically. I tend to think it is better for it to ensure the fip is actually down before reporting that status which is the current implementation. | 22:09 |
| carl_baldwin | armax: jogo: mtreinish: I’ll put up a patch and let you know. | 22:09 |
| *** wenlock has joined #openstack-infra | 22:09 | |
| *** JayJ has quit IRC | 22:09 | |
| armax | carl_baldwin: both strategies have flaws | 22:10 |
| *** arnaud__ has quit IRC | 22:11 | |
| *** arnaud has quit IRC | 22:11 | |
| armax | carl_baldwin: so it’s really hard to say | 22:11 |
| *** dane_leblanc has quit IRC | 22:11 | |
| carl_baldwin | armax: Yes, both have flaws. | 22:12 |
| *** gokrokve has quit IRC | 22:12 | |
| *** wenlock has quit IRC | 22:12 | |
| *** sarob has quit IRC | 22:12 | |
| *** sarob_ has joined #openstack-infra | 22:12 | |
| *** david-lyle has quit IRC | 22:13 | |
| clarkb | ok mostly semi back | 22:13 |
| *** david-lyle has joined #openstack-infra | 22:13 | |
| clarkb | and look at that scrollback | 22:13 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1384949 https://review.openstack.org/130646 | 22:14 |
| uvirtbot | Launchpad bug 1384949 in nova "Boto 2.34.0 causing stacktraces" [Undecided,New] https://launchpad.net/bugs/1384949 | 22:14 |
| armax | carl_baldwin: I would be inclined to think that being upfront and say, I am gonna mess with your FIP (i.e. consider it down for what I am about do) is more conservative and hence preferred from a client perspective | 22:15 |
| armax | carl_baldwin: that said, both strategies will require state recovery in face of failures, which we don’t currently have | 22:15 |
| *** gokrokve has joined #openstack-infra | 22:20 | |
| *** jgrimm is now known as zz_jgrimm | 22:21 | |
| carl_baldwin | armax: agree re: state recovery | 22:21 |
| clarkb | jkt: update your pip | 22:22 |
| armax | carl_baldwin: perhaps that’s something we’ll add to the summit agenda, so that we don’t forget | 22:22 |
| clarkb | jkt: if you update your pip it should not install prereleases unless you ask for the | 22:22 |
| *** dangers is now known as dangers_away | 22:23 | |
| jkt | clarkb: this is pip 1.3.1 from el7 | 22:23 |
| clarkb | jkt: ancient | 22:23 |
| clarkb | you want >1.4.1 iirc | 22:23 |
| clarkb | 1.3 doesn't even check ssl certs | 22:23 |
| clarkb | jogo: fungi: any update on the fedora image thing? | 22:23 |
| jkt | clarkb: and FYI, it even refused to install a correct version when I fed it with a ==2.1.2 because I had some file already downloaded below /tmp/... | 22:24 |
| fungi | clarkb: i believe the decision was to drop the icehouse heat slow job | 22:24 |
| clarkb | jkt: new pip is much much better | 22:24 |
| fungi | clarkb: though separately we ought to do the url change to the mirror pool maybe | 22:25 |
| clarkb | fungi: ++ we really should | 22:25 |
| clarkb | it iwll make image builds much faster | 22:25 |
| mtreinish | clarkb: jogo pushed patches out to all the devstack branches which switched the image url to the round robin one | 22:26 |
| fungi | ahh, great--i wasn't sure whether that had happened yet | 22:26 |
| jkt | the fact that the pypi website doesn't show older versions (see for yourself at https://pypi.python.org/pypi/APScheduler and https://bitbucket.org/pypa/pypi/issue/52/website-should-list-all-versions-of-a) | 22:27 |
| *** mfink has quit IRC | 22:27 | |
| jkt | ...is extremely annoying. | 22:27 |
| mtreinish | fungi: yeah, it looks like the 2 stable patches have already landed. Master bounced off the gate | 22:28 |
| fungi | jkt: a hackaround is to switch /pypi/ to /simple/ in the url | 22:28 |
| lifeless | jkt: can always look at https://pypi.python.org/simple/apscheduler/ as a workaround | 22:28 |
| fungi | jkt: ugly but usable | 22:28 |
| jkt | what? :) | 22:28 |
| jkt | this is almost as bad as gerrit's build system which changes itself based on what plugin it's building (each project can require a specific git commit as a version...) | 22:29 |
| viglesias | ashp: thanks for the pointer i tried that already | 22:29 |
| jkt | anyway, thanks, I'll stop wasting your time with pointless rants now | 22:29 |
| *** wenlock has joined #openstack-infra | 22:29 | |
| clarkb | fungi: I added a paragraph to the py26-deprecation etherpad | 22:30 |
| *** harlowja has quit IRC | 22:30 | |
| clarkb | fungi: to talk about timeline for these changes. does it look good to you? | 22:30 |
| fungi | clarkb: what's a "Novement"? | 22:31 |
| clarkb | wow | 22:31 |
| clarkb | this is why I get other people to read my things | 22:31 |
| fungi | i didn't....correct...it since it didn't quite know what it should have been | 22:31 |
| mtreinish | clarkb: is that november or no movement? | 22:31 |
| fungi | i assumed it was the nova elevating moment | 22:31 |
| clarkb | no vembers | 22:31 |
| *** __TheDodd__ has quit IRC | 22:32 | |
| fungi | yes we have no vembers | 22:32 |
| *** gyee has joined #openstack-infra | 22:32 | |
| fungi | clarkb: lgtm, though it loses a little flavor without the novement | 22:33 |
| clarkb | if there is no opposition to that I can go ahead and bottle it up and ship it | 22:33 |
| clarkb | will give it a bit more time | 22:33 |
| fungi | cool. pack and ship at your leisure | 22:34 |
| fungi | i'm relaxing by trying to figure out how to hack a new pipeline trigger into zuul | 22:35 |
| *** bitblt has quit IRC | 22:36 | |
| fungi | for what i want to do, i think i can actually consume change-merged events | 22:36 |
| clarkb | fungi: oh I was going to ask if you are working on the github trigger ;) | 22:36 |
| fungi | since unlike for the post pipeline, i don't need the ref, just the refname | 22:36 |
| *** mjturek has quit IRC | 22:36 | |
| fungi | clarkb: what's a Gi-Thub? | 22:37 |
| clarkb | fungi: its a bit like a no vember | 22:37 |
| fungi | yeah, no, i'm hacking on free software ;) | 22:37 |
| *** dimtruck is now known as zz_dimtruck | 22:37 | |
| *** zz_dimtruck is now known as dimtruck | 22:40 | |
| *** harlowja has joined #openstack-infra | 22:41 | |
| wenlock | hi, wondering what i need to do for https://review.openstack.org/#/c/117892/ to merge | 22:41 |
| clarkb | wenlock: I think you need to line up someone to babysit it | 22:42 |
| zaro | clarkb: GP it's already been tagged and released, ver 0.1.1 -> http://repo.jenkins-ci.org/repo/org/jenkins-ci/plugins/gearman-plugin/ | 22:42 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Plugins may now register cron workers. https://review.openstack.org/129609 | 22:43 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added application cache https://review.openstack.org/130649 | 22:43 |
| *** ZZelle_ has quit IRC | 22:43 | |
| fungi | ashp: if you want to upgrade your preview tarball of gearman-plugin to the official point release, sounds like you can now | 22:43 |
| clarkb | zaro: when did that happen? | 22:43 |
| *** dkranz has quit IRC | 22:43 | |
| clarkb | zaro: I completely missed it. Any chacne you want ot update the version in puppet or should I/ | 22:44 |
| *** camunoz has joined #openstack-infra | 22:44 | |
| ashp | fungi: awesome, I spent the day reintegrating my changes into our cookbooks as I have to redeploy the whole thing in another region | 22:44 |
| *** signed8bit is now known as signed8bit_ZZZzz | 22:44 | |
| ashp | so it'll probably just pick up the latest when I do that | 22:44 |
| zaro | clarkb: it was done yesterday. You should do, i'm not going to be available anytime soon. | 22:45 |
| *** gokrokve has quit IRC | 22:46 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: multi-node dg setup https://review.openstack.org/106043 | 22:46 |
| *** gokrokve has joined #openstack-infra | 22:46 | |
| fungi | wenlock: that change looks okay to me... i can merge and watch it on the puppetmaster if you want to keep an eye on http://puppetboard.openstack.org/ but first it looks like we have some cleanup i should take a look at (judging from the state of the board) | 22:46 |
| fungi | in particular, nodepool is teh unhappiness | 22:47 |
| wenlock | fungi, that would be awesome | 22:47 |
| wenlock | i'll keep an eye on the board if you want | 22:47 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/nodepool: Support multiple image formats in a diskimage https://review.openstack.org/126747 | 22:48 |
| fungi | ianw: we fail at reviewing | 22:48 |
| *** gondoi is now known as zz_gondoi | 22:48 | |
| clarkb | fungi: uh oh what broke? | 22:48 |
| fungi | clarkb: thankfully a not-often-used apache site | 22:48 |
| wenlock | fungi: on jenkins this error: Could not retrieve information from environment production source(s) puppet:///modules/openstack_project/nodepool/scripts | 22:49 |
| clarkb | fungi: looking at zuul status we don't seem to be addressing demand | 22:49 |
| fungi | Syntax error on line 3 of /etc/apache2/sites-enabled/50-nodepool.openstack.org.conf: DocumentRoot takes one argument, Root directory of the document tree | 22:49 |
| *** amitgandhinz has joined #openstack-infra | 22:49 | |
| clarkb | any chance that is related? | 22:50 |
| clarkb | seems the world broke just under an hour ago | 22:50 |
| fungi | clarkb: doesn't look related, but there's a '/bin/sh /etc/init.d/nodepool restart' in the process table since 21:50 utc | 22:50 |
| clarkb | that wasn't me | 22:51 |
| fungi | so something wanted to gracefully restart nodepool and it hung... does puppet try to do that? | 22:51 |
| fungi | checking logs now | 22:51 |
| *** freemanbrandon has quit IRC | 22:51 | |
| clarkb | I don't think puppet does that | 22:51 |
| clarkb | mordred: jeblair: SergeyLukjanov ^ | 22:52 |
| wenlock | clarkb nodepool/scripts folder move? | 22:52 |
| *** jaypipes has joined #openstack-infra | 22:52 | |
| *** hashar has quit IRC | 22:52 | |
| fungi | Oct 23 21:50:24 nodepool puppet-agent[23728]: (/Stage[main]/Nodepool/File[/etc/nodepool/logging.conf]) Scheduling refresh of Service[nodepool] | 22:53 |
| fungi | i think it must | 22:53 |
| clarkb | fungi: wow | 22:53 |
| clarkb | I thought we didn't do that anywhere | 22:53 |
| fungi | anyway, i'll get that cleaned up real quick and then see what's broken with ianw's vhost improvements | 22:54 |
| *** amitgandhinz has quit IRC | 22:54 | |
| *** david-lyle has quit IRC | 22:54 | |
| wenlock | fungi clarkb http://paste.openstack.org/show/123766/ | 22:54 |
| *** lbiasi has joined #openstack-infra | 22:54 | |
| *** xyang0 has quit IRC | 22:56 | |
| *** SumitNaiksatam has joined #openstack-infra | 22:56 | |
| wenlock | thanks fungi | 22:58 |
| *** dmsimard is now known as dmsimard_away | 22:59 | |
| clarkb | there is indeed a single notify of nodepool on the logging conf | 22:59 |
| clarkb | I am going to remove that | 22:59 |
| *** sdake_ has joined #openstack-infra | 23:00 | |
| anteaya | novement indivirtually and temptex | 23:01 |
| anteaya | quite the day for coined words | 23:01 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/system-config: Correct nodepool log document root reference https://review.openstack.org/130651 | 23:01 |
| fungi | brown bag ^ | 23:01 |
| *** doug-fish has left #openstack-infra | 23:02 | |
| *** HenryG has joined #openstack-infra | 23:04 | |
| *** julim has quit IRC | 23:05 | |
| *** gokrokve has quit IRC | 23:05 | |
| *** sabeen has quit IRC | 23:06 | |
| openstackgerrit | Jan Kundrát proposed a change to openstack-infra/zuul: Validate top level of the layout configuration, too https://review.openstack.org/130652 | 23:06 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/system-config: Stop notifying nodepool on log config changes https://review.openstack.org/130653 | 23:06 |
| clarkb | fungi: ^ trade you | 23:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/system-config: Host a devstack.org redirect on static https://review.openstack.org/130001 | 23:07 |
| *** arnaud has joined #openstack-infra | 23:07 | |
| ianw | fungi: sorry :( | 23:07 |
| fungi | ianw: no worries. the :: make me go boss-eyed when reviewing puppet, so easy to miss | 23:08 |
| ianw | fungi: that's kind of why i'd like to get https://review.openstack.org/#/c/129496/ in; i think the small .htaccess for this case is fine, rather than keeping the whole template full of unneeded boilerplate | 23:10 |
| ianw | fungi: we could also clean up a bunch of MEANINGLESS_ARGUMENTS :) | 23:11 |
| *** andreaf has quit IRC | 23:11 | |
| *** andreaf has joined #openstack-infra | 23:11 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/system-config: Update gearman-plugin to version 0.1.1. https://review.openstack.org/130656 | 23:12 |
| clarkb | zaro: ^ | 23:12 |
| zaro | viglesias: you still running your own gearman or using the python gear implementation?` | 23:13 |
| *** sdake_ has quit IRC | 23:13 | |
| clarkb | ianw: so you have heard my argument right? | 23:13 |
| clarkb | ianw: I don't think I would -2 any conversions but I would leave -1s on them for sure | 23:13 |
| clarkb | ianw: shipping mostly complete vhosts is super valuable | 23:13 |
| clarkb | ianw: and converting them to more puppet native types is a major problem imo | 23:14 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/system-config: Update gearman-plugin to version 0.1.1. https://review.openstack.org/130656 | 23:14 |
| clarkb | (I really dislike that puppet feels the need to treat files as not files, it makes your code less portable) | 23:14 |
| clarkb | unix is cool because file all the things | 23:14 |
| fungi | clarkb: ^ corrected a potentially confusing typo in the commit message | 23:14 |
| *** gokrokve has joined #openstack-infra | 23:14 | |
| ianw | clarkb: yes, i do kind of agree; heaven knows i don't want to sit there and try to essentially translate apache syntax into puppet-module argument syntax | 23:14 |
| clarkb | fungi: thanks | 23:15 |
| clarkb | fungi: for a second I was worried I pulled a tchaypo and spelled gearman as german | 23:15 |
| *** ashp has quit IRC | 23:15 | |
| ianw | clarkb: but, it would be nice if we could add a two line patch to it so we could set overrides without having to write out a whole template, when that template is otherwise not required | 23:15 |
| tchaypo | baha | 23:15 |
| clarkb | tchaypo: :) | 23:15 |
| zaro | clarkb: i'm @ jenkins conf. lots of interest in JJB this year. | 23:16 |
| ianw | clarkb: and also get rid of the silly MEANINGLESS_ARGUMENT docroot that's carried everywhere | 23:16 |
| clarkb | zaro: neat | 23:16 |
| clarkb | ianw: so my understanding is that to do that we have to convert the vhosts to puppet types | 23:16 |
| clarkb | ianw: bcause new apache module does not allow raw vhosts | 23:16 |
| * zaro takes the rest of the week off (it's a date weekend) | 23:17 | |
| ianw | clarkb: not if we fork 0.0.4 a-la https://review.openstack.org/#/c/129496/ and then just add a few small, sensible things to it | 23:17 |
| clarkb | oh you want to fork fork it | 23:17 |
| clarkb | I think I am -1 on that too but need to think about it more | 23:17 |
| *** gokrokve has quit IRC | 23:17 | |
| clarkb | I was thinking just rename it to avoid name collisions and have considered properly forking it | 23:17 |
| clarkb | zaro: have fun | 23:17 |
| ianw | clarkb: well as it is, we can't go forward, we can't fix anything, so ... | 23:17 |
| *** tonytan4ever has quit IRC | 23:18 | |
| anteaya | tchaypo: so I forgot to mention, but https://github.com/jamezpolley/prep-source-repos has 7 branches | 23:18 |
| clarkb | oh I meant to point tchaypo at the zuul cloner | 23:18 |
| anteaya | tchaypo: once a repo is pulled into gerrit, you need an infra core to delete branches | 23:18 |
| clarkb | do zuul cloner and prep source repos collide? | 23:18 |
| *** gokrokve has joined #openstack-infra | 23:18 | |
| anteaya | tchaypo: how many of those do you want to keep to pull in? | 23:18 |
| anteaya | *hint* the fewer the better | 23:19 |
| *** dkliban is now known as dkliban_pto | 23:20 | |
| anteaya | oh my I love this eclipse so far | 23:20 |
| *** gokrokve has quit IRC | 23:23 | |
| *** otter768 has quit IRC | 23:23 | |
| clarkb | jesusaurus: about? I amre reviewing that nodepool chnage | 23:24 |
| *** david-lyle has joined #openstack-infra | 23:24 | |
| clarkb | jesusaurus: I am not sure how the test would work. BAD_CLIENT always throws an exception. Don't you need a client that throws first time then doesn't throw on second attempt? | 23:24 |
| clarkb | jesusaurus: I don't understand how the test passes | 23:26 |
| jesusaurus | clarkb: well, i want to make sure that i am creating a new client object when the exception is caught | 23:26 |
| jesusaurus | clarkb: so when resetClient() runs it goes through the normal code path and sets the self.client to be FAKE_CLIENT | 23:26 |
| clarkb | jesusaurus: but we aren't asserting that. intsead we are just checking that we can list extensions | 23:26 |
| clarkb | jesusaurus: oh | 23:27 |
| clarkb | I think that is the bit I am missing. I am going to ask you to comment that in the test :) | 23:27 |
| jesusaurus | clarkb: is that being too clever? | 23:27 |
| clarkb | jesusaurus: I think its too clever from the perspective of the test | 23:28 |
| clarkb | jesusaurus: the code itself is fine | 23:28 |
| clarkb | so we can just document what should happen in the test | 23:28 |
| jesusaurus | okay, I'll add a comment | 23:28 |
| clarkb | jesusaurus: I left comments on the change. a couple other things too | 23:28 |
| clarkb | (minor0 | 23:28 |
| jesusaurus | clarkb: thanks | 23:29 |
| *** arnaud__ has joined #openstack-infra | 23:30 | |
| *** david-lyle has quit IRC | 23:31 | |
| openstackgerrit | A change was merged to openstack-infra/system-config: Correct nodepool log document root reference https://review.openstack.org/130651 | 23:31 |
| openstackgerrit | A change was merged to openstack-infra/system-config: Stop notifying nodepool on log config changes https://review.openstack.org/130653 | 23:31 |
| *** Guest1921 has quit IRC | 23:32 | |
| *** gokrokve has joined #openstack-infra | 23:34 | |
| *** gokrokve has quit IRC | 23:34 | |
| *** gokrokve has joined #openstack-infra | 23:34 | |
| *** carl_baldwin has quit IRC | 23:34 | |
| *** ChuckC has quit IRC | 23:36 | |
| viglesias | zaro: still doing gear | 23:38 |
| *** gokrokve has quit IRC | 23:42 | |
| viglesias | zaro: i can switch back to my own gearman trivially, so will try that again later tonight | 23:43 |
| fungi | viglesias: i think there was at least one feature implemented in gear which isn't in the official gearmand (yet anyway). the zuul docs mention it... | 23:44 |
| viglesias | ah ack i remember jeblair mentioning that | 23:45 |
| fungi | viglesias: yep, see http://ci.openstack.org/zuul/launchers.html#id2 | 23:45 |
| viglesias | cool thanks for the pointer | 23:46 |
| fungi | viglesias: i think there may be patches or work in progress to get those in gearmand as well but not sure what the status is | 23:46 |
| fungi | Shrews might know if he's lame enough to still be online at this time of night | 23:46 |
| fungi | but hopefully he is not and has a life instead | 23:47 |
| clarkb | time of night! bah | 23:49 |
| clarkb | you NCers | 23:49 |
| *** oomichi has joined #openstack-infra | 23:49 | |
| fungi | i'm speaking to you from _THE_FUTURE_ | 23:49 |
| *** oomichi__ has joined #openstack-infra | 23:49 | |
| fungi | (echo, echo, echo) | 23:49 |
| *** ashp has joined #openstack-infra | 23:50 | |
| * viglesias runs from the channel!!!!!! | 23:51 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!