Thursday, 2016-03-03

rcarrillocruzlulz, groovy...00:00
*** roxanaghe has quit IRC00:01
clarkbI wonder if 2.0 will fix the slaves in config problem00:01
*** sridhar_ram has joined #openstack-infra00:02
*** Sukhdev has quit IRC00:03
*** nelsnels_ has quit IRC00:04
*** ybathia has quit IRC00:05
*** Sukhdev has joined #openstack-infra00:05
*** sridhar_ram has quit IRC00:06
*** dims has joined #openstack-infra00:06
*** markvoelker has quit IRC00:06
*** nelsnelson has joined #openstack-infra00:06
*** gildub has joined #openstack-infra00:07
*** zz_dimtruck is now known as dimtruck00:07
*** sridhar_ram has joined #openstack-infra00:10
kevinbentoncan the graphite graphs for job failures fall behind like logstash?00:11
*** amotoki has quit IRC00:11
*** sarob has joined #openstack-infra00:11
clarkbkevinbenton: shouldn't those don't end up behind a processing queue other than the statsd server applying the info00:11
*** mriedem has joined #openstack-infra00:12
kevinbentonack. thanks00:12
*** rbrndt_ has quit IRC00:14
*** ybathia has joined #openstack-infra00:14
*** IlyaG has quit IRC00:14
lifelesstchaypo: thats groovy00:17
cjoh hey!  Finally!  jenkins likes my patch :-)00:17
*** sdague has quit IRC00:18
*** annegentle has joined #openstack-infra00:19
*** rhallisey has quit IRC00:19
*** Sukhdev has quit IRC00:20
*** bpokorny has quit IRC00:21
*** ybathia has quit IRC00:22
*** annegentle has quit IRC00:24
*** yamamoto has joined #openstack-infra00:24
*** Jeffrey4l has quit IRC00:27
*** abregman has quit IRC00:31
*** yamamoto has quit IRC00:31
*** andreykurilin__ has quit IRC00:33
*** bpokorny has joined #openstack-infra00:33
clarkbso far everything I see running on vexxhost seems happy00:35
nibalizerdo we have nodes running there now?00:35
nibalizerthats awesome00:35
clarkbthe neutron-full jobs have increased their run time (maybe we are our own noisy neighbors?) but I think its till within the time frame as our other clouds00:35
clarkbnibalizer: quota of 25 test instances00:35
jheskethclarkb: awesome work :-)00:36
nibalizer seems to be things00:36
nibalizerclarkb: yes great job?00:36
clarkbjhesketh: it was mostly mnaser, they made it very easy for us00:36
*** dimtruck is now known as zz_dimtruck00:36
*** xarses has quit IRC00:36
nibalizerer great job! not ?00:36
jheskethwell kudos to mnaser too then :-)00:36
clarkbit is very vanilla and other than the glance image upload hiccup everything on our side just worked with their side00:37
mnaserthanks :) clarkb carried it out super quick00:38
clarkbmnaser: and do feel free to ask for more instance information if tracking down hiccups and odd behavior. Happy to dig it up00:39
mnaserclarkb: i'll probably bug about it every once in a while instead of every single time a single instance has something go on00:40
*** amotoki has joined #openstack-infra00:40
jheskethclarkb: what was that hiccup btw?00:41
*** sdake_ has quit IRC00:41
*** david_cu has joined #openstack-infra00:41
clarkbjhesketh: nginx in front of glance was proxying in front of slower glance backend00:41
clarkbjhesketh: this caused glance clien tto think upload was done well before it got confirmation of that fact causing "connectivity" problems in glance client00:41
clarkbjhesketh: mnaser updated their proxy settings and everything worked just fine00:42
jheskethah cool00:42
jheskethnice work00:42
mnaseryeah, we use nginx to do all of our ssl termination00:43
mnaserand the request buffering was confusin glanceclient00:43
openstackgerritTravis Tripp proposed openstack/requirements: Add python-searchlightclient>=0.2.0
*** thorst has joined #openstack-infra00:44
*** sdake has joined #openstack-infra00:45
openstackgerritMerged openstack-infra/project-config: mostly standardize dsvm tempest jobs on 180 minute backstop
*** ZZelle_ has quit IRC00:45
*** resker has joined #openstack-infra00:45
mgagnewould be great to mention those details somewhere (nginx vs glance) so others can benefit from your finding/experience00:46
*** esker has quit IRC00:46
mnasermgagne: good idea, i'll see where i can put it out (though i'm not sure how many people run nginx infront of openstack services for api termination)00:46
clarkbmgagne: speaking of details any chance you are willing to tl;dr the gre + security groups thing for us again so I can respond on the neutron security groups thread or maybe you want to respond yourself?00:47
mgagnemnaser: neither do I know where to store that info00:47
mgagneclarkb: load gre conntrack helper, be happy00:47
clarkbmgagne: mnaser might make sense in the deployment guides?00:47
clarkbmgagne: mnaser under a "using http proxies" section or something00:47
mnaserperhaps, though none of the deployment guides say anything about using another serviec for SSL termination :p00:48
mgagneclarkb: I can dig the exact details as to why you should load it00:48
mnaserfiling a bug would be a good first step till i have sometime to document it but it's just 3 nginx lines00:48
clarkbmgagne: that would be great, I was trying to explain that security groups don't always honor the rules that are present and GRE not working was my example but I didn't have all the details handy00:48
mgagneclarkb: could be similar to FTP passive mode I guess00:48
openstackgerritMerged openstack-infra/project-config: Add missing gate-{name}-ansible-functional-ubuntu-trusty-nv
*** roxanaghe has joined #openstack-infra00:54
mgagneclarkb: I don't know much about GRE and its inner working but I suspect that it requires RELATED rules in iptables and RELATED requires a conntrack helper to work.
*** sarob has quit IRC00:56
*** amotoki has quit IRC00:57
mgagneclarkb: the coworker that helped us debug the issue is on PTO so won't be able to have details before next week00:57
clarkbmgagne: ok no rush as long as I/we can follow up on that00:57
*** bpokorny_ has joined #openstack-infra00:58
mgagne"Ok find it, must load the nf_conntrack_proto_gre kernel module and everything is fine." :-/00:58
*** ashleighfarnham has joined #openstack-infra00:58
*** rhallisey has joined #openstack-infra01:01
*** zz_dimtruck is now known as dimtruck01:01
*** bpokorny has quit IRC01:01
*** xarses has joined #openstack-infra01:03
*** larrie_kong is now known as kong01:04
openstackgerritMerged openstack-infra/project-config: Publish renderspec docs
*** xarses_ has joined #openstack-infra01:07
*** ddieterly has joined #openstack-infra01:07
*** xarses_ has quit IRC01:07
*** dims has quit IRC01:08
*** xarses_ has joined #openstack-infra01:08
*** xarses has quit IRC01:08
*** zhurong has joined #openstack-infra01:08
openstackgerritMerged openstack-infra/project-config: Add experimental Manila LVM job with minimal services
*** kzaitsev_mb has quit IRC01:10
*** markvoelker has joined #openstack-infra01:10
*** sarob has joined #openstack-infra01:11
*** geekinutah has quit IRC01:11
*** keedya has quit IRC01:12
*** doug-fish has joined #openstack-infra01:12
openstackgerritMerged openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty
*** xarses_ has quit IRC01:18
openstackgerritMerged openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests
openstackgerritMerged openstack-infra/project-config: gate-rally-install-devstack-centos7 job is ready for check & gate queue
*** Qiming has quit IRC01:20
openstackgerritMerged openstack-infra/project-config: Make lbaasv2-minimal job voting in check and gate
openstackgerritMerged openstack-infra/project-config: Add tacker liberty to gerrit bot
*** amotoki has joined #openstack-infra01:23
*** gildub has quit IRC01:23
*** Sukhdev has joined #openstack-infra01:29
*** sdake_ has joined #openstack-infra01:29
*** sdake_ has quit IRC01:30
*** xarses_ has joined #openstack-infra01:30
*** sdake has quit IRC01:30
*** sdake_ has joined #openstack-infra01:30
*** Sukhdev has quit IRC01:31
fungikudos mnaser and clarkb! from zero to nodepool in 24 hours flat. our fastest provider donation addition to production ever01:31
*** andreww has joined #openstack-infra01:32
*** sc68cal_ has quit IRC01:33
*** dizquierdo has quit IRC01:33
*** gildub has joined #openstack-infra01:33
*** xarses_ has quit IRC01:36
*** annegentle has joined #openstack-infra01:36
openstackgerritJeremy Stanley proposed openstack-infra/nodepool: Add an error log with the server fault message
fungimordred: ^ minor fix to that change01:37
*** gongysh has joined #openstack-infra01:37
*** cznewt has quit IRC01:38
*** cznewt has joined #openstack-infra01:38
clarkbmy local dib build VM is all kinds of unhappy updating its cache :/01:38
clarkbhaven't been able to get anyhwere on that today01:39
clarkbmay use cloud VM01:39
clarkbI want to try removing the extra/different elements one by one until I can see the delta in file size go away01:39
clarkbfirst build without growroot, then without simple-init, then s/ubuntu-minimal/ubuntu/01:40
*** amit213 has joined #openstack-infra01:41
clarkbthey don't need to be functional if we can identify where the file size is differnet01:41
*** sam_wan has joined #openstack-infra01:41
*** pvaneck has quit IRC01:41
*** bpokorny_ has quit IRC01:41
*** bpokorny has joined #openstack-infra01:42
*** annegentle has quit IRC01:43
*** rfolco has quit IRC01:44
*** gongysh has quit IRC01:45
mtreinishclarkb: you might like:
*** annegentle has joined #openstack-infra01:45
fungiyep, whether it's bootable or not is irrelevant for this experiment01:47
*** Swami has quit IRC01:47
*** openstackgerrit has quit IRC01:47
*** openstackgerrit has joined #openstack-infra01:48
bkeroTime to build every permutation of the dib elements in parallel01:48
*** tiswanso has joined #openstack-infra01:48
greghaynesclarkb: hrm?01:49
fungibkero: dib element matrixing as a service! our latest openstack project team01:49
greghaynesclarkb: dib caching is broken?01:49
*** tiswanso has quit IRC01:49
fungigreghaynes: our ubuntu-trusty images are ~2x the size of our devstack-trusty images, while using basically the same set of elements01:49
*** Sukhdev has joined #openstack-infra01:49
*** tiswanso has joined #openstack-infra01:50
greghaynesfungi: ah, right, I think clarkb mentioned that to me a bit ago01:50
clarkbfungi: one thing I noticed is we still cache stackforge01:50
greghaynesfungi: I think one of those is -minimal and one is not?01:50
clarkbfungi: and we probably don't need to01:50
fungiclarkb: meh, it's tiny01:50
clarkbgreghaynes: ya its ubuntu vs ubuntu-minimal + growroot + simple-init01:50
fungii mean, i'm fine if we make a filter list of repo patterns to skip from caching, but that's verging on premature optimization01:50
greghaynesubuntu vs ubuntu-minimal seems like the only real difference01:51
clarkbfungi: sure it won't affect the 11GB vs 5.7GB problem01:51
clarkbfungi: but it will affect general image size and we upload them a lot and across oceans01:51
*** sdake_ has quit IRC01:51
greghaynesand -minimal will definitely have more stuff in the apt cache since it will be installing everything from apt rather than just copying a cloudimage rootfs01:51
*** Sukhdev has quit IRC01:52
*** esikache1 has joined #openstack-infra01:52
clarkbgreghaynes: hrm is it doubled up packages, once installed and once in cache?01:52
fungiclarkb: yep, but fixing the extra 5gb of ubuntu-trusty will likely buy us a few years of waste caching stackforge repos to break even01:52
greghaynesclarkb: IIRC we copy in the deb cache in to the images we build, so yes01:52
clarkbin theory its the same cache on both though01:52
clarkbbecause the build host is shared01:53
greghaynesclarkb: ah01:53
fungimmm, i thought we disabled the local apt cache via envvars01:53
greghaynesoh, thats possible01:53
clarkbthat would explain it then01:53
clarkbso I guess the experiment to do is without the apt cache01:53
*** fawadkhaliq has quit IRC01:53
fungialso, the apt cache would show up as eating space in /var/cache01:53
greghaynesoh, you think one of them has the apt cache and one does not?01:53
clarkbgreghaynes: well I think fungi was saying we don't share the cache across the builds01:54
clarkbso both are caching but minimal may be caching a lot more01:54
greghaynesyes, I would expect minimal to do that01:54
clarkbvar is less than a gig01:54
bkeroweird, posting pastebins in the future01:54
fungiwe had to do some gyrations to get dib to not clean the apt cache out of the images, and to not dirty the apt cache in the images with things we weren't explicitly caching (e.g. old cache cruft)01:55
bkeroalso yeah, didn't we find out it's in /opt?01:55
clarkbbkero: we found that /opt is the biggest disk consumer but its the same on both01:55
clarkbbecause that cache is shared01:55
*** roxanaghe has quit IRC01:55
greghaynesclarkb: do you have a similar du for the other image?01:55
bkeroclarkb: But the discrepency is 5gb?01:55
clarkbgreghaynes: no but I can get one01:55
fungicould there be unlinked inodes wasting space on one and not the other? i'm stabbing in the dark now01:56
*** claudiub|2 has joined #openstack-infra01:56
greghaynesonly if tar is somehow picking them up01:56
greghaynesor cp01:56
fungibecause your df said something like 10gb of space used in the rootfs on an ubuntu-trusty instance, so it's gotta be something in there01:56
bkeroSo these are qcow2s, right?01:56
clarkbbkero: yes01:56
bkeroqcow2s won't shrink01:56
*** esikache1 has quit IRC01:56
clarkbthe raw image is 17GB01:56
clarkbso we start both from the same disk usage01:57
bkeroclarkb: What if that extra 5gb is deleted stuff?01:57
*** taron1 has quit IRC01:57
greghaynesoh, let me remember how we calculate the partition size01:57
bkeroqcow2 will balloon, but it won't shrink unless you do a conversion01:57
clarkbWe don't really delete anything01:57
clarkbits all additive01:57
bkeroYou don't delete debs from /var/cache/apt/archive?01:58
clarkbwe want them in our images01:58
fungiwe want them in there because we only download them but don't install them01:58
clarkbso that installing mysql is fast01:58
fungijobs install packages if they need them, and having them in a warm cache on the instance is the fastest solution to that01:58
*** claudiub has quit IRC01:58
bkeroYou want the debs you're going to use for local cache, but not hte ones you aren't01:58
greghayneshah, so I just noticed something interesting in dib01:59
*** dims has joined #openstack-infra01:59
greghayneswe determine the root partition size by doign a du on the built directory01:59
greghaynesthat happens *before* cleanup phase runs01:59
bkeroclarkb: That's 500mb of difference in du size01:59
clarkbya nowhere near ~5.3 gb02:00
fungihuh, so the 6.8gb on disk in the devstack-trusty instance is smaller than the entire qcow2 it was booted from?02:00
bkeroSo some extra non-file padding in the image?02:00
greghaynesok, so I wonder if thats what is going on - dib is calculating a larger root fs size than we need02:00
fungi6.8gb on disk in /opt i mean02:00
clarkbfungi: yes it compresses02:00
clarkbfungi: 17GB is raw02:00
clarkbgreghaynes: maybe02:00
*** mtanino has quit IRC02:01
greghaynesclarkb: can you build this image again?02:01
fungiso maybe the ubuntu-trusty image isn't compressing as efficiently?02:01
bkeroclarkb: what's the block device size on both of them?02:01
greghaynesor that02:01
bkerofungi: I think so02:01
*** taron has joined #openstack-infra02:01
greghaynesI was going to suggest adding a set -x at the top of disk-image-create, we can then see what du_size ends up getting set to02:01
greghaynesbut you could also just check the partition size for both the images partition tables02:01
clarkbbkero: isnt that determined by where you boot not image?02:02
clarkbpartition size is broken by grow root02:02
fungiyeah, block device size of the booted instances should be the same02:03
greghaynesclarkb: yea, youd have to look at it by converting the image and mounting on loopback02:03
greghaynesclarkb: before boot02:03
clarkbok I just left computer for kids but maybe fungi can check that02:03
fungii can give it a shot02:03
greghaynesdont let the kids get at that computer!02:03
bkeroIf the kids get ahold of those images they'll probably be 99999gb02:04
fungiconvert them from qcow2 to raw? or do we keep raw copies of these around already02:04
greghaynesfungi: Unsure if we keep raws, if we dont then yes they need to be converted to raw02:04
bkerocan qemu-nbd loopback mount, then fdisk -l the loopback device02:04
clarkbwe keep raws02:04
fungitrying to remember what directory it holds them in02:04
clarkbbasically you need to nodepool dib image list to get filename mapping02:05
*** aeng has quit IRC02:05
clarkbthey are in /opt/nodepool_din02:05
clarkber _dib02:05
fungithough it lies about the filenames02:06
clarkboh its the id num02:07
fungilet's call that a cosmetic bug for now02:07
greghaynesI may have to run in a few to deal with inlaws, but when we see that the partition sizes are significantly different the next question I'd have is how much of that space is used, and then id want to drop a bash in disk-image-create righ after it calculates du_size to see what is going on02:07
*** Qiming has joined #openstack-infra02:07
*** annegentle has quit IRC02:07
*** annegentle has joined #openstack-infra02:08
*** yamamoto_ has joined #openstack-infra02:09
greghaynesI think its possible that -minimal does more cleanup after a build, but we do cleanup after calculating root partition size so we overestimate what we need02:09
bkeroIs that cleanup going to result in lower disk size though?02:09
bkeroThe dusage reported by the filesystem is the same02:09
greghaynesbkero: itll result in a lower used space but the actual fs and partition size will not be smaller - we calculate the size of that before cleanup02:10
*** thorst has quit IRC02:10
*** dimtruck is now known as zz_dimtruck02:10
*** thorst has joined #openstack-infra02:10
bkeroWhat I mean can disk compression reap that now-free space?02:11
greghaynesoh, thats up to qcow2 in this case, and I have no idea how effective that is02:12
bkeroThat's what I'm suspecting is causing the discrepency02:12
*** annegentle has quit IRC02:12
fungii'm flailing mounting these raw images. mount wants a filesystem type... is there an mbr i need to offset past?02:12
bkeroIf I have a 5gb ubuntu qcow2, then dd 50gb of crap to it, then delete the crap, the resulting qcow2 will still be > 5gb02:13
bkerofungi: if the images have partitions, then kpartx -a $image, then look in /dev/mapper for the partitions02:13
fungiobvious filesystem types are erroring02:13
greghaynesfungi: usually once I make a loopback devide for that file the different partitions show up in /dev/loopXN02:13
greghayneser, loopback device02:13
fungioh, losetup it02:13
bkero(can losetup /dev/loop0 $FILE; kpartx -a /dev/loop0; mount /dev/mapper/loop0p1 /mnt)02:14
fungimount -o loop not gonna work for raw disk. derp02:14
bkeroHuh, I've never gotten loopback devices to make partition nodes by themselves before02:14
greghaynesYea, its a pretty annoying series of hoops to jump through02:14
bkerolibguestfs was supposed to help with that. Nerp.02:15
*** yuanying_ has joined #openstack-infra02:15
*** lezbar__ has joined #openstack-infra02:15
*** doug-fis_ has joined #openstack-infra02:16
*** markmcd__ has joined #openstack-infra02:16
*** armax_ has joined #openstack-infra02:16
*** hrybacki_ has joined #openstack-infra02:16
fungithey're basically the same:
fungiand no extra partitions in either one02:18
bkeroqcow2's just doing a better job shrinking or one has some deleted bloat during the install that qcow2 isn't compressing02:18
greghaynesso that points to compression simply being less effective for one02:18
*** IlyaG has joined #openstack-infra02:18
fungistrangely, ubuntu-trusty is using some 300mb more even though it should in theory be more minimal, but whatevs, these are not the bytes you're looking for02:18
*** thorst has quit IRC02:18
*** SotK_ has joined #openstack-infra02:18
*** dosaboy_ has joined #openstack-infra02:18
bkerowere both images made at the same time on the same system? Maybe different versions of qemu-img resulted in different vresions of qcow2?02:19
*** dangers_` has joined #openstack-infra02:19
*** jokke__ has joined #openstack-infra02:19
bkerofungi: what does 'file' show for them?02:19
*** elmiko_ has joined #openstack-infra02:19
*** aloga_ has joined #openstack-infra02:19
*** icey_ has joined #openstack-infra02:19
greghaynesfungi: can you mount those partitions and see how much free space is in each?02:19
fungibkero: "x86 boot sector" for both02:19
bkerofungi: I mean the qcow2 files02:20
bkeroThey list versions in the magic file string02:20
bkerothing: QEMU QCOW Image (v3), 1073741824 bytes02:20
fungi/opt/nodepool_dib/176350.qcow2: QEMU QCOW Image (v2), 17681350656 bytes02:20
fungi/opt/nodepool_dib/176353.qcow2: QEMU QCOW Image (v2), 18188664832 bytes02:20
greghaynesI also wonder if we leave a lot more garbage in the fs for the -minimal image, possibly by using more space then deleting files at which point qcow2 would be less effective at compression becuase itll be random data rather then 0's02:20
fungithe first is the devstack-trusty i mounted, the second is the ubuntu-trusty i mounted02:20
*** eliqiao_ has joined #openstack-infra02:20
*** dolpher1 has joined #openstack-infra02:20
fungiwell, the qcow2 versions of the raws i mounted anyway02:21
*** alex_xu_ has joined #openstack-infra02:21
fungigreghaynes: yeah, that's where i've been leaning02:21
bkeroYeah, that's what I've been saying. The FS won't zero deleted data, so it's not compressible. The deleted data will just sit there.02:21
openstackgerritJamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services
bswartzyou can force the FS to zero empty blocks02:22
*** claudiub|2 has quit IRC02:22
*** stevebak` has joined #openstack-infra02:22
*** sgordon` has joined #openstack-infra02:22
fungiwe can hackishly `cp /dev/zero /foo;rm foo` but is there a better way?02:22
greghaynesTIL there is a tool called zerofree02:22
bkerobswartz: do you know the tool for that?02:22
bswartzwhat fungi said02:22
bkeroor the tune2fs option?02:22
greghaynesI wonder if we should try that02:22
*** odyssey4me_ has joined #openstack-infra02:22
bswartzdd if=/dev/zero of=/delete_me bs=1M02:22
*** andreww has quit IRC02:23
bkeroEven then you might not be able to shrink it unless you convert to raw then back to qcow2 again02:23
fungiworth adding a zerofree element in dib (or just calling it directly in each minimal element?)02:23
*** manfred has joined #openstack-infra02:23
bkeroLet me try that02:23
*** xarses has joined #openstack-infra02:23
greghaynesfungi: element would be great02:23
*** xarses has quit IRC02:23
*** bhunter__ has joined #openstack-infra02:23
*** dosaboy_ has quit IRC02:23
bkerozerofree only works on ext* fs fwiw02:23
fungibkero: afaik we generate raw and then convert to qcow202:23
*** zhurong has quit IRC02:23
greghaynesbkero: yea, but thats what we are creating here02:23
*** stevebaker has quit IRC02:23
*** mkoderer__ has quit IRC02:23
*** sgordon has quit IRC02:23
*** dangers_away has quit IRC02:23
*** therve has quit IRC02:23
*** odyssey4me has quit IRC02:23
*** gtmanfred has quit IRC02:23
*** dosaboy has quit IRC02:23
*** SotK has quit IRC02:23
*** armax has quit IRC02:23
*** yuanying has quit IRC02:23
*** beisner has quit IRC02:23
*** stevemar has quit IRC02:23
*** calebb has quit IRC02:23
*** jokke_ has quit IRC02:23
*** aloga has quit IRC02:23
*** markmcd1 has quit IRC02:23
*** hrybacki has quit IRC02:23
*** alex_xu has quit IRC02:23
*** eliqiao has quit IRC02:23
*** dolpher has quit IRC02:23
*** elmiko has quit IRC02:23
*** doug-fish has quit IRC02:23
*** bhunter_ has quit IRC02:23
*** icey has quit IRC02:23
*** lezbar has quit IRC02:23
*** manfred is now known as gtmanfred02:23
*** mkoderer___ has joined #openstack-infra02:23
*** armax_ is now known as armax02:23
*** calebb1 has joined #openstack-infra02:23
*** dolpher1 is now known as dolpher02:23
*** therve has joined #openstack-infra02:23
fungiso adding a step to zero free space on the filesystem before finalizing the raw ought to do it if this is the culprit02:24
*** beisner has joined #openstack-infra02:24
*** stevemar has joined #openstack-infra02:24
*** zhurong has joined #openstack-infra02:24
*** dosaboy has joined #openstack-infra02:24
fungihowever it's getting late here so this is an exercise better left for tomorrow unless someone else wants to take up the torch02:24
*** doug-fis_ has quit IRC02:24
*** yamahata has quit IRC02:24
greghaynesfungi: yep, and even if thats not the fix I would love that feature in dib :)02:24
bkerosounds like a good dib finalize step02:24
*** hrybacki_ is now known as hrybacki02:24
*** roxanagh_ has joined #openstack-infra02:24
*** xarses has joined #openstack-infra02:24
* greghaynes needs to family so will wait for tomorrow02:25
*** xarses has quit IRC02:25
fungimost of the distros do this by default in their official images already and i used to use the same copy /dev/zero trick then delete when i made my xen base images years ago, so sort of assumed this was already something dib did too02:25
*** jcooley_ has quit IRC02:26
fungiactually surprised to learn it does not02:26
fungiand definitely worth fixing02:26
*** xarses has joined #openstack-infra02:26
greghaynesYea, AFAIK it does not02:26
*** doug-fish has joined #openstack-infra02:27
bswartzguys I actually have a question about images and disk image builder -- hoping to learn from your wisdom02:27
*** xarses has quit IRC02:27
bswartzmanila builds some images -- we have a project called manila-image-elements02:27
*** xarses_ has joined #openstack-infra02:27
bswartzhowever the output from those builds doesn't get saved anywhere official02:27
fungibut yeah, if the difference between the ubuntu and ubuntu-minimal elements is "mount a distro provided image" vs "run debootstrap" then yes they've done most of the cleanup for us in the first scenario02:27
*** xarses_ has quit IRC02:27
bswartzis there an official repository where we could be uploading images?02:28
fungibswartz: closest thing we have to that is
bswartzideally something similar to the tarballs site that just take the output of the build after every commit and saves it?02:28
fungi(it has more than tarballs, but not much more)02:28
*** xarses_ has joined #openstack-infra02:28
*** sarob has quit IRC02:28
bswartzokay how do I figure out how to make a job that uploads disk image builder qcows to the tarballs site?02:29
bswartzis there an example I can follow?02:29
fungibswartz: what size are the images? how many commits do you merge? what do you expect the year-over-year growth would be for storage?02:29
fungiwe're sort of limited since we're stuck using a filesystem on a logical volume spread across cinder devices right now, so max out around 14tb02:30
bswartzwell the project was started 9 months ago and we have 36 commits...
*** sputnik13 has quit IRC02:30
bswartzso not a ton of traffic02:30
bswartzlet me check on size02:30
*** andreww has joined #openstack-infra02:30
fungibut as far as images, for example ironic-python-agent builds disk images it uses in its own testing and stashes them on tarballs.o.o, if you're looking for an example02:31
bswartzlooks like the images we're building weigh in at 300MB currently02:31
bswartzfungi: thanks02:32
bkeroPhew, zerofree takes a while to run.02:32
fungibswartz: though they don't keep them all, they just replace each time
*** yamamoto_ has quit IRC02:33
bkeroSo I created a 10gb qcow2 volume, exported with qemu-nbd, mkfs.ext4'd, mounted, then dd if=/dev/unrandom size=1G to it. The size rose from 133mb (ext metadata) to 1300mb. After running zerofree the size is still 1300mb. If I use qemu-img to convert from qcow2 to qcow2 the resulting image is 896k.02:33
*** xarses_ has quit IRC02:33
bswartzwell I'd like tagged commits to get saved -- similar to how actual tarballs work02:34
bswartzbut ordinary commits could get clobbered02:34
clarkb bkero so zerofree in diblikely would help02:34
bswartzor reaped after some max number02:34
fungibswartz: yep, that's doable... just base the image name off something like $ZUUL_REFNAME and only run the versioned builds in the release pipeline02:34
bkeroclarkb: try this: qemu-img convert -O qcow2 -c oldimage newimage02:34
bkeroSee if the filesize changes02:35
fungiclarkb: yeah, if greghaynes is right that we don't zero deleted garbage in the fs, starting from canonical's image (where they most certainly have done that already) vs running debootstrap (which does a ton of file deletion) will result in significantly different compressed sizes02:36
*** rhallisey has quit IRC02:36
fungibut even freeing in the devstack-trusty images will probably get us back some compressed image size there too02:37
*** eliqiao_ is now known as eliqiao02:37
fungii'm sure some of the elements we're running besides ubuntu-minimal are deleting at least some files they temporarily create02:38
fungiso this needs to be a very generic element run in dib's finalizing steps for pretty much any image it creates ever02:39
bswartzhow many zeros are on these disk images? are they like 10% full of data? 50%? 90%?02:39
fungicall it 25% in the devstack-trusty image and 50% in the ubuntu-trusty image02:40
bkeroclarkb: so I created a 10gb raw image, mkfs.ext4'd it, mounted it, dd'd a random 1gb file to it. Resulting image size was 1.2GB. Did a qemu-img convert to qcow2 on it. Resulting qcow2 was 1001MB. Unmounted the raw image, ran zerofree. Ran a qemu-convert to qcow2 again. resulting size was 896k.02:40
bswartzfor disk images with tons of free space it could be more efficient to "cp -a" the whole filesystem to a new empty filesystem of the same size rather than zeroing the free space02:40
bkeroSo the proof of concept shows that zerofree-ing a raw image, then converting to qcow2 will recover all deleted file space \o/02:40
fungibswartz: agreed, you get more contiguousness that way02:41
fungifragmentation makes zeroing and then compressing less effective02:41
bkeroThat's true02:41
clarkbbkero: you going to write the dib patch? not sure if it makes sense in an element or in dib itself02:41
fungifor some reason i thought dib did that (via an intermediate tarball)02:41
fungibut my information is likely quite outdated/inaccurate02:42
bkeroclarkb: Likely a package-install somewhere, then a finalize02:42
bkeroI can do it02:42
clarkbno dib is raw to its outputs02:42
clarkbwell its fancy chroot to raw to else02:42
fungii may even be remembering some of the preliminary design discussions in this case, from before dib was really a thing02:43
clarkbif you ask for tarball you do chroot to tsrball02:43
*** rhallisey has joined #openstack-infra02:44
*** rhallisey has quit IRC02:44
fungibut bswartz has a great point, we're better off copying the content to a fresh filesystem and then compressing instead of trying to zero the cruft02:44
mnasermaybe im poking too much at things but02:44
*** Shiv_ has joined #openstack-infra02:45
*** gongysh has joined #openstack-infra02:45
*** Shiv_ has left #openstack-infra02:45
mnaseris the queue just idle so it's running at 22 machines only?02:45
bswartzif it's a whole disk image though, you need to worry about duplicating the partition table, boot sector, etc02:45
bkeroThat doesn't sound fun02:45
clarkbmnaser: ya if demand falls use will fall and its proportional to quota02:46
bswartzit's not as bad as you think02:46
clarkbso dib does do the copy already02:46
clarkbdib works by building in a chroot02:46
bswartzdo these disk images have swap partitions by any chance?02:46
*** tphummel has quit IRC02:46
fungioh, it works in a chroot then copies to a loopback at the end?02:47
clarkbthen towards the end it copies everything into a raw disk image loopback mounted02:47
clarkbthen it converts to qcow2 or whatever02:47
greghaynesfungi: bswartz oh, dib does do a cp of the built image, I need to remember where though02:47
greghayneswhich might be why we dont zero the fs02:47
fungihrm, then this may be not the problem02:47
*** resker has quit IRC02:48
greghaynesyea, so it does an mv of the built artifacts right before finalise.d runs02:48
greghaynesSo I think that theory isnt valid :(02:49
bswartzmaybe all the extra space is taken up by the NSA rootkit02:49
* bswartz dons tinfoil hat02:49
fungiwell, the nsa _is_ inefficient, but i don't think even their rootkits take up 5gb02:50
greghaynesit is possible that stuff happens in finalise.d which creates this issue, but I would hope there isn't much going on there02:50
bkerogreghaynes: around line 352 in disk-image-create?02:50
greghaynesbkero: 34802:51
*** esker has joined #openstack-infra02:52
greghaynesso, apt-get clean runs in finalise.d02:52
fungithough we're disabling that i think02:52
bkeroI'm assuming finalise.d is run after things are copied to the image02:52
greghaynesbkero: yea, see line 35002:53
*** sobersabre has quit IRC02:53
fungibecause we set an envvar to tell it not to clear the apt cache02:53
greghaynesfungi: ah, ok02:53
greghaynesMaybe a good test would be to run zerofree on one of the images after the fact and see what happens when converted back to qcow202:53
fungisince building up the apt cache (via --download-only) is one of the main reasons we make out own images02:54
greghaynesjust to rule out that theory or not02:54
fungigreghaynes: yep, i concur, just not me tonight since i don't feel like figuring out the qcow2 conversion command line this late at night02:54
bkeroqemu-nbd -c /dev/nbd0 oldimage.qcow2; kpartx -a /dev/nbd0; zerofree /dev/mapper/nbd0p1; qemu-nbd -d /dev/nbd0; qemu-img convert -O qcow2 -c oldimage.qcow2 newimage.qcow202:54
bkerocompare filesizes02:54
*** gongysh has quit IRC02:54
greghaynesfungi: hah, totally understand that :)02:55
fungithough apparently bkero has it all memorized anyway02:55
bkeroI didn't think those life skills would ever come in handy.02:55
*** camunoz has quit IRC02:55
fungiamuse your friends. fun at parties02:56
bkeroA blast at parties, let me tell you02:56
fungidisk image conversions in between keg stands02:56
greghaynesI need to find better parties02:57
* fungi remembers all the keg stands he built in college. they have to be sturdy to support all that beer, you know02:58
bkeroMaybe I'm failing at life, but I don't remember building any.02:59
clarkbgreghaynes: the only parties you get to go to now involve small children02:59
*** sam_wan has quit IRC03:00
bkeroclarkb: sometimes they involve brisket03:00
fungicorralling toddlers and trying not to look zippy the clown directly in the eyes03:00
* Clint shakes fist03:01
cjhey folks03:01
greghaynesclarkb: wait until the kids learn to convert diskimages03:02
cj <- under "entry_points" here, I'm confused as to what I should put...03:02
cjI'm trying to create a setup.cfg for python-irclib03:02
fungigreghaynes: wait until the kids learn to do keg stands03:02
cjthe first for the 'irccat' script...03:03
* bkero gives up on computer and goes to find a dinner.03:03
cjthere's no main entrypoint there, since it executes the code directly03:03
*** sridhar_ram has quit IRC03:03
fungicj: so no main() you can call into?03:03
cjwhat value do I use in this case?  I don't want pbr to create a new script for me, I just want it to take the one that's there and put it in /usr/local/bin or whatever the prefix is set to03:03
greghaynesfungi: haha, I'm not sure which is scarier03:04
fungimordred: ^ if you're around, do you recall how pbr-ized packages are expected to deal with directly installing executable scripts?03:06
fungiis it as simple as using a file declaration for them, or is more magic required?03:06
*** apoorvad has quit IRC03:07
*** rguillebert has quit IRC03:07
*** camunoz has joined #openstack-infra03:08
*** bpokorny has quit IRC03:08
*** sobersabre has joined #openstack-infra03:08
fungicj: my first guess would be in the [files] section do "datafiles = irccat = bin/irccat" or something along those lines. i'm not immediately finding any really great examples since we tend to rely heavily on entrypoints03:10
*** doug-fish has quit IRC03:10
fungilifeless: it's possible you're more awake than mordred this time of day, perhaps you happen to know the answer to cj's pbr question?03:10
mordredyeah. we pretty much don't do that03:11
mordredI'm sure it's possible some how03:11
mordredbut I'm not really sure why you'd do it03:11
*** achanda has joined #openstack-infra03:11
fungiright, pbr's design expects heavily libbish packages where the only executable scripts are very lightweight wrappers calling into library functions03:12
*** ashleighfarnham has quit IRC03:12
fungiso you're likely stuck relying on whatever raw file copying solutions it has if you need that03:13
fungiand files.data_files is the closest thing that springs to mind03:14
*** thorst has joined #openstack-infra03:16
craigeheya infra root people, is there somewhere I can go to get an eye on the state of
*** sam_wan has joined #openstack-infra03:17
*** fedexo has joined #openstack-infra03:20
*** jswarren_ has joined #openstack-infra03:21
*** MCoLo___ has joined #openstack-infra03:21
*** beisner- has joined #openstack-infra03:21
*** yuanying has joined #openstack-infra03:21
*** beisner has quit IRC03:22
*** beisner- is now known as beisner03:22
*** markmcd1 has joined #openstack-infra03:22
*** mwhahaha_ has joined #openstack-infra03:23
*** jraim_ has joined #openstack-infra03:23
*** kmartin__ has joined #openstack-infra03:23
*** thorst has quit IRC03:24
*** nijaba_ has joined #openstack-infra03:24
*** ddieterl_ has joined #openstack-infra03:24
*** sulo_ has joined #openstack-infra03:24
*** yamamoto_ has joined #openstack-infra03:25
*** bodepd_ has joined #openstack-infra03:25
*** dansmith_ has joined #openstack-infra03:25
*** mikap has joined #openstack-infra03:25
*** f1ller has joined #openstack-infra03:25
*** markmcclain1 has joined #openstack-infra03:25
*** sambetts has joined #openstack-infra03:26
*** jvrbanac_ has joined #openstack-infra03:26
*** dtantsur has joined #openstack-infra03:26
*** mestery_ has joined #openstack-infra03:26
*** NikitaKonovalov2 has joined #openstack-infra03:26
*** bradm_ has joined #openstack-infra03:26
*** Odd_Blok1 has joined #openstack-infra03:26
*** jraim has quit IRC03:26
*** laurelm has quit IRC03:26
*** dansmith has quit IRC03:26
*** MCoLo__ has quit IRC03:26
*** mwhahaha has quit IRC03:26
*** sambetts|afk has quit IRC03:26
*** mrunge has quit IRC03:26
*** igormarnat has quit IRC03:26
*** dtantsur|afk has quit IRC03:26
*** camunoz has quit IRC03:26
*** markmcd__ has quit IRC03:26
*** vponomaryov has quit IRC03:26
*** LinuxJedi has quit IRC03:26
*** jvrbanac has quit IRC03:26
*** sulo has quit IRC03:26
*** markmcclain has quit IRC03:26
*** Odd_Bloke has quit IRC03:26
*** filler has quit IRC03:26
*** jswarren has quit IRC03:26
*** nijaba has quit IRC03:26
*** NikitaKonovalov has quit IRC03:26
*** ZZelle has quit IRC03:26
*** kmartin_ has quit IRC03:26
*** Mmike has quit IRC03:26
*** gibi has quit IRC03:26
*** mnaser has quit IRC03:26
*** vdrok_ has quit IRC03:26
*** clayton has quit IRC03:26
*** dolpher has quit IRC03:26
*** IlyaG has quit IRC03:26
*** yuanying_ has quit IRC03:26
*** ddieterly has quit IRC03:26
*** msuriar has quit IRC03:26
*** ildikov has quit IRC03:26
*** lxsli has quit IRC03:26
*** lennyb has quit IRC03:26
*** ajo has quit IRC03:26
*** HenryG has quit IRC03:26
*** hamzy has quit IRC03:26
*** Roamer` has quit IRC03:26
*** mika has quit IRC03:26
*** bodepd has quit IRC03:26
*** briancline has quit IRC03:26
*** IlyaG has joined #openstack-infra03:26
*** Roamer` has joined #openstack-infra03:26
*** leifmadsen has quit IRC03:26
*** zaro has quit IRC03:26
*** hamzy has joined #openstack-infra03:26
*** mestery has quit IRC03:26
*** briancline has joined #openstack-infra03:26
*** bradm has quit IRC03:26
*** mrunge_ has joined #openstack-infra03:26
*** igormarnat_ has joined #openstack-infra03:26
*** yamahata has joined #openstack-infra03:26
*** NikitaKonovalov2 is now known as NikitaKonovalov03:26
*** mikap is now known as mika03:26
*** LinuxJedi_ has joined #openstack-infra03:26
*** markmcclain1 is now known as markmcclain03:26
*** HenryG has joined #openstack-infra03:26
*** msuriar has joined #openstack-infra03:26
*** igormarnat_ is now known as igormarnat03:26
*** ajo has joined #openstack-infra03:26
*** jraim_ is now known as jraim03:26
*** vdrok_ has joined #openstack-infra03:27
*** LinuxJedi_ is now known as LinuxJedi03:27
*** dansmith_ is now known as dansmith03:27
*** ildikov has joined #openstack-infra03:27
*** woodster_ has quit IRC03:27
*** vponomaryov has joined #openstack-infra03:27
*** camunoz has joined #openstack-infra03:27
*** laurelm has joined #openstack-infra03:27
*** mwhahaha_ is now known as mwhahaha03:27
*** lennyb has joined #openstack-infra03:27
*** Mmike has joined #openstack-infra03:27
*** ZZelle has joined #openstack-infra03:27
*** dolpher has joined #openstack-infra03:27
*** dansmith is now known as Guest6049703:27
*** leifmadsen has joined #openstack-infra03:27
*** clayton has joined #openstack-infra03:28
*** mestery_ is now known as mestery03:28
*** gibi has joined #openstack-infra03:28
openstackgerritTony Breeds proposed openstack-infra/infra-specs: Create
*** links has joined #openstack-infra03:29
*** mnaser has joined #openstack-infra03:30
*** ansiwen has quit IRC03:31
*** ansiwen has joined #openstack-infra03:31
*** christiana has quit IRC03:31
*** aeng has joined #openstack-infra03:34
*** lennyb__ has joined #openstack-infra03:35
*** tojuvone_ has joined #openstack-infra03:36
*** pfallenop has quit IRC03:36
*** kzaitsev_mb has joined #openstack-infra03:37
*** [1]Thelo has joined #openstack-infra03:39
*** mrunge_ has quit IRC03:39
*** lennyb has quit IRC03:40
*** tojuvone has quit IRC03:40
*** Thelo has quit IRC03:41
*** [1]Thelo is now known as Thelo03:41
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** pfallenop has joined #openstack-infra03:44
*** baoli has joined #openstack-infra03:45
*** mrunge has joined #openstack-infra03:45
*** baoli has quit IRC03:47
*** dims has quit IRC03:48
openstackgerritBob Haddleton proposed openstack/requirements: Add heat-translator>=0.4.0
*** bradm_ is now known as bradm03:49
*** keedya has joined #openstack-infra03:51
*** lxsli has joined #openstack-infra03:52
*** eric_lopez has quit IRC03:56
*** eric_lopez has joined #openstack-infra03:56
*** camunoz has quit IRC03:58
mnaserinteresting metric of seeing how going from 1 to 25 servers brings the list servers response time from ~0.25s to ~0.5s03:59
*** doug-fish has joined #openstack-infra04:00
*** doug-fish has quit IRC04:05
*** kzaitsev_mb has quit IRC04:05
*** camunoz has joined #openstack-infra04:11
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** Mmike has quit IRC04:15
*** Mmike has joined #openstack-infra04:15
*** mriedem has quit IRC04:16
*** IlyaG has quit IRC04:17
*** sam_wan has quit IRC04:18
*** amrith is now known as _amrith_04:19
*** woodster_ has joined #openstack-infra04:20
*** marg7175_ has quit IRC04:21
*** thorst has joined #openstack-infra04:21
*** sam_wan has joined #openstack-infra04:22
*** harlowja_at_home has quit IRC04:27
*** Jeffrey4l has joined #openstack-infra04:27
*** achanda has quit IRC04:28
*** Daisy has joined #openstack-infra04:28
*** thorst has quit IRC04:29
*** bhunter__ has quit IRC04:30
*** tiswanso has quit IRC04:33
*** Daisy has quit IRC04:34
*** Daisy has joined #openstack-infra04:34
*** denisra has quit IRC04:38
*** denisra has joined #openstack-infra04:38
*** yonglihe has quit IRC04:42
*** roxanagh_ has quit IRC04:44
*** resker has joined #openstack-infra04:45
*** esker has quit IRC04:47
*** esikache1 has joined #openstack-infra04:51
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** rossella_s has quit IRC04:53
*** rossella_s has joined #openstack-infra04:54
*** esikache1 has quit IRC04:56
*** Jeffrey4l has quit IRC04:59
*** kzaitsev_mb has joined #openstack-infra05:02
*** jamespage has quit IRC05:05
*** gnuoy has quit IRC05:05
*** gnuoy has joined #openstack-infra05:06
*** jamespage has joined #openstack-infra05:06
*** kzaitsev_mb has quit IRC05:07
lifelessfungi: what was cj's pbr question ?05:10
lifelesscj: ^05:10
*** achanda has joined #openstack-infra05:11
*** sridhar_ram has joined #openstack-infra05:13
*** eliqiao_ has joined #openstack-infra05:13
*** eliqiao has quit IRC05:16
*** alex_xu has joined #openstack-infra05:18
*** salv-orl_ has joined #openstack-infra05:19
*** roxanaghe has joined #openstack-infra05:20
*** maishsk has joined #openstack-infra05:20
*** alex_xu_ has quit IRC05:21
*** oanson has joined #openstack-infra05:21
fungilifeless: he wanted to know if pbr had a mechanism for directly installing an executable script rather than relying on an entrypoint05:22
*** salv-orlando has quit IRC05:22
lifelesswe do; its just he underlying setuptools stuff.05:23
lifelessDon't use it.05:23
*** zaro has joined #openstack-infra05:25
*** thorst has joined #openstack-infra05:27
*** claudiub|2 has joined #openstack-infra05:28
*** noorul has joined #openstack-infra05:28
openstackgerritFrank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite
*** sarob has joined #openstack-infra05:29
*** Douhet has quit IRC05:31
*** sam_wan has quit IRC05:31
*** Douhet has joined #openstack-infra05:32
*** thorst has quit IRC05:33
*** sarob has quit IRC05:33
*** claudiub|2 has quit IRC05:34
*** kiran-r has joined #openstack-infra05:37
*** maishsk has quit IRC05:39
*** sobersabre has quit IRC05:43
*** Daisy has quit IRC05:44
*** Daisy has joined #openstack-infra05:44
*** abregman has joined #openstack-infra05:46
*** Daisy has quit IRC05:46
*** Daisy has joined #openstack-infra05:47
*** sarob has joined #openstack-infra05:50
*** alex_xu_ has joined #openstack-infra05:50
*** _sarob has joined #openstack-infra05:51
*** Daisy has quit IRC05:51
*** Daisy has joined #openstack-infra05:52
*** alex_xu has quit IRC05:53
*** sarob has quit IRC05:54
*** Daisy has quit IRC05:55
*** Daisy has joined #openstack-infra05:55
*** Daisy has quit IRC05:56
*** Daisy has joined #openstack-infra05:56
*** gildub has quit IRC05:56
*** gildub has joined #openstack-infra05:57
*** baoli has joined #openstack-infra05:59
*** sdake has joined #openstack-infra05:59
*** resker has quit IRC06:01
prometheanfirecould someone tell me why this isn't getting workflowed?
prometheanfireSpamapS seems to be absent06:02
*** kzaitsev_mb has joined #openstack-infra06:03
*** baoli has quit IRC06:04
*** esker has joined #openstack-infra06:05
greghaynesprometheanfire: done06:05
greghaynesprometheanfire: he is on leave ATM06:05
prometheanfireah, thanks06:06
prometheanfireat the time it was behind another, dependant commit06:07
openstackgerritMerged openstack/diskimage-builder: Add Gentoo to the list of supported distributions
*** armax has quit IRC06:07
prometheanfirehuh, that was quick06:08
greghaynesYea, for doc-only changes we short-circuit some of the testing06:08
*** kzaitsev_mb has quit IRC06:09
prometheanfirewish we did that in rpc-openstack06:10
prometheanfirewaiting 1.5hr for jenkins to fail isn't fun06:10
*** gongysh has joined #openstack-infra06:10
*** sam_wan has joined #openstack-infra06:13
*** yonglihe has joined #openstack-infra06:14
*** sdake has quit IRC06:16
*** Thelo has quit IRC06:23
*** woodster_ has quit IRC06:27
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** thorst has joined #openstack-infra06:31
*** gongysh has quit IRC06:32
*** amit213 has quit IRC06:33
*** jtomasek has joined #openstack-infra06:33
*** x00350071 is now known as xiangxinyong06:35
*** sridhar_ram has quit IRC06:35
*** cfarquhar has quit IRC06:37
*** eliqiao_ is now known as eliqiao06:39
*** thorst has quit IRC06:39
*** cfarquhar has joined #openstack-infra06:40
*** cfarquhar has quit IRC06:40
*** cfarquhar has joined #openstack-infra06:40
*** tojuvone_ is now known as tojuvone06:44
*** gildub has quit IRC06:53
*** gildub has joined #openstack-infra06:53
*** roxanaghe has quit IRC06:53
*** esikache1 has joined #openstack-infra06:54
*** camunoz has quit IRC06:54
*** roxanaghe has joined #openstack-infra06:55
*** esikache1 has quit IRC06:58
*** esikachev has joined #openstack-infra06:59
*** aeng has quit IRC07:00
*** sam_wan has quit IRC07:02
*** gnuoy_ has joined #openstack-infra07:03
*** rlandy has quit IRC07:04
*** amit213 has joined #openstack-infra07:04
*** kzaitsev_mb has joined #openstack-infra07:04
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP]
*** gnuoy_ has quit IRC07:06
*** amit213 has quit IRC07:06
*** amit213 has joined #openstack-infra07:08
*** kzaitsev_mb has quit IRC07:09
*** amit213 has quit IRC07:10
*** Daisy has quit IRC07:13
*** Daisy has joined #openstack-infra07:13
*** jaosorior has joined #openstack-infra07:15
*** amit213 has joined #openstack-infra07:15
*** Daisy has quit IRC07:18
*** amit213 has quit IRC07:18
*** scheuran has joined #openstack-infra07:20
openstackgerritMerged openstack-infra/project-config: midonet: Rename rally job so that it matches regex for doc-only changes
AJaegerNakato: are you still around? Problem with post jobs and constraints is a limitation of zuul-cloner, zuul-cloner will not setup the repository correctly. We need to enhance zuul-cloner first07:24
*** nmagnezi has joined #openstack-infra07:26
*** claudiub|2 has joined #openstack-infra07:26
openstackgerritMerged openstack-infra/project-config: Enable pep8 job on tripleo-heat-templates
openstackgerritMerged openstack-infra/project-config: Add freezer-api job for tempest tests
openstackgerritMerged openstack-infra/project-config: Add total singlestats for nodepool dashboard
*** claudiub|2 has quit IRC07:28
openstackgerritMerged openstack-infra/project-config: Make Packstack integration jobs voting, add to gate
*** achanda has quit IRC07:31
*** zhurong has quit IRC07:32
*** zhurong has joined #openstack-infra07:34
*** fedexo has quit IRC07:34
*** boris-42 has quit IRC07:34
openstackgerritSergey Kraynev proposed openstack/requirements: Bump upper-constraint for heatclient to 1.0.0
*** boris-42 has joined #openstack-infra07:35
eliqiaohello, does anyone know why my in PYTHON, /usr/lib appears before /usr/local/lib ?07:35
openstackgerrityolanda.robla proposed openstack-infra/system-config: Only run jjb if that is not already running
*** korzen has joined #openstack-infra07:37
*** thorst has joined #openstack-infra07:37
*** maishsk has joined #openstack-infra07:39
*** amit213 has joined #openstack-infra07:40
openstackgerritCsaba Henk proposed openstack-infra/project-config: Add Manila with GlusterFS driver and Heketi layout  CI job
*** amit213 has quit IRC07:42
*** jcoufal has joined #openstack-infra07:42
*** thorst has quit IRC07:43
*** andymaier has joined #openstack-infra07:44
*** _sarob has quit IRC07:45
*** pcaruana has joined #openstack-infra07:45
*** k4n0 has joined #openstack-infra07:46
*** roxanaghe has quit IRC07:46
openstackgerritMerged openstack-infra/project-config: Add openstack client command overlap checking
*** k4n0 has quit IRC07:48
*** achanda has joined #openstack-infra07:49
*** kushal has joined #openstack-infra07:53
*** mrmartin has joined #openstack-infra07:53
*** e0ne has joined #openstack-infra07:55
*** e0ne has quit IRC07:56
*** jed56 has joined #openstack-infra07:57
*** mikelk has joined #openstack-infra07:59
*** resker has joined #openstack-infra08:01
*** esker has quit IRC08:01
*** eliqiao has quit IRC08:01
*** sorantis has joined #openstack-infra08:01
*** amotoki has quit IRC08:02
*** markvoelker has quit IRC08:03
*** kzaitsev_mb has joined #openstack-infra08:05
openstackgerritMerged openstack/requirements: Bump upper constraint on python-novaclient to 3.3.0
*** eliqiao has joined #openstack-infra08:07
*** amit213 has joined #openstack-infra08:09
*** andymaier has quit IRC08:09
*** kzaitsev_mb has quit IRC08:10
*** amit213 has quit IRC08:13
*** [HeOS] has joined #openstack-infra08:15
*** gildub has quit IRC08:15
*** _nadya_ has joined #openstack-infra08:16
*** amit213 has joined #openstack-infra08:16
*** claudiub|2 has joined #openstack-infra08:16
*** mrmartin has quit IRC08:18
*** sflanigan has quit IRC08:18
*** amit213 has quit IRC08:18
*** HeOS has quit IRC08:19
*** yaume has joined #openstack-infra08:20
claudiub|2AJaeger: hello. Do you know if the pypi jobs were down last night? I pushed a signed tag to gerrit, it exists on, it exists on github, but not on pypi. project name: os-win08:22
*** sobersabre has joined #openstack-infra08:22
sobersabrehi. Is there a way to retrigger CI on a patch?08:22
*** mrmartin has joined #openstack-infra08:22
claudiub|2sobersabre: recheck08:23
*** esikachev has quit IRC08:23
claudiub|2sobersabre: or if it's a 3rd party ci, it usually says how to in the comment it posts.08:23
*** amit213 has joined #openstack-infra08:24
*** andymaier has joined #openstack-infra08:25
openstackgerritSteve Martinelli proposed openstack-infra/project-config: do not propose a new keystone config file if already approved
*** vincentll has joined #openstack-infra08:29
openstackgerritSteve Martinelli proposed openstack-infra/project-config: do not propose a new keystone config file if already approved
sobersabreclaudiub|2: thanks, I added "recheck". How long does it usually take until there is something I can identify that CI picked this up ?08:32
*** ihrachys has joined #openstack-infra08:32
*** achanda has quit IRC08:32
claudiub|2sobersabre: if it's jenkins, check
*** dingyichen has quit IRC08:32
openstackgerritMerged openstack-infra/project-config: Create DVR multinode grenade job for Neutron upgrade tests
*** esikachev has joined #openstack-infra08:33
claudiub|2sobersabre: or this: you'll see all the jobs currently in queue.08:34
*** matrohon has joined #openstack-infra08:34
*** eliqiao_ has joined #openstack-infra08:35
*** dolpher has quit IRC08:35
*** eliqiao has quit IRC08:35
*** e0ne has joined #openstack-infra08:36
AJaegerclaudiub|2: run git show-ref YOUR_TAG and check the log files on logs.openstack.org08:37
*** shardy has joined #openstack-infra08:37
AJaegerIf your tag is a0b... look at for it08:37
*** jlanoux has joined #openstack-infra08:38
*** e0ne has quit IRC08:38
claudiub|2ok thanks. :)08:38
*** arxcruz has joined #openstack-infra08:40
*** vgridnev has joined #openstack-infra08:40
*** thorst has joined #openstack-infra08:41
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Setup translation for networking-midonet
*** sorantis has quit IRC08:49
*** thorst has quit IRC08:49
AJaegeryolanda, system-config cores: for mitaka the German translators need a place to discuss and want an email list, could you review , please?08:50
AJaegerclaudiub|2: What do the logs say? Where's the problem?08:51
claudiub|2AJaeger: honestly, can't find them.. not really sure where to look. the pushed tag is 0.2.3:  git show-ref 0.2.3  ->   800d261a12fde9cdd6c5cedc150aea8965da7752 refs/tags/0.2.308:52
claudiub|2oh, i see.08:53
AJaegerLooking at the logfile, it looks like a problem on pypi - ask an infra-root to reupload, please08:53
claudiub|2ok, sure. thanks for the help. I appreciate it. :)08:54
*** vgridnev has quit IRC08:54
*** vgridnev has joined #openstack-infra08:55
claudiub|2fungi: hello08:55
AJaegerclaudiub|2: please don't wake him ;) He should be around 15:00 UTC08:56
claudiub|2ok. :D08:57
*** hashar has joined #openstack-infra09:00
*** BobBall_AWOL is now known as BobBall09:01
*** resker has quit IRC09:01
*** fhubik has joined #openstack-infra09:02
*** andymaier has quit IRC09:03
*** markvoelker has joined #openstack-infra09:03
*** kushal has quit IRC09:05
*** esker has joined #openstack-infra09:05
*** kzaitsev_mb has joined #openstack-infra09:06
sobersabreclaudiub|2: thanks, I got it, and the builds passed ok.09:06
*** eliqiao_ is now known as eliqiao09:07
*** rraja has joined #openstack-infra09:10
*** kzaitsev_mb has quit IRC09:11
*** asettle has joined #openstack-infra09:11
openstackgerritMax V.K. proposed openstack-infra/jenkins-job-builder: upd google-cloud-storage plugin support
*** fhubik is now known as fhubik_brb09:12
*** jordanP has joined #openstack-infra09:14
*** andymaier has joined #openstack-infra09:15
claudiub|2sobersabre: no problem. :)09:15
*** jistr has joined #openstack-infra09:17
*** lucas-dinner is now known as lucasagomes09:17
*** daemontool_ has joined #openstack-infra09:19
*** zhurong has quit IRC09:22
*** kzaitsev_mb has joined #openstack-infra09:24
*** e0ne has joined #openstack-infra09:24
*** Daisy has joined #openstack-infra09:26
*** Daisy has quit IRC09:27
*** Daisy has joined #openstack-infra09:28
*** Daisy has quit IRC09:29
*** Daisy has joined #openstack-infra09:29
*** Daisy has quit IRC09:30
*** Daisy has joined #openstack-infra09:30
*** mrmartin has quit IRC09:31
abregmanAJaeger: hi. can you take a look?:
*** Daisy has quit IRC09:32
*** sorantis has joined #openstack-infra09:32
*** Daisy has joined #openstack-infra09:33
*** jokke__ is now known as jokke_09:33
*** Daisy has quit IRC09:33
*** Daisy has joined #openstack-infra09:33
*** Daisy has quit IRC09:35
*** Daisy has joined #openstack-infra09:36
*** Daisy has quit IRC09:36
*** Daisy has joined #openstack-infra09:37
ttxJob at top of gate stuck for the last 2 hours09:41
ttxAJaeger: any idea what the current job timeout is ?09:43
*** gongysh has joined #openstack-infra09:44
AJaegerttx, let me check...09:46
AJaegerttx: 180 mins09:46
openstackgerritJames Page proposed openstack-infra/project-config: Update groups for charm projects
*** thorst has joined #openstack-infra09:46
ttxoh well, I guess it's simpler to just wait for it to hit then09:47
AJaegerabregman: why is that job run with tempest?09:48
abregmanAJaeger: what do you mean? why not?09:48
AJaegerI mean part of devstack - it should test neutron, shouldn't it?09:49
AJaegerI'll give it a -1 on naming - jobs start with gate. And sdague should review further...09:50
*** kushal has joined #openstack-infra09:51
*** openstackgerrit_ has joined #openstack-infra09:53
*** thorst has quit IRC09:54
abregmanAJaeger: not sure I'm following. "we do not use experimental"?  there are other jobs there that using it. also, the job is not testing neutron, it installs it instead of using nova network09:58
AJaegerttx, job timed out finally ;(09:59
AJaegerabregman: the name should be "gate-XXX" not "experimental-XXX"09:59
abregmanAJaeger: why? this is an experimental job09:59
AJaegerabregman: a convention we have.09:59
AJaegerabregman: you will move it to check/gate later and should not need to rename it...10:00
*** e0ne has quit IRC10:00
abregmanAJaeger: but look on line 1273, 1274 here:
*** kukacz has joined #openstack-infra10:01
abregmanAJaeger: it's also using experimental. is it a mistake?10:01
*** kzaitsev_mb has quit IRC10:01
*** Odd_Blok1 is now known as Odd_Bloke10:02
*** openstackgerrit_ has quit IRC10:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename experimental-kolla-dsvm-deploy-multinode-ubuntu-source
*** Odd_Bloke has quit IRC10:02
*** Odd_Bloke has joined #openstack-infra10:02
AJaegerabregman: yes, a mistake10:02
*** Jeffrey4l has joined #openstack-infra10:03
*** _degorenko|afk is now known as degorenko10:03
*** boris-42 has quit IRC10:04
*** gildub has joined #openstack-infra10:04
openstackgerritQiming Teng proposed openstack/requirements: Bump python-senlinclient version to 0.3.0
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Rename experimental jobs
AJaegerabregman: fixed ^10:06
*** rguillebert has joined #openstack-infra10:06
*** Daisy has quit IRC10:06
*** Daisy has joined #openstack-infra10:07
*** odyssey4me_ is now known as odyssey4me10:07
openstackgerritabregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron
*** fhubik_brb is now known as fhubik10:09
AJaegerabregman: change looks fine on syntax/convention. But I don't understand the reason why this is needed and where this should go, so adding some more explanation to it would be great.10:09
*** mrmartin has joined #openstack-infra10:09
abregmanAJaeger: ack. adding.10:10
*** e0ne has joined #openstack-infra10:10
*** dizquierdo has joined #openstack-infra10:10
*** Daisy has quit IRC10:11
*** Qiming has quit IRC10:11
AJaegerabregman: I think I'm getting what you want finally and thus adding to devstack is proper way. Just one more question: Why do we need both centos7 and centos7-neutron? Shouldn't one be enough?10:12
abregmanAJaeger: when centos7-network will be stable, I'll make sure to remove centos7. At this point it might be buggy. we'll have to see.10:13
*** sfinucan has joined #openstack-infra10:13
AJaegerabregman: then mention that plan in the commit message, please10:14
AJaegerThat's a perfectly fine way forward and with that information I'm happy to +2 ;)10:14
AJaegerAh, I see ianw commenting as well...10:15
ianwAJaeger: yeah, sorry, I asked abregman to look at this :)10:15
AJaegerabregman: I'll wait for jenkins tests and then +2A... The comments give enough information10:15
*** amotoki has joined #openstack-infra10:16
*** amotoki has quit IRC10:16
*** jordan_ has joined #openstack-infra10:16
AJaegerianw: which is perfectly fine - just the information why this was needed and what your plans are belong into a commit message.10:16
openstackgerritabregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron
AJaegerI can see what is done but I need to understand the why sometimes as well ;)10:16
abregmanAJaeger: fair enough :) extended commit message.10:17
AJaegerThanks, abregman for the update!10:17
abregmanthank you :)10:17
*** jordanP has quit IRC10:20
*** abregman is now known as abregman|afk10:22
*** SotK_ is now known as SotK10:22
*** sulo_ is now known as sulo10:22
*** |-paul-| has joined #openstack-infra10:24
wznoinskhi all10:26
wznoinskanteaya, are you around?10:26
wznoinskactually that's a general infra question I have here...10:27
wznoinskdo you anticipate moving CI/jenkins to Kolla in next some time?10:27
openstackgerritMerged openstack-infra/project-config: Add new expermintal job - centos with neutron
*** sgordon` is now known as sgordon10:37
*** Daisy has joined #openstack-infra10:38
*** sfinucan has quit IRC10:40
*** Daisy has quit IRC10:42
*** Daisy has joined #openstack-infra10:42
*** Daisy has quit IRC10:47
yolandahi AJaeger, i've been out all morning, reading the backlog now10:50
*** thorst has joined #openstack-infra10:51
dguitarbiteAJaeger: I realized that the package "zip" is missing on the infra hosts10:53
dguitarbiteAJaeger: Is there anything I can do to add/install "zip" to create zip files?10:53
*** tpsilva has joined #openstack-infra10:55
*** mrmartin has quit IRC10:56
*** gongysh has quit IRC10:57
*** thorst has quit IRC10:59
*** doug-fish has joined #openstack-infra11:00
*** kushal has quit IRC11:00
*** alex_xu_ has quit IRC11:01
*** abregman|afk is now known as abregman11:01
*** salv-orl_ has quit IRC11:01
*** salv-orlando has joined #openstack-infra11:02
*** alex_xu has joined #openstack-infra11:02
*** kzaitsev_mb has joined #openstack-infra11:02
*** christiana has joined #openstack-infra11:02
*** fhubik is now known as fhubik_brb11:02
*** jed56 has quit IRC11:03
*** doug-fish has quit IRC11:05
*** ociuhandu has joined #openstack-infra11:06
*** lezbar__ has quit IRC11:06
*** lezbar has joined #openstack-infra11:07
*** fhubik_brb is now known as fhubik11:07
*** rraja has quit IRC11:08
claudiub|2yolanda: hello. I need a bit of help. I've pushed the tag 0.2.3 for os-win last night and the pypi jobs failed. the tag exists on github, as well as Logs:
*** links has quit IRC11:08
yolandahi claudiub|2 i'll try to help11:08
*** rraja has joined #openstack-infra11:08
claudiub|2yolanda: cool, thanks! let me know how it goes. :)11:09
yolandai'll take a loko11:09
*** kzaitsev_mb has quit IRC11:10
*** dmsimard has quit IRC11:10
*** links has joined #openstack-infra11:10
*** jpr has joined #openstack-infra11:12
*** dims has joined #openstack-infra11:14
*** hashar has quit IRC11:19
*** salv-orl_ has joined #openstack-infra11:19
jamespageAJaeger, yolanda: after some digging into the update-bug command in jeepyb I think will target the right bug tasks for the charms...11:19
jamespageexample of API -
openstackLaunchpad bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] - Assigned to MFauzilkamil Zainuddin (apogee)11:20
*** ikalnitsky has left #openstack-infra11:21
*** salv-orlando has quit IRC11:22
openstackgerritSergey Nikitin proposed openstack-infra/project-config: Move job gate-tempest-dsvm-nova-wsgi-full to the check list
yolandajamespage checking, i didn't have knowledge about that11:26
*** dims has quit IRC11:26
jamespageyolanda, took me a while to figure out digging about in jeepyb11:26
yolandafor sure, groups is not the best name for that setting11:27
*** dims has joined #openstack-infra11:27
*** kashyap has joined #openstack-infra11:28
*** jed56 has joined #openstack-infra11:28
kashyapAJaeger: Hi, any tool to locally validate rST (I'm using `rstcheck` but it isn't catching invalid links)11:29
*** gildub has quit IRC11:31
*** keedya has quit IRC11:32
yolandaAJaeger, do you know the procedure to retry a pypi upload? i tried to rebuild the job on jenkins but i don't have that option11:36
yolandawhat i usually do downstream, is that there is a rebuild with parameters option so you can do manually on jenkins, passed the parameters you need, but i don't see this option upstream11:36
*** electrofelix has joined #openstack-infra11:37
openstackgerritKashyap Chamarthy proposed openstack-infra/infra-manual: creators.rst: Fix broken internel hyperlinks
AJaegeryolanda: no worries, nothing urgent here ;)11:38
*** kzaitsev_mb has joined #openstack-infra11:38
AJaegeryolanda: I can't help with the pypi upload. I think fungi is just running the job again but I'm not sure11:39
*** yamahata has quit IRC11:39
AJaegerkashyap: there's doc8 that can catch some issues and sphinx has a linkcheck option11:39
kashyapAJaeger: Thank you.  I clearly failed in rST test, eh11:40
yolandai tried that but it needs to be run with the same parameter, and i miss that option in jenkins.openstack11:40
AJaegerdguitarbite: no zip? I thought that was on the images - better ask the experts later today.11:40
*** fhubik is now known as fhubik_brb11:40
*** kushal has joined #openstack-infra11:41
AJaegerdguitarbite: do you have a log file to look at?11:41
AJaegeryolanda: then best ask fungi later11:41
*** tqtran has quit IRC11:42
*** sorantis has quit IRC11:42
yolandai'll do11:42
AJaegerdguitarbite: jenkins/data/bindep-fallback.txt in project-config includes zip - so that means it should be there...11:46
AJaeger(note that list is not used for the image your job run, just derived from it)11:47
AJaegerdguitarbite: has a master tarball - isn't that what you need?11:49
*** sdague has joined #openstack-infra11:50
*** gildub has joined #openstack-infra11:50
*** snikitin has joined #openstack-infra11:51
*** rfolco has joined #openstack-infra11:52
openstackgerritMerged openstack-infra/project-config: Change ACL for fuel projects
openstackgerritMerged openstack-infra/project-config: Dedicated specs repo for Freezer
*** asettle has quit IRC11:55
*** kdas__ has joined #openstack-infra11:55
*** kushal has quit IRC11:56
*** kdas__ has quit IRC11:56
*** kushal has joined #openstack-infra11:56
*** thorst has joined #openstack-infra11:56
openstackgerritSergey Nikitin proposed openstack-infra/project-config: Move job gate-tempest-dsvm-nova-wsgi-full to the check list
*** csaba has joined #openstack-infra11:59
csabaAJaeger: ping12:01
csabaAJaeger: can you please merge for me?12:01
*** ldnunes has joined #openstack-infra12:02
*** openstackgerrit has quit IRC12:03
*** openstackgerrit has joined #openstack-infra12:03
*** thorst has quit IRC12:04
*** aysyd has joined #openstack-infra12:04
*** gildub has quit IRC12:06
*** aysyd has quit IRC12:09
*** _amrith_ is now known as amrith12:09
*** aysyd has joined #openstack-infra12:10
*** esikachev has quit IRC12:11
openstackgerritErno Kuvaja proposed openstack/requirements: Bump python-glanceclient to 2.0.0
*** |-paul-| has quit IRC12:17
*** rhallisey has joined #openstack-infra12:17
*** esikachev has joined #openstack-infra12:17
AJaegercsaba: please explain what your change is and why it's important. I'll review later...12:18
*** rhallisey has quit IRC12:18
*** rhallisey has joined #openstack-infra12:18
*** doug-fish has joined #openstack-infra12:19
*** sorantis has joined #openstack-infra12:21
*** sshnaidm has quit IRC12:30
*** doug-fish has quit IRC12:31
*** NobodyCam has quit IRC12:31
*** NobodyCam has joined #openstack-infra12:31
*** fhubik_brb is now known as fhubik12:32
*** kushal has quit IRC12:33
*** doug-fish has joined #openstack-infra12:34
*** sshnaidm has joined #openstack-infra12:35
openstackgerritMerged openstack/diskimage-builder: Replace sfdisk partitioning with parted
*** thorst has joined #openstack-infra12:37
*** yamamoto_ has quit IRC12:38
*** jaosorior has quit IRC12:39
*** jaosorior has joined #openstack-infra12:39
*** jaosorior has quit IRC12:39
*** esikachev has quit IRC12:39
*** jaosorior has joined #openstack-infra12:40
*** dprince has joined #openstack-infra12:40
*** gordc has joined #openstack-infra12:41
AJaegersdague: is a new review that adds a job for nova check queue12:43
dguitarbiteAJaeger: Yes, but the zip command was failing in the jenkins jobs so I disabled it for the time being. Ill see if I can find the jenkins failures12:43
AJaegerdguitarbite: Ah, I see. Best ask fungi or clarkb later about this and reference a log file showing the failure.12:44
*** dprince has quit IRC12:46
openstackgerritCsaba Henk proposed openstack-infra/project-config: Add jobs for Manila glusterfs, glusterfs-native drivers with Heketi layout
dguitarbiteok, thanks :) Finally I can rest in peace as the tar balls are automatically hosted :D12:46
csabaAJaeger: I submitted a new driver mode to Manila's glusterfs drivers which is eligible to merge only if it gets gateway tested. I'm adding the testcases which do that. (The particular urgency that made me venture to bug you with it is that Manila m3 is to be tagged today.)12:48
openstackgerritEmilien Macchi proposed openstack-infra/project-config: add scenario002 jobs for puppet-cinder
flaper87Do jobs in the gate use `` scripts? Or do they just use tox for individual projects? I'd expect the latter, AFAIK12:50
*** dprince has joined #openstack-infra12:53
*** rossella_s has quit IRC12:54
*** rossella_s has joined #openstack-infra12:54
*** _nadya_ has quit IRC12:55
*** _nadya_ has joined #openstack-infra12:55
*** thiagop has joined #openstack-infra12:56
AJaegercsaba: PLesae get a +1 by manila folks on that change first.12:56
AJaegercsaba: btw I cannot merge a change by myself - we always use two core reviewers...12:57
AJaegerflaper87: give me an example job name...12:57
AJaegerI expect it will use jenkins/scripts/ which runs tox -e ... - and looking at the log file should show that as well12:58
*** jpr has quit IRC12:58
openstackgerritabregman proposed openstack-infra/devstack-gate: DO NOT MERGE - Testing Gates
flaper87AJaeger: I'm interested in Glance's jobs in this case. I'd like to get rid of the old run_tests script, which none of us is using but I'm not sure if it's being used in the gate (which I doubt)12:59
*** _nadya_ has quit IRC13:00
EmilienMfungi: hello! we found out makes Puppet OpenStack CI jobs failing quite often13:00
EmilienMfungi: because SYS_UID_MIN is missing and in our case, rabbitmq is stealing the UID of keystone13:00
EmilienMfungi: and boom.13:01
AJaegerflaper87: a quick grep shows no occurence in project-config. Remove it - your tests should fail if my grep is wrong13:01
AJaegerEmilienM: good catch13:01
*** apevec has joined #openstack-infra13:02
flaper87AJaeger: did it, waiting for the gate! Thanks for looking into it13:02
EmilienMAJaeger: well, I think there is a reason of such a patch, so before sending a patch, I want to discuss with fungi / pabelanger about that13:02
AJaegerEmilienM: sure ;)13:02
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP]
apevecyeah, I'd like to understand the issue about "Declare a non-overlapping dynamic UID/GID range"13:03
*** baoli has joined #openstack-infra13:03
apevecand why replacing the whole file...13:03
*** maishsk has quit IRC13:05
openstackgerritZara proposed openstack-infra/storyboard-webclient: Move search icon
*** noorul has quit IRC13:07
*** fhubik has quit IRC13:08
csabaAJaeger: thanks, I'm to do that.13:08
*** dkranz has joined #openstack-infra13:08
*** placko has joined #openstack-infra13:09
*** amrith is now known as _amrith_13:10
openstackgerritPurandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP]
openstackgerritZara proposed openstack-infra/storyboard-webclient: Move search icon
*** markvoelker has quit IRC13:11
*** markvoelker has joined #openstack-infra13:11
openstackgerritZara proposed openstack-infra/storyboard-webclient: Move search icon
*** pradk has joined #openstack-infra13:13
*** sfinucan has joined #openstack-infra13:13
*** dmsimard has joined #openstack-infra13:14
*** sdake has joined #openstack-infra13:14
*** gongysh has joined #openstack-infra13:14
*** kgiusti has joined #openstack-infra13:15
openstackgerritCsaba Henk proposed openstack-infra/project-config: Add jobs for Manila glusterfs, glusterfs-native drivers with Heketi layout
*** dizquierdo has quit IRC13:18
*** _oanson has joined #openstack-infra13:19
*** gongysh has quit IRC13:19
*** moravec has quit IRC13:19
*** __oanson has joined #openstack-infra13:20
snikitinsdague: Hi Sean! I saw your comment about WSGI nova job So you propose to increase the number of tests for wsgi and then move this job to the list of check jobs?13:20
openstackgerritZara proposed openstack-infra/storyboard-webclient: Basic Task Links
*** oanson has quit IRC13:23
*** _oanson has quit IRC13:23
*** esikachev has joined #openstack-infra13:24
*** yamamoto_ has joined #openstack-infra13:25
*** yamamoto_ has quit IRC13:26
*** rlandy has joined #openstack-infra13:28
*** yamamoto_ has joined #openstack-infra13:31
*** keedya has joined #openstack-infra13:31
sdaguesnikitin: honestly, I don't know right now, but we're constrained on test resources, and I don't want to pile on a new full job that is 99% the same as the existing ones13:32
*** bhunter__ has joined #openstack-infra13:32
*** links has quit IRC13:34
*** zhurong has joined #openstack-infra13:34
*** hashar has joined #openstack-infra13:35
*** yamamoto_ has quit IRC13:35
*** sbelous_ has joined #openstack-infra13:37
*** xyang1 has joined #openstack-infra13:37
*** maishsk has joined #openstack-infra13:39
*** kiran-r has quit IRC13:40
*** vmbrasseur has quit IRC13:40
*** eezhova has joined #openstack-infra13:41
openstackgerritKashyap Chamarthy proposed openstack-infra/infra-manual: creators.rst: Fix broken internal hyperlinks
fricklersomeone might want to cancel as it seems I created a loop there :D13:42
frickleron a side note, when and why did curl get removed from ubuntu instances?13:43
pabelangerfrickler: it should get killed after 30mins13:43
*** kashyap has left #openstack-infra13:43
pabelangerfrickler: we use DIBs for ubuntu-trusty so we likely need to ensure that package is installed by default13:44
pabelangerfrickler: should be easy to do once we land -bindep jobs13:44
*** larainema has quit IRC13:44
pabelangerEmilienM: fungi: I am not sure the reason for the change.13:45
fricklerpabelanger: the thing is, this worked two days ago: now it fails:
pabelangerfrickler: look at the 3rd line of the output.  We have changed base images13:46
AJaegerpabelanger: so, we miss curl on ubuntu-trusty?13:46
pabelangerbare-trusty vs ubuntu-trusty13:46
pabelangerAJaeger: likely13:47
pabelangershould be easy to fix13:47
pabelangermostly need to figure out where to add it13:47
pabelangereither dib element, puppet or bindep-fallback13:47
AJaegerIs there a guideline what goes where?13:48
*** edmondsw has joined #openstack-infra13:48
pabelangernot that I know off13:48
pabelangerwe should likely define that13:48
mnaserwoo not a single error node in the past 6 hours13:49
pabelangerSo, if JJB requires it to run a test, I think having dib element or puppet do it is the place to be.  Other wise, every projects other-requirements.txt will need to include curl13:49
*** larainema has joined #openstack-infra13:49
*** weshay has joined #openstack-infra13:49
mordredmnaser: nicely done!13:49
*** yamamoto_ has joined #openstack-infra13:50
*** kiran-r has joined #openstack-infra13:50
mnaserthanks mordred!  this is a nice external health meter of the cloud13:50
AJaegermnaser: This is great, thanks a lot for your contribution and help!13:52
mnaserno problem, we're doing a few things but we'll probably bump it up soon13:52
mnaser(the quota)13:53
pabelangerAJaeger: fungi: when you have a moment, 287488 is to enable the installation of bindep on fedora-23, but we still don't cache packages yet.13:54
*** krtaylor has quit IRC13:54
pabelangerwe'll need the gate-bindep-fallback-fedora-23 experimental job to pass first13:54
pabelangerand moving it to the gate is an easy way for other to see the results of bindep13:55
*** elmiko_ is now known as elmiko13:55
*** sfinucan has quit IRC13:56
*** placko has quit IRC13:57
*** regXboi has joined #openstack-infra13:57
AJaegerpabelanger: did you see ianw 's comment on 287488?13:57
*** vmbrasseur has joined #openstack-infra13:58
pabelangerAJaeger: Yup, I left a comment13:58
pabelangerwell, I thought I did13:58
*** bhunter__ has quit IRC13:58
pabelangerlet me repost it13:59
*** vgridnev has quit IRC14:00
*** ddieterl_ is now known as ddieterly14:01
pabelangerAJaeger: update14:01
*** vgridnev has joined #openstack-infra14:02
sdaguefungi / mordred - vexxhost is now lit?14:02
sdaguehas anyone confirmed if multinode works on it?14:02
*** jed56 has quit IRC14:03
*** flepied has quit IRC14:04
*** flepied has joined #openstack-infra14:04
*** maishsk has quit IRC14:05
*** mrmartin has joined #openstack-infra14:05
*** yamamoto_ has quit IRC14:07
*** flepied1 has joined #openstack-infra14:08
*** zhurong has quit IRC14:08
*** flepied has quit IRC14:09
*** flepied1 is now known as flepied14:09
pabelangersdague: is currently running on vexxhost14:10
pabelangerI haven't heard anybody say anything about failures14:10
pabelangerbut only been running for the last 12 hours or so14:10
sdagueright, vexxhost have never worked for a multinode job14:11
sdaguecan we pull it out of that job class?14:11
*** jaypipes has joined #openstack-infra14:12
sdagueI think we should also have a policy of lighting new providers that we don't put them into the multinode pool by default14:12
sdaguebecause the experience has been we need to tweak for every one of them14:12
sdagueand this seems to also be true for vexxhost14:12
*** jordan__ has joined #openstack-infra14:13
pabelangerwill defer to clarkb on that14:13
*** kencjohnston has joined #openstack-infra14:13
sdagueok, well right now we only have 2 data points, and they are both failures14:14
anteayawznoinsk: what do you mean by moving CI/jenkins to Kolla?14:14
sdagueand this created a 7 hour delay in the gate overnight. It would be better to err on the side of safety14:14
*** links has joined #openstack-infra14:15
anteayawznoinsk: I don't know what you are asking14:15
*** doug-fish has quit IRC14:17
*** amitgandhinz has joined #openstack-infra14:17
*** jordan_ has quit IRC14:17
*** alivigni has joined #openstack-infra14:19
*** _nadya_ has joined #openstack-infra14:20
*** rkukura has quit IRC14:20
mnaserbummer re: failures14:20
mnaserlet me know if there's anything that's on our side which we can do14:21
*** mriedem has joined #openstack-infra14:21
anteayamnaser: thanks sdague is refering to this setup:
*** doug-fish has joined #openstack-infra14:22
mnaserim looking at failure logs14:22
anteayanote fungi has a patch up to change gre to vxlan in those diagrams14:22
*** denisra has quit IRC14:23
anteayalogs are best to be sure14:23
anteayaand good morning14:23
sdaguemnaser: there is some kind of networking issue it appears, the failure I just saw is not yet in elastic search14:23
*** tiswanso has joined #openstack-infra14:23
fungihrm, i wonder why we wouldn't be able to vxlan tunnel between instances across the provider network there14:24
*** jpr has joined #openstack-infra14:24
*** tiswanso has quit IRC14:24
mnaserfailures here seem to be related to >AttributeError: 'NetworkClient' object has no attribute 'add_router_interface_with_subnet_id'14:24
sdaguethe issue is that an ssh is hanging at a socket level for 15 minutes before a failure14:24
mnaserso that one failing doesnt seem to be related to us, im looking at the other failure14:24
*** tiswanso has joined #openstack-infra14:24
*** doug-fish has quit IRC14:24
fungibut yeah, i'll get a patch up to stop multi-noding there temporarily14:24
*** doug-fish has joined #openstack-infra14:24
sdaguefungi: awesome, great14:25
fungiwhile investigation ensues14:25
sdagueI think it's just a thing we need to be cautious of14:25
*** dmsimard has quit IRC14:25
sdaguebecause with ovh, internap, and now vexxhost there have been bumps on the multinode jobs14:25
*** mrmartin has quit IRC14:25
sdagueso probably just assume they won't work by default, then test them before we light new providers there14:25
fungiagreed, after finding two providers where we couldn't pass gre between instances (and so switching to vxlan) it's clearly a blind spot in our current evaluation process14:26
*** geekinutah has joined #openstack-infra14:27
mnaserits strange because i cant imagine any reason to cause, we don't block or filter anything either14:27
sdaguemnaser: mtu things?14:28
anteayaAJaeger: for I'm curious if you have any thoughts about why tox -e zuul is outputing nonvoting status on jobs that you have demonstrating are voting14:28
anteayaAJaeger: does your tox -e zuul output disagree with mine?14:28
*** max_lobur has joined #openstack-infra14:28
mnasersdague: that's the only possibility.  but even then, we push out a specific tested mtu via neutron dhcp14:28
sdaguemnaser: right, but then we put our own overlays in14:29
max_loburG'morning All14:29
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Temporarily stop using 2-node workers in vexxhost
*** mrmartin has joined #openstack-infra14:29
mnasersdague: so if you're adjusting mtu, it's likely adjusting to the wrong number14:29
sdagueanyway, grasping at straws, but the fail is definitely at some fundamental networking layer here14:29
max_loburhow do we add people to newly created group in gerrit ?14:29
fungimnaser: yeah, we set an mtu inside the tunnel, but i think we set it to 1450 and not to some function of the interface's mtu14:30
*** jsavak has joined #openstack-infra14:30
fungii'm checking now14:30
mnaserah that can explain things14:30
fungiwe may need to make devstack-gate calculate an mtu reduction for the tunnel based on the mtu of the interface14:30
*** links has quit IRC14:30
*** eharney has quit IRC14:31
yolandahi fungi, how can i rebuild a failed pypi job? i tried to relaunch one that failed today, but i do not see a rebuild option in jenkins14:31
mnaserfungi: we push out 1394 over dhcp14:31
fungi`git grep 1450` in openstack-infra/devstack-gate confirms this hypothesis14:32
*** moravec has joined #openstack-infra14:32
mnaserso if anything 1394-5014:32
fungiyolanda: it depends on how it failed14:32
*** Qiming has joined #openstack-infra14:32
*** erlon has joined #openstack-infra14:33
fungiyolanda: there are a variety of ways that series of release jobs can go wrong, and different failure modes require different sorts of cleanup as some steps are not idempotent14:33
anteayaa peek at grafana tells me all jenkinsii seem to be jenkinsing14:33
*** kencjohnston has quit IRC14:33
yolandafungi if failed uploading the wheel file, with requests.exceptions.HTTPError: 503 Server Error: No healthy backends14:33
fungimnaser: that's almost certainly the problem--thanks. i think this is the first provider we've ended up on with an mtu other than 1500 on the instances14:34
mnaserfungi: cross-dc connections kinda force us to lower our mtus, but soon we'll be able to bump it back up .. so maybe try $link_mtu - 50;14:34
fungiyolanda: right, there are also two ways that error can go. sometimes that means it didn't upload the wheel, sometimes pypi reports that error on wheel upload even though the wheel uploaded successfully14:34
fungiyolanda: if it's the former, then enqueue-ref can be used to rerun that set of release jobs14:35
claudiub|2os-win 0.2.3 doesn't exit on pypi:
*** annegentle has joined #openstack-infra14:35
fungiyolanda: if the latter, then the sdist needs to be uploaded manually (because twine will fail if it tries to upload a wheel which is already there--pypi rejects it) and the used to run the release announcement job specifically14:36
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Increase heat timeout with addition of new lbaasv2 scenario tests
mordredfungi, yolanda: when you get a sec ^^ I wrote that to tryt o get some better logs to figure out what's going on with bluebo14:36
fungiyolanda: but of course, all this assumes that you've first checked that the project is even set up correctly to publish releases on pypi at all14:36
*** Guest60497 is now known as dansmith14:36
max_loburClarifying a little bit: we've recently added bareon-ironic project, with it's own ACL, currently the group is empty,members . Could anoyone tell me how do I add people there?14:37
mordredor, rather, I now know sort of what's going on - but looking for it was hard, so I wanted to update the logs to be friendlier14:37
AJaegeranteaya: I run tox -e zuul this morning on a clean tree with head - and it disagreed with you as I commented in the review.14:37
yolandafungi it failed when a tag was created, previous publishes to pypi worked14:37
anteayagreat, so somehow I must be stale14:37
AJaegerCould you clean your tree and double check, please?14:37
*** mrmartin has quit IRC14:38
anteayayeah I can do that, thanks for checking me14:38
openstackgerritTristan Cacqueray proposed openstack-infra/infra-specs: Create
anteayaAJaeger: I have to run some errands this morning and will fix upon my return14:39
*** yamamoto_ has joined #openstack-infra14:39
*** roxanaghe has joined #openstack-infra14:40
*** roxanaghe has quit IRC14:40
yolandafungi, no 0.2.3 content uploaded14:41
*** Jeffrey4l has quit IRC14:41
*** lykinsbd has joined #openstack-infra14:41
AJaegeranteaya: no urgency on this. And thanks a lot for that patch, you found quite a few problems! But something is strange in your tree (or mine ;)14:41
fungiyolanda: so looking at it appears this time the failure means it didn't actually upload the wheel at all, so zuul enqueue-ref should work14:42
anteayaAJaeger: likely mine14:42
anteayaAJaeger: thanks for reviewing14:42
fungiyolanda: yep, just finished tracing through it, sorry, several things going on at once14:42
*** akshai has joined #openstack-infra14:42
*** mriedem has left #openstack-infra14:42
*** mriedem has joined #openstack-infra14:43
*** sfinucan has joined #openstack-infra14:43
*** cbader has joined #openstack-infra14:44
openstackgerritMerged openstack-infra/project-config: Temporarily stop using 2-node workers in vexxhost
*** rbrndt_ has joined #openstack-infra14:46
*** nelsnels_ has joined #openstack-infra14:46
AJaegermordred, fungi, could either of you create two German mailing lists, please? and
*** vgridnev has quit IRC14:48
*** Jeffrey4l has joined #openstack-infra14:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Reduce template name for ansible bindep jobs
*** vgridnev has joined #openstack-infra14:49
pabelangerAJaeger: odyssey4me: ^ When you have a moment, that will fix the interrupter issue with tox an 128+ char path names14:49
AJaegerpabelanger: yes, we should avoid tox explosions!14:49
*** _amrith_ is now known as amrith14:49
*** nelsnelson has quit IRC14:49
fungiAJaeger: i remember seeing some discussion on the i18n ml about adding other translation-oriented, locale-specific lists. are there more than just those two?14:49
pabelangerAJaeger: fungi so, once we fix tox path names, I think we are good for bindep on ansible. cc odyssey4me14:50
pabelangerAJaeger: fungi we did notice curl is not installed when you add other-requirements.txt so we should likely add it into a dib element or puppet by default. To avoid all projects having to add it14:51
yolandafungi, i enqueued the ref again14:51
AJaegerfungi, there'S a french one I guess as well - let me check...14:52
AJaegerfungi, yolanda : - french translators list, could you review as well, please?14:53
*** mrmartin has joined #openstack-infra14:53
*** pradk_ has joined #openstack-infra14:53
AJaegerfungi, those three are all open mailing list requests that I could find directly14:53
*** pradk has quit IRC14:53
*** pradk_ is now known as pradk14:53
*** andreww has quit IRC14:54
*** andreww has joined #openstack-infra14:54
fungiAJaeger: thanks, will have a look at those in a moment14:55
*** korzen has quit IRC14:55
*** eharney has joined #openstack-infra14:55
claudiub|2yolanda: os-win 0.2.3 is now on pypi. :D thanks! :)14:55
fungipabelanger: oh, wow, we exceeded a pathname max in tox?14:56
*** dizquierdo has joined #openstack-infra14:56
pabelangerfungi: Ya, we've done that before with ansible jobs14:57
pabelangerwe have some long project names14:57
*** andreww has quit IRC14:57
*** andreww has joined #openstack-infra14:57
*** andreww has quit IRC14:57
pabelangerfungi: If you are fine with the testing approach with 287488, we can land that and I should be able to work on fedora-23 bindep on the next round of dib builds14:59
fungiso it seems14:59
markus_zWe get now bug reports in Nova based on the "DocImpact" keyword in a merged commit. That report also gets tagged with "nova", which is redundant. Can we get rid of that?14:59
*** yamamoto_ has quit IRC14:59
*** yamamoto_ has joined #openstack-infra15:00
*** ddieterly has quit IRC15:01
AJaegerfungi, yolanda, if you have some more time ;) - there's a vhost request by rossella_s: - needed for . Could you look at these as well, please?15:01
*** esker has quit IRC15:02
AJaegermarkus_z: discuss with armax: Ic9d2bdffd4acca3bb093f5037e62582961919ebf#15:02
*** kevinsho_ has joined #openstack-infra15:02
markus_zAJaeger: thanks! I didn't find that.15:03
AJaegermarkus_z: wrong change - but it's that area ...15:03
rossella_sAJaeger, thanks a lot!!!15:03
AJaegermarkus_z: having nova there makes it easier to assign to openstack-manuals - it's tagged correct for that ;)15:03
*** kevinsho_ has quit IRC15:04
*** rcernin has joined #openstack-infra15:04
markus_zAJaeger: Ah, you mean this is a preparation if we move a bug to the manuals project?15:04
*** rkukura has joined #openstack-infra15:04
markus_zAJaeger: for example15:04
openstackLaunchpad bug 1551782 in OpenStack Compute (nova) " API: Improve os-migrateLive input parameters" [Undecided,New]15:04
AJaegermarkus_z: it would help in that case15:05
* AJaeger has to attend a meeting now, be back later...15:05
*** amotoki has joined #openstack-infra15:06
*** esker has joined #openstack-infra15:07
mtreinishfungi, clarkb: mriedem and I were talking about e-r in -qa and we're seeing a lot of timeouts on the bot in the -qa channel and limited reporting to gerrit15:08
mtreinishis there anything in the logs for the bot15:08
mtreinishalso the logstash queue graph is in a zig zag pattern which is never a good sign15:09
fungipabelanger: on 287850, a 6-character reduction is sufficient to get their longest job name under the limit for tox to be happy?15:10
*** kingia has quit IRC15:10
*** rkukura has quit IRC15:12
*** zz_dimtruck is now known as dimtruck15:13
*** icey_ is now known as icey15:14
*** nmagnezi has quit IRC15:14
csabaAJaeger: got the manila +1 to
pabelangerfungi: if I counted correctly yes, let me confirm again15:15
fungiclarkb: sdague: so thinking about the mtu challenge a little more, this is probably going to get complicated to make dynamic, since i think i remember that the test scenarios have some hard-coded minimum mtu assumptions baked in as well?15:16
fungiso even if we have d-g calculate the tunneled mtu at interface mtu - 50 or something, the tests are going to blackhole traffic because they're assuming their tunnels-in-tunnels can do 1400, right?15:17
pabelangerfungi: ya, should drop us to 97 chars15:17
pabelangererr wait15:17
fungipabelanger: 128-6 != 97, so i'm missing something15:18
*** bhunter__ has joined #openstack-infra15:18
pabelangerfungi: 121 chars15:18
fungiis the missing piece len('/home/jenkins/workspace/')?15:18
pabelangerI forgot the .tox/functional/bin part15:18
fungioh, yep15:18
*** sigmavirus24_awa is now known as sigmavirus2415:18
fungiokay, cool by me then15:18
pabelanger/home/jenkins/workspace/gate-openstack-ansible-lxc_container_create-ansible-func-ubuntu-trusty-nv/.tox/functional/bin/pip is what I am using15:19
pabelangerhopefully zuulv3 allows for custom dir names :)15:19
pabelangeror mktemp15:19
fungimktemp for the job workspace would be nice15:19
*** annegentle has quit IRC15:21
openstackgerritMerged openstack-infra/openstackid-resources: Added get all events endpoints
openstackgerritMarkus Zoeller (markus_z) proposed openstack-infra/jeepyb: notify_impact: remove project name from the tags list
*** krtaylor has joined #openstack-infra15:23
*** jpr has quit IRC15:23
*** asettle has joined #openstack-infra15:23
openstackgerritMerged openstack-infra/elastic-recheck: Disable failing tests
*** geekinutah has quit IRC15:25
jeblairmtreinish: i'm looking at erbot's log file15:25
jeblairmtreinish: it is outputting things.15:25
jeblair(a lot of things.  our gerrit is busy)15:26
mtreinishjeblair: ok, heh yeah it really is15:27
jeblair2016-03-03 15:26:01,762 DEBUG recheckwatchbot: Console logs not ready for gate-nova-pypy-bindep 280653,9,e37fe2d15:27
mtreinishjeblair: yeah, I'm guessing we're having a lot of timeouts like that15:27
mtreinishbecause something is up with logstash, or it just got slower15:27
*** ddieterly has joined #openstack-infra15:27
*** asettle has quit IRC15:28
*** asettle has joined #openstack-infra15:28
jeblair2016-03-03 15:28:01,906 INFO recheckwatchbot: Sending "Console logs not available after 13:24s for gate-nova-pypy-bindep 280653,9,e37fe2d" to #openstack-qa15:28
fungipabelanger: on 287488, that won't cause the image build to fail presumably? the calls to yum there work?15:28
jeblairmtreinish: so it looks like the bot is working?15:28
*** mtanino has joined #openstack-infra15:29
mtreinishjeblair: yeah the bot seems to be working, its just almost always reporting timeouts :)15:29
pabelangerfungi: no, because we have some logic to skip packaging caching when dnf is used15:29
jeblaircurl -XGET '' -d '{"sort": {"@timestamp": {"order": "desc"}}, "query": {"query_string": {"query": "filename:\"console.html\" AND (message:\"[SCP] Copying console log\" OR message:\"Grabbing consoleLog\") AND build_status:\"FAILURE\" AND build_change:\"128940\" AND build_patchset:\"122\" AND build_name:\"gate-nova-pypy-bindep\"AND ...15:30
jeblair... build_short_uuid:feaad18"}}}'15:30
pabelangerfungi: so, bindep will be installed but bindep-fallback will not be used15:30
jeblairmtreinish: ^ there's a request it recently tried to run and got 0 hits for15:30
*** vgridnev has quit IRC15:30
pabelangeronce bindep is installed, we can test fedora-23 in the gate15:30
*** angdraug has joined #openstack-infra15:31
openstackgerritMerged openstack-infra/project-config: Reduce template name for ansible bindep jobs
dhellmannis there any way to filter the zuul status page ( by queue? When I'm running releases, I'd like to be able to watch the release-related queues regardless of the project15:33
*** achanda has joined #openstack-infra15:33
mtreinishjeblair: right, that's the query it does up front to check if a job's results are on logstash yet15:33
mtreinishit tries that in a loop for ~13min and reports timeout if there isn't anything15:33
openstackgerritMarton Kiss proposed openstack-infra/puppet-askbot: Add missing venv python to cron scripts
*** achanda has quit IRC15:39
mrmartinyolanda ^^^15:39
yolandathat was fast!15:39
openstackgerritMerged openstack-infra/system-config: Add openstack-de mailinglist
*** yamamoto_ has quit IRC15:40
*** jamesmcarthur has joined #openstack-infra15:40
openstackgerritMerged openstack-infra/system-config: Add mailinglist openstack-i18n-de
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions
openstackgerritMerged openstack-infra/system-config: Add a mailing list for the French i18n local group.
*** ccrouch has quit IRC15:42
*** esker has quit IRC15:45
openstackgerritQiming Teng proposed openstack/requirements: Bump openstacksdk version to 0.8.1
fungipabelanger: thanks, perfect15:48
openstackgerritMerged openstack-infra/nodepool: Add an error log with the server fault message
clarkbdguitarbite: AJaeger gzip should be on all of our machines15:50
*** Daisy has joined #openstack-infra15:50
*** jcoufal has quit IRC15:51
AJaegerclarkb: dguitarbite used "zip" instead of "gzip"15:51
*** jlanoux has quit IRC15:52
*** ccrouch has joined #openstack-infra15:52
*** kevinsho_ has joined #openstack-infra15:52
*** marg7175 has joined #openstack-infra15:52
*** kevinsho_ has quit IRC15:52
*** kevinsho_ has joined #openstack-infra15:53
clarkbmordred: so shade release on monday maybe?15:54
mordredclarkb: yah15:55
*** esker has joined #openstack-infra15:55
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
clarkbcool, and that will have ianw's fixes? I can make sure to restart nodepool services when that gets in if so15:55
mordredclarkb: I want to restart nodepool when the next patch lands15:55
mordredclarkb: yah15:55
mordredclarkb: (new patches to improve logging around timeouts)15:55
*** yamamoto_ has joined #openstack-infra15:56
clarkbgreat and that should have the fix for volume waiting which my launch node script updates use too. So win all around15:56
EmilienMfungi: fungi: hello! apevec and I found out that makes Puppet OpenStack CI jobs failing, because rabbitmq is stealing keystone's uid now, I think we're missing SYS_UID_MIN in this file15:57
aarefievianw: hi,  could you elaborate on your comment  please15:58
*** yamahata has joined #openstack-infra15:58
*** esker has quit IRC15:58
*** amotoki has quit IRC15:59
*** annegentle has joined #openstack-infra16:00
openstackgerritMerged openstack-infra/project-config: Enable bindep / devstack cache for fedora23 nodes
*** baoli has quit IRC16:01
*** baoli has joined #openstack-infra16:02
*** ogelbukh has joined #openstack-infra16:02
*** Qiming has quit IRC16:02
*** e0ne has quit IRC16:02
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on routes
fungiEmilienM: interesting. did you diff that against the vanilla one shipped by ubuntu?16:02
EmilienMfungi: we're running centos716:03
fungii don't think it set SYS_UID_MIN either, but i don't have time to check just yet... in a meeting16:03
fungioh, different file, sorry16:03
EmilienMfungi: no problem, we can tlak later16:03
openstackgerritMerged openstack-infra/openstackid-resources: Fix on routes
fungiEmilienM: but same story. i think i copied it from centos 6, maybe that's a little dated and needs updating?16:03
EmilienMI'll check16:04
fungiif memory serves i simply set the SYS_UID_MAX to the old UID_MIN-1 and then upped the UID_MIN by a couple thousand to give us breathing room for static id assignments (same for gid)16:04
*** jpr has joined #openstack-infra16:06
*** jsavak has quit IRC16:06
*** rcernin has quit IRC16:06
openstackgerritMerged openstack/requirements: Adding python-solumclient
*** jistr is now known as jistr|call16:08
*** dmsimard has joined #openstack-infra16:08
*** thorst is now known as thorst_afk16:09
*** jpr has quit IRC16:10
*** BobBall is now known as BobBall_AWOL16:14
*** rkukura has joined #openstack-infra16:14
mgagneEmilienM left #openstack-infra channel16:15
EmilienMmgagne: I'll never leave :-P sorry for that16:15
*** esker has joined #openstack-infra16:16
*** Daisy has quit IRC16:17
*** Daisy has joined #openstack-infra16:17
*** arxcruz has quit IRC16:18
*** sobersabre has quit IRC16:19
openstackgerritJim Rollenhagen proposed openstack/requirements: Bump ironic-lib to 1.1.0
clarkbsdague: fungi has someone written a change ot dynamically calculate the MTU?16:20
clarkbif not I can do that real quick16:20
fungiclarkb: i haven't. i started to look at d-g and then remembered that neutron had hard-set the mtu in devstack too i think?16:21
*** ddieterly has quit IRC16:21
clarkbfungi: not sure, but definitely worth checking16:21
*** Daisy has quit IRC16:22
fungiwhich will complicate matters for the jobs running there if they make assumptions about what mtu we have for them16:22
*** jcoufal has joined #openstack-infra16:22
openstackgerritMerged openstack/requirements: bump ceilometermiddleware upper constraints
*** ddieterly has joined #openstack-infra16:22
openstackgerritPaul Bourke proposed openstack-infra/project-config: Add new experimental jobs for Kolla oraclelinux
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Set the bridge MTU dynamically
clarkbfungi: sdague ^ thats one half of it at least16:23
*** jlanoux has joined #openstack-infra16:24
*** jsavak has joined #openstack-infra16:25
*** armax has joined #openstack-infra16:25
*** jpr has joined #openstack-infra16:26
*** yamahata has quit IRC16:26
*** jistr|call is now known as jistr16:26
*** vgridnev has joined #openstack-infra16:29
*** ryanpetrello has quit IRC16:30
*** Swami has joined #openstack-infra16:30
*** ddieterl_ has joined #openstack-infra16:31
*** sorantis has quit IRC16:31
*** yamahata has joined #openstack-infra16:32
*** jsavak has quit IRC16:32
*** jsavak has joined #openstack-infra16:33
*** ddieterly has quit IRC16:33
*** rkukura has quit IRC16:33
openstackgerritDoug Hellmann proposed openstack/requirements: Update upper-constraints for barbianclient 4.0.0
*** moravec has quit IRC16:35
*** jaypipes has quit IRC16:36
*** Jeffrey4l has quit IRC16:36
clarkbfungi: just confirmed we are setting a config setting in d-g for devstack with the lower mtu16:38
clarkbso that change above should be sufficient16:38
openstackgerritJohn Dickinson proposed openstack/requirements: bump python-swiftclient version
fungiclarkb: awesome. i'll prep a revert of my removal change and set it dependent on that16:39
*** moravec has joined #openstack-infra16:40
*** harlowja_at_home has joined #openstack-infra16:40
*** bhunter__ is now known as bhunter7116:40
*** korzen has joined #openstack-infra16:40
*** baoli has quit IRC16:41
*** yamahata has quit IRC16:41
*** baoli has joined #openstack-infra16:41
*** ddieterl_ is now known as ddieterly16:41
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
*** scheuran has quit IRC16:42
*** exploreshaifali has joined #openstack-infra16:45
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: WIP: Decouple gerrit processing and es queries
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: WIP: Decouple gerrit processing and es queries
*** bpokorny has joined #openstack-infra16:49
*** armax has quit IRC16:49
*** ryanpetrello has joined #openstack-infra16:51
*** armax has joined #openstack-infra16:51
*** sdague has quit IRC16:52
*** mikelk has quit IRC16:53
*** rossella_s has quit IRC16:53
*** rossella_s has joined #openstack-infra16:54
*** woodster_ has joined #openstack-infra16:55
*** asettle has quit IRC16:56
*** dmsimard has quit IRC16:57
*** esker has quit IRC16:58
nibalizergood morning16:58
*** thorst_afk is now known as thorst_16:59
*** yamahata has joined #openstack-infra17:00
*** hashar has quit IRC17:01
*** dtantsur is now known as dtantsur|afk17:01
*** gyee has joined #openstack-infra17:01
*** ashleighfarnham has joined #openstack-infra17:01
wznoinskanteaya, are there any intentions at moving jenkins integration/scenario builds to use Kolla instead of devstack?17:01
*** doug-fish has quit IRC17:02
*** jaosorior has quit IRC17:04
*** exploreshaifali has quit IRC17:04
*** ryanpetrello has quit IRC17:05
*** vincentll has quit IRC17:05
clarkbfungi: greghaynes bkero funny story, a local build of the dib image using the same elements ubuntu-trusty uses is 4.8GB large17:05
clarkbso now I really don't know what is going on17:05
*** ryanpetrello has joined #openstack-infra17:05
bkeroclarkb: your build system is different than the one the previous image was built on?17:05
bkeroclarkb: run 'file' on the qcow2 file. Is it v2 or v3?17:05
clarkbtest-image.qcow2: QEMU QCOW Image (v2), 12870418432 bytes17:06
bkeroHuh, ok17:06
bkeroI'm generating v3 files, but that's lolarch17:06
clarkbI use the same ubuntu release on the same cpu architecture and the same versiosn of elements17:06
clarkbso in theory it is the "same"17:06
*** nmagnezi has joined #openstack-infra17:06
fungigiven your local ubuntu-trusty build is smaller even than the production devstack-trusty build, we've clearly got some crufty something going on on our builders17:06
clarkbfungi: yup I think it may be more related to that than anything else17:07
clarkblike the cache growing cruft17:07
fungismaller by a gigabyte17:07
fungicurious what part of the cache is leaking into the images17:07
*** dmsimard has joined #openstack-infra17:07
fungioh, unless dib is copying a large and crufty cached something into the image and then deleting it17:08
fungiclarkb: anyway, one thing still to check is whether zeroing the free space in the production raw images and then converting them to qcow2 significantly alters their size compared to the production qcow2 images17:08
*** sbelous_ has quit IRC17:08
clarkbyup I think it is still something to check as that may be an easy workaround17:09
*** Guest5714 has quit IRC17:09
*** Hal has joined #openstack-infra17:09
*** doug-fish has joined #openstack-infra17:09
*** Hal is now known as Guest6391917:10
openstackgerritDavid Lyle proposed openstack/requirements: Bump upper constraint django_openstack_auth
mgagnewould there be interest in maintaining smaller Nodepool nodes to run smaller jobs like pep8, lint or docs? This could allow better utilization of the resources given by provider and/or run more jobs.17:11
*** sfinucan_ has joined #openstack-infra17:11
*** IlyaG has joined #openstack-infra17:13
lucasagomesclarkb, morning, if you have a time today do you mind taking a look at ?17:13
clarkbmgagne: sort of, there are two thing swe would have ot solve with that to make it viable. The first is in many cases our quota limits the number of instances regardless of size so we can't actually be bigger until we fix that (this is the easy one, send some emails get quota tweaks). The other is the nodepool scheduler would probably need to learn how to do multi axis scheduling in order to do this17:13
lucasagomesit's a small patch and very wanted for ironic since that enables we to run tests in gate with our refence driver (ipmi ones)17:13
*** sbelous_ has joined #openstack-infra17:13
clarkbmgagne: it is something that would probably be worth mocking up in the nodepool load simulator to see how it shakes out17:14
mgagneclarkb: would there be a way to set max-servers per images/label?17:14
*** sfinucan has quit IRC17:14
clarkbmgagne: not today, what you can do today is have two "different" provider images with different flavor settings.17:15
mgagneclarkb: true, nice workaround =)17:15
clarkbmgagne: then you would assign the pep8 jobs and docs jobs to the smaller provider image and devstack and unittests to the larger provider image17:15
*** apoorvad has joined #openstack-infra17:16
clarkbmgagne: I think the trouble is that nodepool won't proper factor the two different sizes in when making its assignments against the max servers which is what I think needs fixing with multi axis scheduling support17:16
mgagneclarkb: because I believe those jobs could benefit from smaller and less performant nodes.17:16
clarkbbut it is possible that this would just work17:16
*** ccrouch1 has joined #openstack-infra17:16
clarkband the load simulator thing that jeblair wrote up should in theory check?17:16
mgagneI know some of those words =)17:16
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Promote gate-neutron-fwaas-dsvm-tempest from experimental to non-voting
*** asettle has joined #openstack-infra17:17
clarkbmgagne: basically I think the nodepool scheduler only considers max-servers and not the actual flavor size17:18
*** ccrouch has quit IRC17:18
*** sbelous_ has quit IRC17:18
mgagneclarkb: ok, having 2 providers would "fix" that right?17:18
clarkbmgagne: so if we have max servers of 100 nodepool will run a less informed mix of flavors rather than say the 50 small and 50 large you want or whatever17:18
clarkbmgagne: oh yes, sorry we can trick it with multiple providers17:19
mgagneclarkb: but I guess this would need more work than it looks where a job would still be able to run on bigger nodes if they are available as not all providers will be able to host smaller nodes17:19
*** salv-orlando has joined #openstack-infra17:20
*** MarkAtwood has joined #openstack-infra17:20
*** salv-orl_ has quit IRC17:22
mordredmgagne, clarkb: so, this problem _is_ a thing that we're hoping to solve with zuul v3, btw17:22
clarkbmgagne: that too is "tricky". We can tell gearman to run a job on A or B but it won't prefer one over the other iirc. Instead gearman says "here have a job" to the next hting in line17:22
clarkbmordred: yup17:22
mgagneclarkb: true17:22
mordredlike, it's one of the tricky problems that has led to the design of zuul v317:23
mordredan in which we hope to even be able to just run pep8 jobs in a container on a reusable machine perhaps17:23
mordredbut it does require a fairly deep rework of the scheduler logic17:23
mgagnemordred: I was waiting for that container suggestion to happen =)17:23
jeblairthe problem is, we can't run it in a container on a host that we might use for something we actually trust17:24
jeblairbecause container security is lolwat?17:24
mordredbut we could potentially have some hosts that only work as container sources ...17:24
mgagnedon't know why it wouldn't be possible17:24
mordredin any case, it's a capability we need to grow so that we can talk abou how to use it17:24
*** andymaier has quit IRC17:25
*** jaosorior has joined #openstack-infra17:26
clarkbjeblair: and if you want to start pretending containers are secure a great chunk of our jobs won't function within them (you have to strip away so many rights that stuff starts to break)17:26
mgagnemordred: in the very short term, wouldn't it be possible to duplicate providers and dedicate some resources to smaller jobs/nodes? And tell Jenkins to choose whatever nodes fits the requirements (large or small)17:26
mgagnemordred: wouldn't be the great with low usage but on high usage day, smaller jobs would still be able to run on smaller nodes and free the queue of them17:27
*** SumitNaiksatam has joined #openstack-infra17:27
mgagnemordred: assuming we can find new resources too otherwise I guess we don't gain much17:28
*** jvrbanac_ is now known as jvrbanac17:28
jeblairmgagne: if you configured it to run on multiple types of nodes, that will have an effect on nodepool's allocator; i'm not entirely sure what the effect will be17:28
mgagnejeblair: unless code changed, I once saw jobs with OR operator in its label17:29
jeblairhopefully it will do what you want, and try to allocate nodes appropriately17:29
jeblairmgagne: yes, it works -- i'm just saying that we haven't used the ability for this precise purpose before17:29
jeblairmgagne: usually when we do it, the job is allowed to run on multiple types but be select one of them at runtime, so nodepool always knows exactly which one it needs17:30
jeblairmgagne: but i think when nodepool doesn't know, it assigns the demand to the first node type in the list17:30
jeblairmgagne: so you might end up inadvertently favoring one type17:30
mgagnethe smaller one? :D17:31
*** annegentle has quit IRC17:32
mgagnebut I could be wrong about that OR operator or it could be from a time where slaves were not single-use17:32
*** jistr has quit IRC17:33
jeblairmgagne: you aren't wrong -- i'm just saying we didn't use it like you are suggesting.17:33
*** korzen has quit IRC17:34
mgagnewhy can't I pervert systems to fit my selfish idea? :D17:34
*** mrmartin has quit IRC17:35
*** piet has joined #openstack-infra17:35
jeblairmgagne: zuul can specify which type of node to run a job on, so when we have jobs defined in jenkins with an ||, we actually have zuul pick one before it launches, so nodepool never has to guess.  sorry if i didn't explain that well before :)17:35
mgagnefrom custom python functions right?17:35
jeblairmgagne: yep17:36
mgagneso Zuul doesn't really know if Nodepool has capacity for that label?17:36
*** jaypipes has joined #openstack-infra17:36
fungimy other concern with it is that this could be a premature optimization. more performant nodes finish static analysis jobs faster, and turn over faster clearing the way for new workloads. we probably mostly only gain on the node boot/delete overhead by tweaking this17:36
*** degorenko is now known as _degorenko|afk17:37
jeblairmgagne: right, the choice is not made due to capacity, it's based on the job being run (eg, this branch runs on X, this other branch runs on Y)17:37
mgagnefungi: I'm sure sphinx doesn't use all 8 vcpus =)17:37
fungiat the expense of much additional complexity17:37
anteayawznoinsk: there have been no conversations of which I am aware of using kolla for our testing rather than devstack17:37
jeblairfungi: yep17:37
fungimgagne: but also how long is it wasting those other 7 vcpus?17:37
fungicompared to a devstack-based job that's eating all 8 vcpus for an hour or more17:37
mgagnefungi: long enough to not allow 3-4 others jobs to run?17:37
fungi3-4 other very quick jobs perhaps17:38
mgagnefungi: that's the whole idea17:38
fungibut a comparatively small percentage of the overall workload17:38
*** sc68cal has joined #openstack-infra17:38
mgagnefungi: think about providers too, they don't have infinite capacity, they can provide small nodes in greater quantity than bigger ones17:38
fungii have no solid data on it, but would be surprised if it even netted us 10% additional capacity given the breakdown of our current job workloads17:39
jeblair(also, many of our providers oversubscribe us on cpu, so our "idle" vcpus may not actually be idle -- they may be running devstack)17:39
anteayaI don't even know what kolla does to be honest17:39
mgagnejeblair: I find oversubscribe doesn't work well at all, ask sdague17:39
*** matrohon has quit IRC17:39
asselin_fungi, we have a job in the check queue that is stuck. The jenkins jobs is done, but jenkins failed and now the job is in the queue and unable to complete. What's the best way to restart it?17:40
clarkbasselin_: often that happens when your job crashes the test slave17:40
fungia vast majority of our quotas are spent on devstack-based jobs, so optimizing the utilization of the other faster/smaller jobs doesn't necessarily buy us much17:40
clarkbasselin_: if possible the best way to work with that is to stop crashing the test slave17:40
wznoinskanteaya thanks, fungi, pleia2 would you know any possible leaning towards Kolla in Jenkins builds instead of devstack in next sometime (year?)17:40
asselin_clarkb, in this case we think its caused by the jenkins VM (unexpected) reboot which is time correlated.17:40
anteayawznoinsk: why are you asking17:40
jeblairmgagne: i don't either, but i'm responsible for the architecture of very few public clouds.17:41
jeblairalmost none17:41
anteayawznoinsk: do you want to advocate for us doing so?17:41
anteayaif yes, then do advocate17:41
asselin_but other than restarting there anything else we can do? re-enqueue? promote? other hack?17:41
wznoinskanteaya, I want to see what's the sentiment for Kolla in the OS infra17:41
*** korzen has joined #openstack-infra17:41
*** vgridnev has quit IRC17:41
anteayawznoinsk: well my sentiment is "what is Kolla?"17:41
clarkbasselin_: stop rebooting it then? why is it rebooting?17:42
anteayaI'm not sure if any others have given it much thought17:42
anteayaif you want to expouse the beneifits of us looking at kolla, do expouse17:42
asselin_clarkb, seems there was a failure on the esx compute node that caused all of its VMs to reboot17:42
clarkbanteaya: wznoinsk keep in mind thats largely QAs arena aiui17:42
*** vgridnev has joined #openstack-infra17:42
anteayaclarkb: great, thank you17:43
wznoinskanteaya, I'll be honest here, only because you don't know what it is it doesn't mean it's not an option for jenkins builds ;-)17:43
clarkbwe run the jobs but what goes in them is up to the project(s)17:43
clarkbasselin_: in that case the job should rerun and report17:43
anteayawznoinsk: I didn't give any opioninon about whether it was an option for jenkins builds17:43
*** zeih has joined #openstack-infra17:43
anteayawznoinsk: but you seem to have a reason for asking and I'm curious to know what it is17:43
wznoinskclarkb, thought you'd be managing what scripts you provide for projects to be used for their builds... was anyone asking about Kolla do you know?17:44
asselin_clarkb, seems that didn't happen.....17:44
clarkbasselin_: its a fifo so it goes to the end of the queue17:44
clarkbonce its gearman turn comrs around again it should run the job17:44
wznoinskanteaya, we don't plan to dev in Kolla for now, if there would be sentiment in projects/infra to start using it in next months then maybe we'd build knowledge from features/CI point of view17:44
*** sridhar_ram has joined #openstack-infra17:45
anteayawho is we in your statement?17:45
asselin_clarkb, how can I inspect the gearman queue?17:45
wznoinskteam I'm in, 10 engins17:45
anteayaI don't know what that is either17:45
wznoinskwhich would be 1 developers involved in Kolla17:45
wznoinsk10 engineers17:45
anteayawell so far there is no push in infra to use kolla that I am aware of17:45
wznoinskok, thanks17:46
*** claudiub|2 has quit IRC17:47
mgagnejeblair: I sure would like to know the performance of running those smaller jobs on smaller nodes. While larger nodes are busy running them, capacity isn't free to run devstack. Same for Puppet unit jobs. if it still takes 10m to run the puppet job but I can run 4x times jobs, we can stop wasting 6 larger nodes, use the capacity of 1/2 to create ~8 smaller nodes. And have ~4 free nodes for devstack.17:48
*** denisra has joined #openstack-infra17:48
mgagnethat's my take on it17:48
*** salv-orl_ has joined #openstack-infra17:49
*** david_cu has left #openstack-infra17:50
jeblairmgagne: i'd like to as well, which is why i want to change the nodepool allocator in zuulv317:51
clarkbasselin_: run status against the gearman socket17:51
*** jsavak has quit IRC17:51
*** sc68cal_ has joined #openstack-infra17:51
*** rkukura has joined #openstack-infra17:51
*** sputnik13 has joined #openstack-infra17:51
clarkbanteaya: wznoinsk again thats really a better question for the qa team17:52
anteayaclarkb: yup17:52
clarkbanteaya: wznoinsk devstack is a qa project so it gets wide use17:52
*** salv-orlando has quit IRC17:52
thervelifeless, In case you see this, fixes a pretty big performance hit on testr17:52
thervelifeless, Several minutes lost for each runs in the gate17:53
*** aarefiev22 has joined #openstack-infra17:53
therveWould be cool to push that17:53
*** baoli has quit IRC17:53
anteayasounds like management at some location is using infra as a thymus test to decide what to put dev resourses on17:53
*** kzaitsev_mb has quit IRC17:54
*** mrmartin has joined #openstack-infra17:54
anteayaor what infra is doing as a thymus test17:54
*** sc68cal has quit IRC17:54
openstackgerritMerged openstack-infra/puppet-askbot: Add missing venv python to cron scripts
*** e0ne has joined #openstack-infra17:55
*** sfinucan_ has quit IRC17:55
*** baoli has joined #openstack-infra17:55
openstackgerritMerged openstack/requirements: Change global requirements forcing cliff!=1.17.0
*** dmsimard has quit IRC17:56
*** ybathia has joined #openstack-infra17:56
*** yamamoto_ has quit IRC17:57
anteayanot thymus, litmus17:58
*** marg7175 has quit IRC17:58
anteayawow, I can't even blame that one on poor spelling17:58
openstackgerritMerged openstack-infra/puppet-elasticsearch: Make network.publish_host configurable
asettleHey infra team - wondering if someone is able to help me. I posted a patch in openstack-manuals nearly 2 days ago now. The first patch had an issue with the gate so I posted up another patch, but now it will not at all go through the gate. The patch itself isn't even in zuul. Here's the patch: I'm stumped17:58
bswartzthat was a weird autocorrect17:58
anteayabswartz: that was all my fault17:59
anteayaobviously today is not my best language day17:59
anteayaasettle: it depends on a patch which is in merge conflict:
asettleOHHHHHH *duh*18:00
asettleWhy would it go through the first time though?18:00
anteayaI don't see that jenkins has ever commented on 28653018:01
fungiasettle: because its dependency might not have been in a merge conflict with the state of the repo at the time it was initially tested18:01
*** flepied1 has joined #openstack-infra18:01
*** marg7175 has joined #openstack-infra18:01
*** salv-orl_ has quit IRC18:02
fungichanges are constantly merging to each repository so the state of changes in development will also shift over time18:02
asettleHaha thank you anteaya and fungi :) I thought I was going crazy... turns out I can't read my own commit messages :p18:03
*** korzen has quit IRC18:03
anteayawelcome to the club18:03
*** salv-orl_ has joined #openstack-infra18:03
*** flepied has quit IRC18:03
*** dizquierdo has quit IRC18:04
*** baoli has quit IRC18:04
*** zeih has quit IRC18:04
*** baoli has joined #openstack-infra18:04
*** sarob has joined #openstack-infra18:05
*** flepied has joined #openstack-infra18:05
fungiokay, going to go find some lunch, then come back and start putting together the final batch of contributor invites for austin18:05
anteayaenjoy lunch18:06
*** jsavak has joined #openstack-infra18:06
*** flepied1 has quit IRC18:08
thingeeclarkb: getting volume path based off volume uuid support in brick
*** lucasagomes is now known as lucas-afk18:09
thingeeclarkb: can ask hemna on #openstack-cinder more about it.18:09
rm_workis jenkins-job-builder in pypi or do I REALLY have to clone it and use to install? all the docs say the latter but it seems weird that it isn't packaged somewhere?18:09
rm_worknm found it in pypi :P18:10
*** jlanoux has quit IRC18:10
rm_worki am dumb18:10
rm_workdocs could be updated though, as a note :)18:10
*** mriedem has quit IRC18:10
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL
*** salv-orl_ has quit IRC18:10
*** salv-orlando has joined #openstack-infra18:11
*** nmagnezi has quit IRC18:12
*** _nadya_ has quit IRC18:13
*** sbalukoff has quit IRC18:15
AJaegeranteaya, sdague, dhellmann, fungi, bswartz: Here's a relevant review for the tc discussion about limited resources in our CI system: adds the *fourth* jobs for glusterfs, seems a second driver but still...18:15
anteayarm_work: well you know best what doc is out of date and how it should read, it would be lovely if you were inclined to submit a patch18:16
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib
*** sbelous_ has joined #openstack-infra18:17
*** _nadya_ has joined #openstack-infra18:18
*** ihrachys has quit IRC18:18
pleia2not around much this morning, hope to be back in the afternoon18:18
jeblairAJaeger: i think sdague has raised an interesting point, but i'd really like it if we could avoid the suggestion that we have somehow come to a conclusion, or worse, that the conclusion is that people should not run ci jobs.18:18
*** exploreshaifali has joined #openstack-infra18:19
*** sarob has quit IRC18:19
asselin_clarkb, I couldn't see anything unusual. Anyway, I tried zuul promote on the check queue and it restarted it at the bottom while keeping all its time info. That's what I needed.18:19
AJaegerjeblair: Yes, I understood there was no conclusion. But sometimes it's good to look at one example to understand issues.18:20
*** asettle has quit IRC18:20
jeblairAJaeger: sure, but you're singling something out and i don't see the reason why18:20
*** mriedem has joined #openstack-infra18:20
AJaegerjeblair: since it just came in as review18:20
*** annegentle has joined #openstack-infra18:20
*** _nadya_ has quit IRC18:21
jeblairAJaeger: yes, but unless i'm grossly misunderstanding you, you seem to be suggesting that somehow this job is too much18:21
AJaegerjeblair: I'll continue reviewing it as usual...18:21
*** tphummel has joined #openstack-infra18:21
*** mtanino__ has quit IRC18:22
AJaegerjeblair: Yes, I'm wondering why there are four variants of a driver18:22
dhellmannAJaeger, jeblair : I tend to agree about continuing to add jobs, though I do also wonder why there are so many variants of what seems like the same driver.18:23
jeblairAJaeger: i think that's a fine question to ask18:23
jeblairi'm just really worried about us sending some kind of message that people should be afraid to add jobs to do the testing they need18:23
dhellmannI don't want anyone to not add a job, or remove a job, "just because" but I agree with sdague that it makes some sense to review our resource use to see if we can be more economical18:24
jeblairdhellmann: yes, i hope everyone does that more or less continuously18:24
jeblairand if we need the occasionaly reminder, that's cool too18:24
dhellmannjeblair : I think the fear is we're doing it less continuously than we ought, so -- right, a reminder18:25
jeblairit's easy to lose sight of that18:25
* dhellmann nods18:25
jeblairi just don't want a perfectly valid question which sdague asked to send us off on a witch hunt :)18:25
*** apoorvad has quit IRC18:25
dhellmanni concur18:25
openstackgerritAdam Harwell proposed openstack-infra/jenkins-job-builder: Update docs to include installation via pypi
*** annegentle has quit IRC18:26
*** asettle has joined #openstack-infra18:26
rm_workanteaya: ^^ thanks for the callout on that :P18:27
rm_work"someone should update that" is such a cop-out, lol18:27
*** zeih has joined #openstack-infra18:28
AJaegerjeblair: Understood and agreed on.18:29
openstackgerritAnita Kuno proposed openstack-infra/project-config: Remove non-voting jobs from the gate pipeline.
*** dizquierdo has joined #openstack-infra18:30
anteayarm_work: thanks for the patch :)18:30
mordredrm_work: woot. thanks!18:30
*** marg7175_ has joined #openstack-infra18:30
anteayamorning zaro18:30
anteayapleia2: hope all is well, thanks for the status update18:30
*** jaosorior has quit IRC18:31
*** dizquierdo has quit IRC18:31
*** geekinutah has joined #openstack-infra18:31
*** abregman has quit IRC18:32
*** sdague has joined #openstack-infra18:32
anteayaAJaeger jeblair dhellmann I agree with the sentiments expressed in the conversation thus far and also am curious why we need to test four variants of a driver18:33
*** marg7175 has quit IRC18:34
*** sambetts is now known as sambetts|afk18:35
*** daemontool_ has quit IRC18:38
*** kiran-r has quit IRC18:38
*** ccrouch1 has quit IRC18:38
*** _nadya_ has joined #openstack-infra18:39
*** tphummel has quit IRC18:40
*** baoli has quit IRC18:41
*** rkukura has quit IRC18:41
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Dont include plugin definition in cmd checks
*** sbalukoff has joined #openstack-infra18:42
*** baoli has joined #openstack-infra18:43
*** baoli has quit IRC18:43
*** rcernin has joined #openstack-infra18:44
*** zeih has quit IRC18:44
*** piet has quit IRC18:45
*** gyee has quit IRC18:46
EmilienMmgagne: fyi (see last comment)18:49
*** geekinutah has quit IRC18:50
*** SumitNaiksatam has quit IRC18:50
mgagneEmilienM: so it was fine 6 minutes ago but then failed? :-/18:51
EmilienMmgagne: it's maybe random18:51
mgagneEmilienM: I can bring it to the attention of our netadmins18:52
*** IlyaG_ has joined #openstack-infra18:52
mgagneif infra tells me they didn't reload anything on their side18:52
EmilienMmgagne: thx, just an fyi18:52
openstackgerritJames E. Blair proposed openstack/gertty: Add config option for git clone URL
*** sarob has joined #openstack-infra18:53
*** marg7175_ has quit IRC18:54
*** tphummel has joined #openstack-infra18:55
*** IlyaG has quit IRC18:55
*** shardy has quit IRC18:56
*** e0ne has quit IRC18:57
openstackgerritMerged openstack/requirements: Raise upper-constraint on python-cinderclient to 1.6.0
*** jcoufal has quit IRC18:58
*** yamamoto_ has joined #openstack-infra18:58
*** baoli has joined #openstack-infra18:58
*** kzaitsev_mb has joined #openstack-infra18:58
*** marg7175 has joined #openstack-infra19:00
*** moravec has quit IRC19:01
*** moravec1 has joined #openstack-infra19:01
*** baoli has quit IRC19:02
*** andymaier has joined #openstack-infra19:02
*** baoli has joined #openstack-infra19:03
anteayaAJaeger: oh good thank you19:04
*** asettle has quit IRC19:04
*** tphummel has quit IRC19:04
anteayaEmilienM: can you take another look at 287464 when you get a chance, my patch to remove non-voting jobs from the gate19:04
*** apoorvad has joined #openstack-infra19:04
*** sridhar_ram has quit IRC19:04
EmilienManteaya: sure thing, doing it right now19:04
*** sridhar_ram has joined #openstack-infra19:05
anteayaEmilienM: thank you19:05
*** yamamoto_ has quit IRC19:05
EmilienManteaya: done. +1. Do you know why it was wrong at first?19:05
*** jsavak has quit IRC19:05
openstackgerritBrian Demers proposed openstack-infra/project-config: Change networking-cisco stable all to use networking-cisco-release
anteayamy local copy must have been stale19:06
*** thiagop has quit IRC19:06
EmilienMok no problem, good19:06
anteayaso I rm'd it and git cloned fresh and now it matches19:06
*** jsavak has joined #openstack-infra19:06
anteayasorry for the confusion yesterday19:06
anteayathank you19:06
EmilienManteaya: no problem19:06
AJaegeranteaya: next time, just run: git reset --hard origin/master19:07
AJaegerno need to clone again19:08
anteayathank you19:08
*** austin81 has joined #openstack-infra19:08
anteayaI'll try to remember to do so19:08
AJaegeranteaya: I see that your change was done on master - I really suggest to always work on branches19:09
AJaegeranteaya: sorry, wrong line19:09
*** _nadya_ has quit IRC19:09
anteayaI thought I had19:09
AJaegeranteaya: I'd like to take back that statement19:09
anteayaoh okay thanks19:09
*** ociuhandu has quit IRC19:09
anteayaI'm usually wrong as it is, so welcome correction19:10
sdakeAJaeger any chance we can get some action on this
anteayabut glad I did okay here this time19:10
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Exclude neutronclient 4.1.0
sdakeour centos gates fail during the US daytime because the US is working and the internet is slower19:10
sdakethey fail on timeouts19:10
sdakeat night the gate runs in 30 minutes19:10
sdake(as in 3am my time)19:10
sdakewhat would be ideal is a centos mirror19:10
sdakesomeone did this for ubuntu recently not sure who19:11
*** rkukura has joined #openstack-infra19:11
sdakebut if we coud lget a temporary 90 minute gate until we have a proper centos mirror that would be helpful ;)19:11
pabelangersdake: yes, I am working on rhel afs mirrors19:11
pabelangerhope to start back up on it next week19:11
sdakeotherwise i have to keep typing recheck until everything is green19:11
sdakepabelanger will that be centos + epel ?19:11
sdakein the meantime, can we get a 90 minut egate ?: )19:12
*** moravec1 has quit IRC19:12
sdakeits really more abusive for me to type recheckc in then to have a 90 minut egate for a couple of our 10 jobs19:12
apevecpabelanger, rhel?19:12
apevecor centos?19:12
pabelangersdake: moving to bindep will help too19:12
*** sbelous_ has quit IRC19:12
pabelangersdake: because we cache packages on the image19:12
sdakepabelanger bindep is what?19:12
*** piet has joined #openstack-infra19:12
pabelangerapevec: yes, centos + fedora19:12
sdakepabelanger would you mind keeping me in the loop on the centos + epel mirrors being finished sos we can take advantage of  them when they are done19:13
pabelangerwill be the new way of how we create DIBs19:13
pabelangersdake: well, once they come online, they should be seemless for you. We'll likely send an email to ML about it19:14
*** esikachev has quit IRC19:14
*** kzaitsev_mb has quit IRC19:14
*** asettle has joined #openstack-infra19:15
sdakethe ubuntu mirrors were not seamless19:15
sdakebut i'll take your word for it ;)19:15
sdakewe are using the fastestmirror plugin if that is what you mean by seamless19:15
sdakeatm we override the ubuntu mirrors in our dockerfiles i  htink19:16
*** baoli has quit IRC19:16
openstackgerritArmando Migliaccio proposed openstack/requirements: OUT
*** amitgandhinz has quit IRC19:16
*** ifarkas has quit IRC19:17
*** SumitNaiksatam has joined #openstack-infra19:18
*** SumitNaiksatam has quit IRC19:18
*** asettle has quit IRC19:20
*** _nadya_ has joined #openstack-infra19:21
AJaegerarmax: is networking-cisco leaving the big neutron tent? Or why are they changing ACLs?
AJaegersdake: reviewed19:21
armaxAJaeger: not yet19:22
AJaegerarmax: do you want to -1 that - or is it ok as is?19:22
armaxAJaeger: I sent an update to the ML19:22
armaxI just did19:22
fungiclarkb: reviewing your dynamic mtu change, is gate-tempest-dsvm-multinode-full expected to fail? (gate-grenade-dsvm-multinode and gate-tempest-dsvm-neutron-multinode-full passed fwiw)19:23
*** ccrouch has joined #openstack-infra19:23
*** jtomasek_ has joined #openstack-infra19:23
clarkbfungi: I think it is less reliable for other reasons19:24
clarkbfungi: I checked the mtu and it came out as 1450 so that worked19:24
pabelangersdake: bindep won't do too much for your gate time actually, but the afs mirror will.  I thought you were using bare nodes, but that is not the case19:24
fungirm_work: what's most awesome of all is that someone _did_ update that ;)19:24
*** _nadya_ has quit IRC19:27
openstackgerritAustin Clark proposed openstack-infra/infra-specs: Deploy-Stackviz spec
openstackgerritMerged openstack-infra/jenkins-job-builder: Update docs to include installation via pypi
austin81AJaeger: Fixed your comments on ^^19:27
austin81mtreinish, sdague:
sdakeAJaeger thanks :)19:29
*** amrith is now known as _amrith_19:30
dimsfungi : clarkb : has been flaky for me for a couple of days now. is there an ongoing issue with that?19:31
anteayadims: it restarts everytime a new project is added and has to be indexed19:32
fungidims: as in sometimes refurns a 5xx error for a minute when you first go to use it?19:32
fungiif so, that's what anteaya just described19:32
anteayataron: is working on an upstream patch so that reindexing doesn't require a restart19:32
dimsfungi : that's one. i also see weird errors in the UI too19:32
fungidims: what errors?19:32
anteayabut I haven't heard a progress update in a bit19:32
*** electrofelix has quit IRC19:33
dimsanteaya : thanks.19:33
anteayadims: I'm not familiar with the ui errors complaint19:33
anteayadims: do expand19:33
dimsanteaya : fungi : let me try a few times and ping back19:33
anteayadims: sure, screenshots would be grand if you have them19:34
dimsanteaya : ack19:34
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost"
*** sdake_ has joined #openstack-infra19:35
*** sdake has quit IRC19:35
*** fungi is now known as backup_openstack19:36
anteayabackup_openstack: see in a while19:36
backup_openstackJeremy Stanley proposed openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost"
*** backup_openstack is now known as fungi19:37
fungioh, it did echo19:37
fungifor some reason i didn't see it19:37
* Clint squints.19:37
fungiclarkb: ^ the thing19:37
fungii approved your dynamic mtu solution19:38
*** Sukhdev has joined #openstack-infra19:38
fungiso once it merges we can try that again19:38
openstackgerritMonty Taylor proposed openstack-infra/shade: Use isinstance not type for dict conversions
*** pvaneck has joined #openstack-infra19:39
mordredShrews, rcarrillocruz: ^^ that19:41
mordredShrews, rcarrillocruz: I think should fix the neutronclient 4.1.0 + shade breakage19:41
*** ihrachys has joined #openstack-infra19:41
openstackgerritMerged openstack-infra/puppet-httpd: Add SSL Procotol and Cipher config to default vhost
*** abregman has joined #openstack-infra19:44
openstackgerritArmando Migliaccio proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0
armaxdhellmann: ^19:45
*** _amrith_ is now known as amrith19:45
armaxthis needed a rebase19:45
*** amitgandhinz has joined #openstack-infra19:46
*** sgordon has quit IRC19:46
*** moravec has joined #openstack-infra19:47
*** jpr has quit IRC19:47
*** exploreshaifali has quit IRC19:48
*** marg7175 has quit IRC19:49
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
rcarrillocruzmordred: cool thx, i'll try in a bit19:52
rcarrillocruzso issue on shade side then19:52
*** hashar has joined #openstack-infra19:52
rcarrillocruzreading scrollback on neutron channel...19:52
*** jordan__ has quit IRC19:52
*** vgridnev has quit IRC19:53
rcarrillocruzi see19:54
*** boris-42 has joined #openstack-infra19:55
*** annegentle has joined #openstack-infra19:55
mordredinfra-root: I may want to either cut a shade release sooner than next week, or figure out how to cut a release with only that patch above in it19:55
mordredotherwise, anyone using shade out in the world with neutron who got a neutronclient upgrade will be dead in the water19:55
fungimordred: i think the ff rush is over anyway19:55
fungithe gate is mostly a ghost town19:56
anteayayeah the graphs aren't working at full strength right now19:56
fungii'd go for it19:56
mordredfungi: cool. good. then maybe once that lands we can cut the release19:56
nibalizermordred: 287504 or a different patch?19:56
mordredwell, once rcarrillocruz verifies it fixes the issue :)19:56
dimsfungi : anteaya : seems to happen when the UI is up and a click on the button hits a 50319:56
mordrednibalizer: yah. 28750419:56
anteayaI'm not sure I'd call 13 in the gate a ghost town19:56
*** ayoung has quit IRC19:56
anteayabut it is less that we have had all week19:56
*** vgridnev has joined #openstack-infra19:56
fungidims: my gut says it started reindexing/restarting when apache tried to hand off the request there19:57
mordredneutron now returns a dict subclass, not a pure dict - so our bug there in how we detect a dict gets exposed and neutron functions stop working19:57
nibalizermordred: this looks like it just adds somee keystone v3 features19:57
*** angdraug has quit IRC19:57
anteayadims: the server broke down, that is a helpful error19:57
mordrednibalizer: oh. piddle. no, wrong patch19:57
fungidims: though i still don't buy the theory that this is only happening when we add new repos, it's far too frequent19:57
dimsfungi : 503 is very perisistent19:57
dimsfungi : yes19:58
*** dtardivel has quit IRC19:58
dimstoo frequent19:58
anteayafungi: okay fair19:58
dimsat least for 2 days straight now19:58
*** ihrachys_ has joined #openstack-infra19:58
fungii'm getting a service unavailable error from codesearch.o.o right now, in fact19:58
anteayaperhaps restarts include when we add new repos19:58
fungihaven't seen a new repo added for some hours19:58
*** jtomasek_ has quit IRC19:58
anteayais is restarting?19:58
*** jcooley_ has joined #openstack-infra19:59
dimsfungi : right19:59
dhellmannarmax : ack, +2a let's see how the jobs do this time19:59
dimsalmost feels like it comes back up and then falls right over again anteaya and fungi19:59
dimsover and over19:59
fungino hound in the process list on the server at the moment19:59
*** jcooley_ has quit IRC20:00
*** jamesmcarthur has quit IRC20:01
*** ihrachys has quit IRC20:01
*** jcooley_ has joined #openstack-infra20:01
* dims stepping out. back in an hour20:01
*** nwkarsten has joined #openstack-infra20:01
openstackgerritPaul Belanger proposed openstack-infra/system-config: Subscribe to project_config for create-hound-config
pabelangerfungi: nibalizer: I think that will help code.o.o ^20:01
ianwclarkb: OMFG!!!!!!!!!!!!!!!!!!!!!
pabelangerif it is the same issue I looked at before20:01
*** jamesmcarthur has joined #openstack-infra20:01
ianwthat is one of the first good fedora 23 runs20:02
asselin_anyone want to review the simple patch that's been hanging around for a while?
ianwwow, that only took, like, 4 months :)20:02
*** amitgandhinz has quit IRC20:02
fungiso... i issued a `sudo service hound start` and now there's a houndd in the process table20:02
anteayadims: thanks for letting us know20:02
anteayafungi: yay?20:03
fungiand codesearch.o.o is working for me again20:03
fungii have a feeling something is stopping houndd or causing it to die20:03
*** maishsk has joined #openstack-infra20:03
fungibut i welcome others to dig deeper since i have other things i need to work on at the moment20:03
*** jsavak has quit IRC20:04
openstackgerritRamy Asselin proposed openstack-infra/puppet-openstackci: Clarify oscc_file_contents instructions
*** jsavak has joined #openstack-infra20:04
*** ccrouch1 has joined #openstack-infra20:05
*** ccrouch has quit IRC20:06
*** Sukhdev has quit IRC20:07
mtreinishpabelanger: on: you realize that there isn't anything being deployed right? it's static content that needs to be built periodically and put on the slaves20:08
*** shardy has joined #openstack-infra20:10
pabelangermtreinish: like I said, my main concern is this would be the first time we are using dib elements to control this.  Where we usually do it with puppet.  Looking at the existing patchset, you are downloading npm and compiling things, but to be puppet would be a more consistent place to do that20:10
*** |-paul-| has joined #openstack-infra20:11
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: 10 day count is too high
mtreinishpabelanger: it's jsut because it's the first time we're adding js content to the output20:11
*** sdake_ has quit IRC20:11
*** sdake has joined #openstack-infra20:12
mtreinishpabelanger: how would you propose doing it in puppet? we just need the js to end up on the slave image, puppet isn't there20:13
jbernard_mriedem: hey matt, ive been looking at improving the ceph ci job and i have a few thoughts20:13
pabelangermtreinish: the other issue, we don't really have a way to test dib elements in the gate.  So moving to puppet-stackviz, will allow us to deploy using puppet-apply jobs or beaker to ensure things work as expected20:13
jbernard_mriedem: since we're not voting at the moment, i would really like to switch away from the in-devstack lib/ code, and to the ceph-plugin20:14
mriedemjbernard_: the ceph plugin job was failing before trusty did that release on 2/2420:14
*** yamahata has quit IRC20:14
mriedemas in the ceph plugin job was already unreliable, much more so than the in-tree ceph job20:15
cjthanks, fungi20:15
cjfungi: luckily, it looks like I was working on an orphaned fork of the library.  The official one is much more up-to-date.20:16
jbernard_mriedem: hmm, id like to look into improving that - if a reliable plugin could be considered as a switch20:16
mtreinishpabelanger: testing what that we can do npm build for stackviz? That should be tested by stackviz20:16
pabelangermtreinish: so would be moved into puppet.  For example, look at puppet-askbot as an example. it clones a git repo, downloads dependencies and install this self.  I would see puppet-stackviz working the same20:16
jbernard_mriedem: is that possible?20:16
pabelangermtreinish: right, testing it builds and gets installed20:16
mriedemjbernard_: i guess, it doesn't really make a difference to me20:16
mriedemjbernard_: i have no idea why the ceph plugin job was failing worse than the devstack job20:16
pabelangermtreinish: otherwise, we have no good way to ensure it wont break dibs20:16
jbernard_mriedem: that i need to look at20:16
mriedemjbernard_: since i thought the ceph plugin job was just a fork lift of the same code as in devstack20:16
mtreinishpabelanger: is your only objection that it's js instead of python? because the steps in that dib element aren't any different from what we do for python stuff20:17
mtreinishit's just using the js tooling to do it20:17
*** amitgandhinz has joined #openstack-infra20:17
mriedemjbernard_: use sdague's failopotamus (sp?) repo to chart the failure rates between those 2 jobs20:17
jbernard_mriedem: the plugin fetches packages from, which would have avoided the problem we hit last week20:17
mriedembut apparently probably introduces oehters20:17
mriedemjbernard_: stable packages or bleeding edge?20:17
mtreinishpabelanger: right, that's a known quantity. So you're proposing we don't anything new to the dib elements because it's only tested once it lands?20:17
pabelangermtreinish: no, my objection is we are using dib to clone, compile and install stackviz, when we usually use puppet to do this20:17
jbernard_if that's the case, then ill try to address them20:17
sdaguethe plugin fail rate has definitely been higher than in tree, I don't know why20:18
jbernard_mriedem: the plugin would substantially increase are ability to iterate20:18
pabelangermtreinish: having this a dib element also makes it difficult for somebody wanting to isntall stackviz downstream20:18
pabelangerif they wanted too20:18
*** max_lobur has quit IRC20:18
sdagueif we can get those fail rates to match, flipping over would be fine, we just can't go backwards on fail rates20:18
mtreinishpabelanger: so it shouldn't need to clone anything, it'll already be cloned by the git cache20:18
mtreinishit should just compile and install it20:19
jbernard_sdague: understood, ill work towards that20:19
*** rraja has quit IRC20:19
jbernard_sdague: you'll need access to the plugin to revert/rollback in case of a problem, correct?20:19
*** eharney has quit IRC20:19
jbernard_sdague: would i need to change anythign project-wise to make is considerable?20:20
sdaguejbernard_: yeh, chatting with dansmith earlier, I think we might want to treat the ceph plugin a bit special and get the qa team to adopt it20:20
sdaguehaving the current core team on it, plus add devstack-core20:20
mtreinishpabelanger: and that feels like a perfect fit for doing in a dib element. Just like we pip install a bunch of stuff.20:20
sdaguemtreinish: opinions?20:20
*** pradk has quit IRC20:20
*** maishsk has quit IRC20:20
dansmithsdague: the plugin fail rate is at base right?20:20
dansmithsdague: it's not failing for the current thing, AFAICT20:21
sdaguethen we could keep the subject matter experts working on issues, but also let the qa generalists help when we need that20:21
sdaguedansmith: ??20:21
jbernard_sdague: that would be great20:21
dansmithsdague: the graph for it is super low20:21
sdaguedansmith: I don't know where things currently stand. I was asked to approve a cut over 2 weeks ago, and I -1ed it because of failure rate differences20:21
rcarrillocruzso works20:21
jbernard_dansmith: turns out, we were already pulling packages in the plugin20:21
jbernard_dansmith: but not in devstack proper20:22
dansmithdid not mean that at all, sorry20:22
mriedemsdague: jbernard_: dansmith: i'll generate a graph to compare20:22
rcarrillocruzif you sort out the test failures i think we are good20:22
dstufftthat's a hell of an URL20:22
pabelangermtreinish: I disagree. the only things we pip install are zuul and some other JJB stuff. everything else dependency wise, we use puppet to manage20:22
sdaguedansmith: tinyurl please20:22
pabelangerjust look to devstack dependencies for example20:22
dansmithcompletely wrong paste :/20:22
mtreinishpabelanger: but if you really think doing a npm build in a dib element is evil an alternative I see is just having a post job build a tarball and just have the dib element pull that instead of doing20:22
mtreinishpabelanger: that's not the only stuff we pip install20:23
mtreinishthere is ostestr stuff and we pip install subunit2sql too20:23
sdagueok, this is a very different graph than I had 2 weeks ago20:23
sdagueso, yes, that looks better20:23
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fixes for latest cinder and neutron clients
mtreinishpabelanger: which is used to create a cache of old subunit streams20:23
*** ybathia has quit IRC20:23
mriedem^ removes devstack full20:23
mriedemoh i guess you can't20:23
sdaguemriedem: well devstack full is useful20:23
mriedemsince that's the baseline20:23
sdaguebecause we want it to be baselining similarly20:24
sdaguewhich, it looks like it is20:24
*** IlyaG_ has quit IRC20:24
*** matrohon has joined #openstack-infra20:24
dansmithit's 0.8% above base right now20:24
pabelangermtreinish: it is not evil, we have never done it before.  I would rather keep using puppet or even ansible to do it.  Having a dib element to manage it, does feel like the right place20:24
mriedemso the plugin looks better20:24
pabelangermriedem: sorry I have to run right now, but we can continue when I get back20:24
sdaguemtreinish: so, how do you feel about ceph plugin being in QA program?20:24
jbernard_the plugin has a few improvements, most noabely official packages for trusty20:24
mriedempabelanger: no problemo!20:24
sdaguejbernard_: yeh, super cool that it does that. And great it's now succeeding well.20:25
mtreinishsdague: I guess we can adopt it, is it an orphan?20:25
mriedemmtreinish: goto pabelanger20:25
sdaguemtreinish: well, here is the concern20:25
sdagueceph is a pretty mainline configuration, so lots of projects want it in their main test configs20:25
jbernard_if not, i can certainly backport to devstack, but this seems like a perfect time to swtich over20:25
sdaguehowever, without qa team with approve on it, if it breaks, then we can be hosed all over the place20:26
sdagueso having both the subject matter experts in the core team for it, as well as devstack generalists, would give us the best of both worlds20:26
dansmithiff mtreinish doesn't want to adopt, we can still give the appropriate folks commit on that project right?20:26
*** dmsimard has joined #openstack-infra20:27
sdaguedansmith: we cold20:27
sdaguewe could20:27
dansmither -> if20:27
sdaguebut as devstack is under qa, and ceph is such a big part of our user base, it seems like a thing we should consider just as the right thing for the qa team to do20:27
mtreinishsdague: sure I understand the concern and the desire here. I'm not opposed to bringing it into qa, I'm just wondering why it's a plugin if we view it as being so important20:28
dansmithsure if it's up for it, absolitely20:28
dansmithmtreinish: being a plugin lets the SMEs work on it easier at least20:28
timothyb89mtreinish: re: stackviz deployment, how safe is it to be running npm install in dib, given we don't have a local cache?20:28
dansmithI don't really care where we end up, I just want more reliable testing of ceph, however that goes20:28
clarkbtimothyb89: its relatively safe our other dib builds dont use our mirrors20:29
*** palendae has joined #openstack-infra20:29
mtreinishdansmith: is that the only reason?20:29
mtreinishI think it makes total sense to bring the ceph devstack stuff under qa20:29
clarkbtimothyb89: mirrors aee more important where we run things over and over 25 thousand times per day20:29
*** kzaitsev_mb has joined #openstack-infra20:29
mtreinishI'm just wondering if keeping it as a plugin makes sense given that20:29
sdaguemtreinish: I think architecturally have ceph as a plugin is good for making sure we can do that right as well20:29
timothyb89clarkb: good to know, will a failed element kill the whole build?20:30
sdaguemtreinish: and, more importantly, right now, they have a good working plugin20:30
dansmithmtreinish: the plugin is the more modern of the two implementations right now.. we could move it into devstack, or just delete the other and keep rolling20:30
*** claudiub|2 has joined #openstack-infra20:30
timothyb89clarkb: we'd probably prefer it to ignore the failure or retry, but I'm not sure how dib behaves20:31
sdagueso, I'd rather run with what's working, and if we really want to integrated it all back in newton, do that20:31
clarkbtimothyb89: yes but we will try again the next day20:31
mtreinishsdague: sure, that sounds like a good plan20:31
clarkbjust keep in mind trusty's ceph packages are or were broken20:31
sdagueok, so, then I think these are the actions20:32
mtreinishbecause I think it's just a bit weird we're saying that ceph is important enough that we as qa want to ensure it's always working20:32
*** e0ne has joined #openstack-infra20:32
mtreinishbut not enough to have it be part of devstack20:32
dansmithclarkb: that's why we're nuking the other one.. the plugin uses official builds for trusty20:32
sdague1) jbernard_ can you add devstack-core to the approvers for the devstack plugin20:32
clarkboh we dont mirror thosr20:32
timothyb89clarkb: okay, thanks for info20:32
*** scottda has joined #openstack-infra20:32
jbernard_sdague: as soon as i figure out how to do that, yes20:32
clarkbbut that may be fine depending on volumr and how good their webservers are20:32
sdague2) mtreinish can you propose governance for that change20:33
*** maishsk has joined #openstack-infra20:33
jbernard_clarkb: the plugin pulls trusty packages from, which are officially supported20:33
sdagueclarkb: the current fail rates look like it's not a huge issue20:33
mtreinishjbernard_: add devstack-core to the included groups20:33
*** ayoung has joined #openstack-infra20:33
jbernard_clarkb: it allows allows us to push package updates more quickly if there is a problem20:33
sdagueclarkb: do we have a facility to mirror other package repos except main mirrors?20:34
clarkbjbernard_: yup thats fine but package repos jave long been a source of frustration so we avoid them20:34
clarkbsdague: I think we can point reprepro at another source20:35
sdaguejbernard_: the big concern is we've found that package repos outside our networks get weird outages20:35
clarkbbut I am not 100% sure how reprepro treats third party package locations20:35
fungiclarkb: hey! e-mail from osic says tomorrow--might be time to mention fixing the keystone catalogue too?20:35
jbernard_clarkb: as long as we get officially supported pacakegs from somewhere20:35
fungior maybe you already did...20:35
sdague3) jbernard_ can you work with clarkb / infra on getting ceph packages into our mirrors from ?20:35
clarkbfungi: I think they mentioned it, its required for them to change the name20:35
jbernard_sdague: can do20:35
sdague4) switch over to voting on the the plugin job20:35
sdagueand 4 can happend before 320:36
sdaguebut we should try to close on 3 over the next couple of weeks20:36
jbernard_clarkb: what would you need from me beyond our repo url?20:36
sdaguethis all seem like a reasonable plan? mtreinish / dansmith / jbernard_20:36
clarkbjbernard_: thats probably the big thing, then someone has to write a puppet change to add it to our reprepro config20:37
sdagueI guess 5 - we should put this job on devstack as well20:37
sdagueI don't remember if it is or not20:37
fungiyeah, including it in our mirrors is probably not hard. caching the packages on our images gets into much trickier trust issues though20:37
* mtreinish waits for 5.520:37
fungisince apt sort of just munges all sources together and highest matching version for the same package name wins20:38
dansmithsdague: yes, definitely20:38
mtreinishsdague: yeah, I'm fine with that as the short term plan20:38
mtreinishwell except for the part where I have to do something :)20:39
fungiso we have to be able to cache sources which we don't make available in default configuration, or somehow assure ourselves that other repos will never be at risk of overriding packages from the main distro except the ones we want them to20:39
dansmithsdague: the ceph dev people are also promising to look over the job in a couple weeks to figure out if we can do anything more efficiently like run with the in-memory driver or something like that20:39
jbernard_clarkb: right now, we're pulling packages from
sdaguedansmith: great20:39
jbernard_clarkb: once we have the packaegs mirrored, ill need to update the plugin20:39
jbernard_clarkb: so we fetch from the mirror over ceph.com20:39
sdaguemtreinish: we might also want to have a dedicated QA session in Austin to sort out any further details here, including a long term plan20:39
jbernard_clarkb: (for trusty)20:40
clarkbjbernard_: thats actually something we may need to sort out since we have per region mirrors20:40
sdaguegiven that ceph deploy is such a huge part of our user base as a community20:40
mtreinishsdague: right, that's what I was thinking20:40
mtreinishI guess now's a good a time as any to start a summit brainstorming etherpad20:40
clarkbjbernard_: when we launch jobs in a region we update its apt/pypi/etc config20:41
clarkbjbernard_: maybe we add apt/sources.d/ceph and you can just check if that is present or not20:42
jbernard_sdague, mtreinish: devstack-core added to approvers20:42
jbernard_clarkb: sure, that'll work20:42
clarkbif its present do nothing, if it isnt add your thing there20:42
jbernard_clarkb: easy change20:42
fungiyeah, that should provide a clean transition20:42
*** pradk has joined #openstack-infra20:43
dimsfungi : anteaya : poof it's gone again20:43
dimsso it's cycling up and down20:43
fungidims: yep, it's not running again20:44
anteayadims: poof :(20:44
anteayabringing up a vm to see if I can replicate an instance20:44
dimsanteaya : nothing in the logs?20:44
anteayaI haven't done this a lot so I would give me a lot of time to do this20:44
*** jsavak has quit IRC20:44
anteayadims: I don't have access to the production server20:44
fungipuppet finished running on the server at 20:38 utc20:45
fungii wonder if it's stopping it for some reason20:45
anteayaand fungi has to do the summit passes20:45
dimsanteaya : fungi : i can't live without it :) it's indispensible :)20:45
anteayadims: it is that20:45
*** mrmartin has quit IRC20:45
mtreinishjbernard_: cool thanks20:45
*** jsavak has joined #openstack-infra20:45
*** sdague has quit IRC20:45
*** ZZelle_ has joined #openstack-infra20:46
palendaeCan someone help me understand why the docs job on doesn't appear to be including a table generated by a script? using build_sphinx produces it locally, but not in jenkins jobs20:46
*** alex_xu has quit IRC20:46
jbernard_clarkb: once you decide the exact file path and you're sure, let me know and ill submit the ceph-plugin patch to match20:46
clarkbjbernard_: ok20:46
anteayadims: and I successfully launched an instance I can't ping20:46
*** piet has quit IRC20:47
*** ihrachys has joined #openstack-infra20:47
fungianteaya: dims: so it looks like puppet triggers a refresh (restart) of the hound service every time it runs, as i suspected20:48
fungiMar  3 20:38:14 codesearch puppet-user[11161]: (/Stage[main]/Jeepyb/Package[libxslt-dev]/ensure) ensure changed 'purged' to 'present'20:48
dimsfungi : ouch20:48
fungiMar  3 20:38:17 codesearch puppet-user[11161]: (/Stage[main]/Openstack_project::Codesearch/Exec[create-hound-config]/returns) executed successfully20:48
fungiMar  3 20:38:18 codesearch puppet-user[11161]: (/Stage[main]/Hound/Service[hound]) Triggered 'refresh' from 1 events20:48
*** ddieterl_ has joined #openstack-infra20:48
fungii have a feeling the restart when it's running causes it to stop but not start, and a restart when it's not running causes it to start, leaving it down ~50% of the time20:49
fungiall theory for the moment. anyway, going back to working on other things20:49
*** alex_xu has joined #openstack-infra20:49
*** doug-fis_ has joined #openstack-infra20:49
*** doug-fish has quit IRC20:50
*** tqtran has joined #openstack-infra20:50
*** hashar has quit IRC20:50
*** ihrachys_ has quit IRC20:50
fungii've started it again, but if my theory's right, it'll be offline again in another 10-15 minutes20:50
anteayaagain wow20:50
*** ddieterly has quit IRC20:51
*** nwkarsten has quit IRC20:52
*** nwkarsten has joined #openstack-infra20:52
openstackgerritMerged openstack/os-testr: Treat xfail output like success
*** rossella_s has quit IRC20:54
*** nwkarste_ has joined #openstack-infra20:54
*** nwkarste_ has quit IRC20:54
*** rossella_s has joined #openstack-infra20:54
*** e0ne has quit IRC20:54
*** esker has joined #openstack-infra20:54
*** salv-orlando has quit IRC20:55
*** nwkarste_ has joined #openstack-infra20:55
*** salv-orlando has joined #openstack-infra20:55
*** gyee has joined #openstack-infra20:55
dimsfungi : anteaya : here's another one -
dims"too many open files"20:56
anteayadims: I've never seen that error before20:57
*** nwkarsten has quit IRC20:57
dimsanteaya : try right now when it's in a tizzy20:57
*** ldnunes has quit IRC20:57
dimsoops too late server broke down now20:57
*** asettle has joined #openstack-infra20:57
fungidims: that may indicate a need to make ulimit adjustments for the process (which looks like it's running as root, so that's a fun one)20:57
dimsguess, that's the sequence i see, up and running, then too many files, then brakes down20:58
fungidims: oh, yep, in the last few seconds houndd disappeared from the process table again20:58
dimsbreaks :)20:58
fungipuppet had already finished running at 20:52:51 utc, so i think it may not be puppet-triggered20:58
dimsthanks fungi anteaya :) you have enough information to go by now20:58
bswartzulimit -n20:59
fungithough puppet did refresh the service again, which suggests it is at least restarting it every ~15 minutes which is also not good, but the too many open files may actually be crashing the process20:59
*** Sukhdev has joined #openstack-infra20:59
*** lykinsbd has quit IRC20:59
*** e0ne has joined #openstack-infra21:00
*** stevebak` is now known as stevebaker21:00
fungiunfortunately hound's log is almost useless (empty except some startup messages with no associated timestamp) so hard to know for certain21:00
anteayasomeone said hound truncates its log on every restart21:01
anteayaI do believe that was an observation last round of why isn't hound running21:01
*** Sukhdev has quit IRC21:01
fungi283810 lines in /var/log/hound.log so i don't think it's truncating on restart21:01
*** rfolco has quit IRC21:01
*** maishsk has quit IRC21:02
openstackgerritTim Hinrichs proposed openstack/requirements: Raise upper-constraint on python-congressclient
fungiif it is truncating on restart, that's a bunch of lines for a process which only ran a couple of minutes21:03
*** sarob has quit IRC21:03
fungibut without timestamps on any of the lines i can't really tell21:03
*** sarob has joined #openstack-infra21:03
fungioh, wow no this is a traceback21:03
fungithere are actually timestamps if i scroll back far enough21:04
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Dont include plugin definition in cmd checks
fungii'm just not used to looking at tracebacks from go21:04
*** zeih has joined #openstack-infra21:04
fungibut yes, looks like it spewed a lot of "fork/exec /usr/bin/git: too many open files" errors, then fell over leaving a tens-of-thousands-of-lines traceback when it did21:05
mtreinishsdague, dansmith, jbernard_:
jbernard_mtreinish: radical21:06
*** baoli has joined #openstack-infra21:06
*** esikachev has joined #openstack-infra21:06
fungigoroutine 6159 [running]: runtime.panic(0x70e180, 0xb7230a) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb621:06
openstackgerritBen Swartzlander proposed openstack/requirements: Raise manilaclient upper to 1.8.0
*** amrith is now known as _amrith_21:07
*** jpr has joined #openstack-infra21:07
fungiso as best i can piece together from the hound log, it is restart related. it gets restarted on every puppet run, starts reindexing, and when it gets to the point where it tries to index openstack/nova it reaches an open file limit on the git subprocess21:08
*** asettle has quit IRC21:08
anteayaha ha ha21:08
*** baoli_ has joined #openstack-infra21:08
fungithis explains the several minutes of delay from when puppet restarts the process until it dies again21:08
mtreinishfungi: do you think it makes sense to resize up subunit-worker01.o.o? It's kinda close to max ram:
fungimtreinish: yeah, even -/+ buffers/cache: it's hovering at 75% used at the moment21:09
dimsnova is too big :)21:10
anteayaway to big21:10
fungimtreinish: subunit-gearman VIRT: 6822m RES: 5.8g SHR: 3540 according to top21:10
*** jamesmcarthur has quit IRC21:10
fungimtreinish: unless you have reason to believe there's a leak in subunit-gearman, i agree a larger instance may be in order21:11
*** moravec has quit IRC21:11
mtreinishfungi: yeah the way the worker is architected right now is very ram hungry (it stores all the streams in needs to process as an StringIO object in a queue)21:11
*** jamesmcarthur has joined #openstack-infra21:11
mtreinishfungi: I don't think it's a leak, just a poor design21:11
*** baoli has quit IRC21:11
fungimtreinish: doesn't look especially leaky, though it does seem to top out21:12
mtreinishit can be rearchitected to be more efficient (and probably write streams to disk instead of keeping it all in memory) but it's a matter of finding the time to do it21:12
fungimtreinish: and it's not dipping into swap at all21:12
mtreinishfungi: that's purely coincidence I think21:13
fungii'll buy that21:13
mtreinishwe've oomkillered twice in the past couple of weeks21:13
*** ashleighfarnham has quit IRC21:13
fungiso yeah, bigger instance i guess, though i don't have time to drive that right now so need other volunteers21:13
mtreinishalso there are no tests for the worker script, so I'd get a little nervous rewriting it (since it mostly works today)21:14
mtreinishinfra-root: ^^^ anyone else got a spare couple of cycles to help with the resize21:14
fungishake the can harder21:15
anteayamtreinish: pleia2 isn't feeling well today but perhaps she might be able to help tomorrow21:15
anteayaI know she has been trying to help you with this21:15
mtreinishfungi: heh21:15
*** zeih has quit IRC21:16
mtreinishanteaya: sure, yeah pleia2 has been helping a lot with the other recent subunit worker issue21:16
anteayashe is less than 100% today21:16
anteayabut maybe tomorrow21:16
*** esker has quit IRC21:17
*** e0ne has quit IRC21:17
*** jpr has quit IRC21:17
*** [HeOS] has quit IRC21:17
*** e0ne has joined #openstack-infra21:17
*** kgiusti has left #openstack-infra21:18
*** bhunter71 has quit IRC21:19
*** angdraug has joined #openstack-infra21:21
nibalizerresize? or just new node?21:22
nibalizeri didn't know rax supported resize21:22
anteayawell to the extent nova supports resize I believe21:23
anteayaand I think the extent to which nova supports resize is kind of a chat over beer kind of topic21:23
anteayafungi tried resize once upon a time21:24
*** abregman has quit IRC21:24
anteayaI think he gave up after waiting for over 24 hours for it to finish21:24
funginibalizer: there is no resize in rackspace, except for trove instances21:24
jeblairthere used to be21:24
fungiyep, there did21:24
*** ayoung has quit IRC21:24
*** ihrachys_ has joined #openstack-infra21:24
fungithey dropped it when they migrated to "new" nova-backed instances21:25
fungiyou could resize legacy instances21:25
anteayadid old resize work?21:25
jeblairwe used it often and it generally worked well aside from the fact that runtime was unpredictable (and seemingly exponentially related to instance size)21:25
anteayawonderful, thanks21:25
fungii don't know why they stopped supporting resize when they switched to real openstack, but they did21:25
*** ihrachys has quit IRC21:25
fungii recall the point at which we had about half our instances in the legacy environment, all the legacy instances had a "resize" option but none of the ones in the new environment had it21:26
nibalizerso we're talking about booting a new node?21:27
nibalizeri can likely do this21:27
jeblairfungi: the webui says it's not available for servers "made from this flavor"21:27
jeblairfungi: so maybe it's a pvhvm thing...21:27
nibalizerin spare cycles found on the floor today21:27
nibalizermtreinish: specs you need?21:27
anteayanibalizer: yes, I think booting a new node is the best way forward21:27
fungijeblair: seems likely, as we switched to pvhvm around the same timeframe21:27
*** e0ne has quit IRC21:28
fungiand there was no pvhvm in the legacy environment, so that would be a pretty reasonable explanation21:28
*** ihrachys has joined #openstack-infra21:28
jeblairthe pvhvm is worth it, i think21:28
nibalizermtreinish: this is the flavor list
nibalizerwe generally get the performance nodes21:29
*** ihrachys_ has quit IRC21:29
nibalizerwe have 8gb of ram on it right now21:29
*** abregman has joined #openstack-infra21:30
*** tqtran has quit IRC21:31
*** dkranz has quit IRC21:32
*** esikachev has quit IRC21:33
*** ddieterl_ is now known as ddieterly21:35
*** annegentle has quit IRC21:35
*** annegentle has joined #openstack-infra21:35
*** vgridnev has quit IRC21:36
bswartzhow do I debug a job that runs in the "post" pipeline? where are the logs?21:36
*** max_lobur has joined #openstack-infra21:36
*** e0ne has joined #openstack-infra21:37
fungibswartz: you need to know the hash of the commit which triggered in post (often the merge commit rather than the has of the commit which got merged) and then look at where XX is the first two characters of the hash (lower cae) and YYYY is the full hash (lower-case)21:38
fungisome day there will be a better dashboard indexing these21:38
bswartzthe commit hash not the gerrit change num?21:39
*** esikachev has joined #openstack-infra21:39
kevinbentonany thoughts on using letsencrypt to get free valid certs for the jenkins nodes?21:39
*** aeng has joined #openstack-infra21:39
kevinbenton(it's fully automatable)21:39
*** ybathia has joined #openstack-infra21:40
bswartzokay that worked21:41
*** abregman is now known as abregman|nb21:41
mtreinishnibalizer: performance 15g then?21:42
mtreinishthe other thing is the workers are horizontally scallable we could just spin up another 8 in parallel and it should work21:42
*** kingia has joined #openstack-infra21:42
*** arxcruz has joined #openstack-infra21:43
*** ihrachys_ has joined #openstack-infra21:43
*** max_lobur1 has joined #openstack-infra21:44
nibalizerwhatever you want to do21:44
nibalizerid rather go parallel if we can21:44
*** jpr has joined #openstack-infra21:44
nibalizerclarkb: does our current launch_node work or are there tricks?21:45
fungikevinbenton: why?21:45
*** rcernin has quit IRC21:45
fungikevinbenton: are you storing or passing sensitive information to/from them?21:45
*** max_lobur has quit IRC21:45
*** ihrachys has quit IRC21:45
fungikevinbenton: self-signed certs are also fully automatable21:45
fungialso we hope to be not running jenkins pretty soon21:47
kevinbentonfungi: to prevent self-signed cert errors every time i want to watch the jenkins progress on a job21:47
fungikevinbenton: i use a pretty terrible web browser, and even it will let me permanently accept the self-signed cert for that server without risk that the cert could somehow be used to compromise other activities on the web21:48
fungithough if you're concerned that someone might be presenting you with fake job results masquerading as our ci system... i guess that's a risk21:48
*** ashleighfarnham has joined #openstack-infra21:49
*** |-paul-| has quit IRC21:49
mtreinishnibalizer: ok, parallel it is then. We've never tested that before :)21:49
kevinbentonfungi: chrome doesn't make it easy to do so, and it would have to be done for every jenkins node21:49
kevinbentonfungi: it's not that i'm concerned about an impersonation21:49
*** jamesmcarthur has quit IRC21:49
fungikevinbenton: put another way, if i could turn off https for the webui on those servers and make them use http, i would21:49
nibalizermtreinish: heeh21:50
*** jtomasek_ has joined #openstack-infra21:50
kevinbentonfungi: just an annoyance to deal with on every visit to jenkins21:50
kevinbentonfungi: not a huge deal21:50
fungii still find it galling that modern web broswers have decided that encrypting with a self-signed cert and tofu model is somehow _less_ secure than not encrypting communication at all21:50
mtreinishnibalizer: I'll push up the system config patch to add the node21:50
*** max_lobur has joined #openstack-infra21:51
*** max_lobur1 has quit IRC21:51
kevinbentonfungi: yes, but as long as they show an error for it. it's presented as a dangerous situation21:51
mtreinishnibalizer: oh, nm it's done as a regex so I don't need to do anything21:51
nibalizerwhat hostname do we want21:52
*** sc68cal_ is now known as sc68cal21:52
nibalizersubunit-worker02.o.o ?21:52
fungikevinbenton: yep. best solution to this problem is to help us get rid of jenkins
mtreinishnibalizer: yeah21:53
fungii'd rather not spend time "throwing good money after bad"21:53
nibalizermtreinish: okay it is chuggin21:54
kevinbentonfungi: (side note, it would be really difficult for browsers to adopt a mode of HTTPS that is still considered insecure for all intents an purposes. HSTS and a myriad of other things would have to special-case unvalidated HTTPS connections)21:54
kevinbentonfungi: ack21:54
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Add gate-tempest-dsvm-full-ceph to experimental queue for devstack
*** thorst_ has quit IRC21:54
nibalizermordred: im reealizing there is an opportunity while launch-node is running for ansible to see the host and cause it to do things21:54
mtreinishnibalizer: ok cool, I think you'll have to manual start the worker daemon on subunit-worker02.o.o once it's all up and running21:54
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Fix manila-image-elements publish job
*** fawadkhaliq has joined #openstack-infra21:55
*** yaume_ has joined #openstack-infra21:55
fungikevinbenton: i consider https pretty insecure as it is (have you looked at some of the certificate authorities allowed to issue certs for any domain they want?), but could care less if browsers stopped showing pretty padlock icons for sites with manually accepted certificates21:56
*** dingyichen has joined #openstack-infra21:57
kevinbentonfungi: right, it's not the padlock icon. it's that the browser has accepted it as a valid HTTPS connection. There is a bunch of crap built on top of the assumption that the site has been validated or the user has manually allowed it once the site is HTTPS-secured21:57
*** alivigni has quit IRC21:58
cody-somervillejeblair, fungi, mordred, other people interested in Gertty: I implemented breadcrumb footer bar for Gertty to help folks keep track of where they are while using the stack based navigation. Here is a screenshot: -- I'm going to tidy up the code a little bit before submitting patch but wanted to see if folks thought that looked good.21:58
fungikevinbenton: sounds like web browsers would be better off adopting a third category of "connections considered secure by the global certificate authority cabal21:58
*** yaume has quit IRC21:58
jeblaircody-somerville: that looks pretty cool :)21:59
fungicody-somerville: oh, neat!21:59
anteayajhesketh: morning21:59
anteayacody-somerville: that looks useful21:59
fungikevinbenton: though take my opinions with a grain of salt. i'm pretty antagonistic to the world wide web and shake my old man cane a lot at those hip youngster protocols22:00
*** annegentle has quit IRC22:00
kevinbentonfungi: you can change your own cert store :)22:00
fungikevinbenton: and indeed i do22:00
*** annegentle has joined #openstack-infra22:00
*** thorst_ has joined #openstack-infra22:01
fungii also sometimes change it by accepting specific self-signed certs for specific sites in fact22:01
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Add subunit-worker02.o.o to the heira data
kevinbentonfungi: then that category of "connections considered secure by the cabal" no longer makes sense22:01
jeblairfungi: i think tcp shows promise.  it might catch on.22:01
mtreinishnibalizer: ^^^22:01
fungijeblair: udp 4evar22:01
fungicody-somerville: wow, neat!22:02
*** sridhar_ram has quit IRC22:02
jeblair(^ the bumper sticker on the back of afs's vw)22:02
fungii especially like the use of unicode triangle codepoints22:02
*** asettle has joined #openstack-infra22:02
*** ihrachys has joined #openstack-infra22:02
*** asettle has quit IRC22:03
*** alivigni has joined #openstack-infra22:03
*** asettle has joined #openstack-infra22:03
jeblairis that a unicode clock?22:03
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Fix manila-image-elements publish job
cody-somervilleAye. That is a unicode clock.22:04
* jeblair swoons22:04
bswartzI hate to beg for reviews, but this change is sort of high priority for manila because we're trying to wrap up our install guide and it needs to reference images published by the job in this patch:
fungikevinbenton: true. just taking a dig at the ca/bforum, who factor into many of my conspiracy theories22:04
mtreinishugh, I guess I really need to fix unicode on my laptop now...22:04
jeblaircody-somerville: hah, you are too graphical for mtreinish!22:05
*** ihrachys_ has quit IRC22:05
*** thorst_ has quit IRC22:05
*** sridhar_ram has joined #openstack-infra22:05
fungimtreinish is even more of a luddite than i am, it seems22:05
*** sarob has quit IRC22:05
*** amotoki has joined #openstack-infra22:05
fungimtreinish: are you nostalgic for iso-8859-1, or msdos cp-437?22:06
*** thorst_ has joined #openstack-infra22:06
mtreinishfungi: I have unicode working fine on all my other machines, but for whatever reason I've never gotten it to work on my laptop at all22:06
*** kzaitsev_mb has quit IRC22:07
*** ihrachys_ has joined #openstack-infra22:07
jeblairi use a cp437->unicode translator for making ansi art for presentty22:07
anteayabswartz: I'm confused, where are you publishing qcow images to and why are you doing so?22:07
mtreinishI want to blame lenovo or archlinux, but it works fine on my old arch thinkpad22:07
fungigordc: do you have a pending governance change to correct ceilometer-specs to telemetry-specs in governance now that it's renamed?22:07
anteayaam I being old fashioned believing that the artifacts we offered were tarballs?22:08
mtreinishI'm probably just missing 1 super obvious config somewhere, but it's eluded me for almost 2 years :)22:08
*** asettle has quit IRC22:09
*** kzaitsev_mb has joined #openstack-infra22:09
gordcfungi: hm.. i believe one exists. let me verify22:09
gordcnot under my name :)22:09
fungigordc: just making sure. my summit invite script alerts me to repos listed in governance which don't exist22:09
gordcah i see.22:09
*** shardy has quit IRC22:10
*** ihrachys has quit IRC22:10
*** esikachev has quit IRC22:10
fungilooks like it missed the window for the last tc meeting. that explains it22:10
fungiwill probably be corrected next week22:10
*** thorst_ has quit IRC22:11
*** jtomasek_ has quit IRC22:11
nibalizerhrm evidently requires roo22:11
nibalizerroot now22:11
nibalizerOSError: [Errno 13] Permission denied: '/var/cache/ansible-inventory/ansible-inventory.cache'22:11
*** esikachev has joined #openstack-infra22:12
fungimordred: the patch to import ansible-build-image into gerrit still hasn't merged, huh?22:12
*** nwkarste_ has quit IRC22:12
fungiis that still planned to get used?22:12
*** zeih has joined #openstack-infra22:12
fungiwe're coming up on a year of it being listed in governance22:13
bswartzanteaya: to the tarballs site, and because there's not a better alternative to my knowledge -- the ironic team uses the same approach to upload their images22:13
*** ociuhandu has joined #openstack-infra22:14
fungianteaya: yeah, at least ironic-python-agent does this, but i want to say the trove agent has something similar as well22:14
bswartzanteaya: the docs team wanted to make sure that any images referred to in the manila install guide came from an "official" openstack place and not a 3rd party site, which I completely agree with22:15
fungiEmilienM: is there a pending patch to create/import puppet-octavia into gerrit?22:15
fungiEmilienM: looks like it's been listed in governance for a while now22:15
EmilienMfungi: no I was about doing that soon22:15
*** sarob has joined #openstack-infra22:16
fungiEmilienM: okay, cool. just making sure it was still planned22:16
*** esikachev has quit IRC22:16
*** jamesmcarthur has joined #openstack-infra22:16
EmilienMfungi: it's my fault, I created the entry in governance first and then got back by urgent stuffs.. I'll initiate the module soon i think22:16
*** zeih has quit IRC22:17
fungibswartz: is manila something that might make sense in the app catalog? i wonder if we should start looking to that for an "official" place that application-layer projects would distribute their released install artifacts22:17
anteayabswartz fungi okay thanks, guess I am behind the times22:17
anteayathanks for the explaination22:17
funginot for now necessarily, but longer term22:17
*** moravec has joined #openstack-infra22:17
anteayaand I agree with the docs folks too22:17
nibalizermtreinish: okay node is online22:18
nibalizerand ansible its pansibling22:18
nibalizerso it should hit it on this run22:18
nibalizeralso dns is setup22:18
fungibswartz: i believe bootable vm images are one of the sorts of artifacts they cater to22:19
*** alivigni has quit IRC22:19
*** jtomasek has quit IRC22:19
nibalizeror more accurately 'dns is comming soon to a resolver near you!'22:19
*** regXboi has quit IRC22:19
bswartzfungi: I don't know anything about the app catalog22:19
openstackgerritSteve Martinelli proposed openstack/requirements: bump upper constraints of openstackclient
bswartzwhere can I learn more22:19
fungibswartz: oh! but also docaedo could probably talk your ear off about it22:19
mtreinishnibalizer: ok, very cool22:19
bswartzokay I have to run right now but I will check it out22:19
fungicool deal. it's some awesome stuff22:20
openstackgerritMerged openstack-infra/project-config: Add auto-discovery test job to ironic-inspector
* docaedo reads scrollback and prepares to talk some ears off22:20
bswartzin the short term I'd appreciate you considering my publish-to-tarballs fix because the install guide team has relatively soon deadlines22:20
mtreinishnibalizer: once it's all finished installing everything I'm pretty sure you'll have to start the subunit worker daemon22:20
bswartzdocaedo: I'll be back in a couple or hours -- or you can talk my ear off tomorrow22:21
mtreinishnibalizer: also: should be good to go, so we can get cacti22:21
mtreinishand I can obsess over the graphs22:21
*** claudiub|2 has quit IRC22:21
docaedobswartz: sounds good, ping me later on #openstack-app-catalog and we'll catch up22:21
nibalizermtreinish: cool22:21
nibalizeri havee +2 and rechecked that22:21
nibalizerhopefully they don't cancel each other out22:22
anteayabswartz: +222:22
*** baoli_ has quit IRC22:22
*** thorst_ has joined #openstack-infra22:23
mtreinishnibalizer: oh, I just double checked the puppet, we can't start the new worker until that hiera patch lands22:24
mtreinishthe elasticsearch_clients variable is used to set the firewall rules to allow gearman traffic22:24
*** sarob has quit IRC22:24
nibalizerfungi: can you look at 288152 when you get a chance22:24
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add roles for CI users
*** amitgandhinz has quit IRC22:25
jeblairmtreinish, nibalizer: theoretically, you should be able to start the worker and it will just keep trying to connect until 288152 lands and then it will succeed22:25
openstackgerritAkihiro Motoki proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0
*** _amrith_ is now known as amrith22:27
jeblair2016-03-03 22:12:43.701 |   Error: Could not get latest version: Timeout while contacting Timeout::Error22:27
*** thorst_ has quit IRC22:27
jeblairwhy are the beaker tests not using our mirrors?22:27
*** edmondsw has quit IRC22:28
*** piet has joined #openstack-infra22:28
jeblairlet me ask a different question.  who understands the beaker tests?22:28
openstackgerritMerged openstack-infra/devstack-gate: Set the bridge MTU dynamically
anteayajeblair: I would start with EmilienM22:28
*** aysyd has quit IRC22:29
*** bhunter71 has joined #openstack-infra22:29
anteayafungi: you want to approve
fungianteaya: you're quick!22:30
fungithanks for watching for that22:30
anteayafungi: welcome22:30
*** kzaitsev_mb has quit IRC22:30
anteayafigured you would want that in asap22:30
jeblairanteaya: EmilienM has never committed a change to puppet-openstackci22:30
anteayajeblair: oh sorry22:30
anteayahe is who I would go to for beaker things personally22:30
*** _sarob has joined #openstack-infra22:30
*** arxcruz has quit IRC22:30
EmilienMI would ping nibalizer :-P22:30
anteayaat least to start22:30
*** _sarob has quit IRC22:31
*** sarob has joined #openstack-infra22:31
*** sarob has quit IRC22:31
*** dimtruck is now known as zz_dimtruck22:31
*** sarob has joined #openstack-infra22:31
crinklejeblair: what needs to be set to tell a node to use a mirror? if it's an environment variable, beaker probably throws it out when it ssh's to localhost22:33
jeblaircrinkle: nothing needs to be done22:34
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: IPv4 network isolation testing for Ceph
jeblaircrinkle: so i'm wondering if something is undoing it22:34
jeblairperhaps our own pip module?22:34
*** matrohon has quit IRC22:35
*** piet has quit IRC22:35
*** e0ne has quit IRC22:35
fungithe mirror determination is based on evaluating the pip.conf we install on the servers, so presumably something in the job is replacing/removing it22:35
*** gnuoy_ has joined #openstack-infra22:35
*** andymaier_ has joined #openstack-infra22:35
*** Sukhdev has joined #openstack-infra22:35
*** gnuoy_ has quit IRC22:35
*** gnuoy_ has joined #openstack-infra22:35
jeblairhowever, pip.conf does not show up in the log at
*** ayoung has joined #openstack-infra22:36
*** dprince has quit IRC22:36
fungiwow, final count for purposes of free summit admission. 29% of the people getting free passes to the summit for contributing have only one change merged during the qualifying period22:36
*** thorst_ has joined #openstack-infra22:36
*** tpsilva has quit IRC22:37
EmilienMthat's weird22:37
jeblairfungi: one good change is worth 200 spelling corrections.22:37
fungii'm all for convincing people to contribute and learn the our development toolchain, but wish they would stick around after they got their free admission22:38
docaedowow that's a high percentage22:38
*** andymaier has quit IRC22:38
bkeroHuh, I figured that to get free pass would need like 5 changes22:39
fungibkero: nah, then the people who are just in it for the free pass would find 5 tiny things to patch instead of 122:39
anteayafungi: woooo22:39
fungiand we'd still see basically the same pattern i think22:40
bkerofungi: What's that? 4 free patches to all projects?22:40
jeblairagain, getting one substantial change in nova can be a signficant undertaking.  and i'm certain nova would not like people to push 5 times as many changes as they are now.22:40
bkeroThat's true. Small changes like that can be a burden on cores.22:40
anteaya29% is an all time high is it not?22:40
loquacitiesit's why docs geta an uptick before every summit22:41
*** camunoz has joined #openstack-infra22:41
anteayaor is asking that question too much work for tonight?22:41
loquacitiesdocs gets*22:41
*** rhallisey has quit IRC22:41
anteayaloquacities: yeah, poor docs22:41
loquacities29% is awful high22:41
jeblairloquacities: drat, i was going to correct you for my free pass22:41
anteayaha ha ha22:41
*** mriedem has quit IRC22:41
loquacitiesanteaya: well, we're kind of used to it these days ;)22:41
anteayajeblair: you need a few co-authors to drag along with you22:41
loquacitiesjeblair: patches welcome :P22:41
*** gnuoy_ has quit IRC22:41
fungiultimately, the point is to find a way to make it as easy as possible for active contributors to attend, and to have a system which can identify as many active contributors as possible. if a bunch of freeloaders come along for the ride, i guess that's the tradeoff for being open and inclusive (and maybe slightly gullible)22:42
ianwi dunno, devstack gets quite a few and i just kick them in.  a fix is a fix, we don't try to spell things wrong.  if, however, it makes financial sense from a conference organising perspective, i don't know22:42
loquacitiesyeah, i don't know that the system could (or even should) be ungameable22:42
loquacitiesit comes down to cores being sane, i think22:42
mtreinishnibalizer: very cool22:43
anteayaianw: it is less financial and more full seats that drain energy yet offer nothing in return22:43
nibalizerjeblair: in theory i understand our beaker tests22:43
loquacitieseven docs won't accept a patch for a patch's sake22:43
docaedoloquacities: is there such a thing as a sane core?22:43
*** baoli has joined #openstack-infra22:43
loquacitiesdocaedo: that's probably an oxymoron22:43
fungidocaedo: only when they first start22:43
loquacitieswe're all at least slightly insane, right ;)22:43
jeblairnibalizer: i'm wondering what's happening to pip.conf; working theory: our own pip module is overwriting it in the openstackci beaker tests.  but i don't see evidence of that in the jenkins log22:43
loquacitiesanyway, the point is we need to continue to apply the "is it better than we already have" test22:44
nibalizeri think i've put my head into that at least once and never got anywhere22:44
anteayaloquacities: agreed22:44
nibalizerhonestly im a bit too fragmented right now to look at it22:44
loquacitiesso, if it's a legit typo that someone has fixed, it goes in, no matter who proposes it22:44
*** apevec has left #openstack-infra22:44
loquacitiesbut if you're rearranging deck chairs, let's not bother22:44
* anteaya offers nibalizer another stripe for his shirt sleeve22:44
openstackgerritMerged openstack-infra/system-config: Add subunit-worker02.o.o to the heira data
crinklejeblair: I see which leads to is that what controls the mirror setting?22:45
*** sc68cal has quit IRC22:45
crinkleI don't see puppet-openstackci setting that anywhere22:45
mtreinishnibalizer: ^^^ once that takes effect on logstash.o.o we should be good to go on the subunit-worker02.o.o daemon22:45
openstackgerritMerged openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost"
fungicrinkle: nodepool ready scripts in project-config22:46
jeblaircrinkle: yeah, that's the only thing i can think of off the top of my head that could be erroneously resetting it22:46
jeblairfungi: right, that's what *correctly* sets it before the job runs22:47
jeblaircrinkle: ^22:47
fungicrinkle: is run by nodepool when the instance is first booted22:47
nibalizerinfra-root i am cleaning uuid nodes from puppetdb22:48
fungicrinkle: that creates the /etc/pip.conf which is in place by the time the job starts22:48
crinkleI see22:48
mtreinishclarkb: on the temporarily stop multinode vexxhost patch, was it failing before devstack?: o-h shows all green on ymq1:
mtreinishgranted it was just reverted, I'm just curious22:48
openstackgerritSam Yaple proposed openstack-infra/shade: Fixes for latest cinder and neutron clients
jeblairso to try to clarify at the risk of being repetitive: the nodepool ready scripts set the mirror in pip.conf before a job runs.  apparently, during the job run, pip.conf is updated to remove the mirror and replace it with  at least, it seems that way because we get timeout errors from  i don't know what could cause the file to be rewritten other than the code crinkle pointed at, but i see no evidence ...22:49
jeblair... it actually is responsible.22:49
pleia2nibalizer: thanks for looking into into that for mtreinish, headache today, ruining my day :(22:49
nibalizerpleia2: no worries22:49
nibalizerrest up and feel better!22:49
fungimtreinish: i believe the symptom was tempest test ssh timeouts to instances booted on the subnode22:49
pleia2nibalizer: thanks <322:49
fungimtreinish: mtu blachhole22:50
fungimtreinish: the fix was to make mtu determination in d-g dynamically calculated from the interface mtu22:50
mtreinishfungi: heh, but yeah that should have be shown in openstack-health (unless it was all check)22:50
crinklejeblair: got it...I also don't see where puppet would be overwriting pip.conf yet22:51
fungimtreinish: rather than blindly assuming all providers give us at least a 1500 byte mtu22:51
mtreinishah, that's probably a good call :)22:51
fungimtreinish: did you have any show up for vexxhost in openstack-health at all? the quota was a relatively small percentage of our overall global quota22:52
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Allow passing tenant_id to create_network
fungiand didn't get ramped up until last night and then got pulled out before we got too far into today22:52
mtreinishit's not many jobs but it's definitely there (w/ 4 multinodes)22:53
fungimtreinish: i know in the past when we had cross-node communication impact it was only when tempest and the instance it was hitting were on different nodes, so maybe the ones in the gate just got lucky?22:54
fungimtreinish: i think sdague linked to some failure examples this morning, but i don't have time to dig them out of the irc log right now22:55
*** ayoung has quit IRC22:55
*** moravec has quit IRC22:55
fungihe said something about hitting it twice this morning on patches of his, which is how he spotted it22:56
*** ihrachys has joined #openstack-infra22:56
mtreinishfungi: yeah, so it could have been on check, o-h is still only for gate22:57
*** sdake has quit IRC22:57
*** ihrachys has quit IRC22:58
anteayaI keeping thinking o-h is for ovh but somehow that is too sacrosanct22:59
*** piet has joined #openstack-infra22:59
*** ihrachys_ has quit IRC22:59
mtreinishanteaya: heh, well you were the one who came up with openstack-health as the name in ft collins :)23:00
*** sdake has joined #openstack-infra23:00
kevinbentonhello all. Can we shutoff the Cisco 3rd party CI accounts for Neutron? They are just spitting out 'failure to merge' errors on each patchset. (e.g.
anteayamtreinish: I only have myself to blame23:02
anteayakevinbenton: thanks I'll take a look23:02
*** yaume_ has quit IRC23:02
*** jsavak has quit IRC23:02
kevinbentonthanks, it's not critical because it's not -1. it's just a little harder to filter through the other CI responses with them23:03
anteayacisco ucsm isn't listed in the third party systems wikipage23:04
anteayakevinbenton: nope I understand, thanks for speaking up23:04
anteayagoing to start by seeing if I can find any of the contacts in irc23:04
kevinbentonanteaya: thanks for doing a thankless job! :)23:04
anteayaha ha ha23:04
anteayaI'm sending you your invoice :)23:05
anteayaand thanks for putting up with me23:05
anteayasame two people on the three accounts that are listed23:05
*** xyang1 has quit IRC23:06
*** xyang1 has joined #openstack-infra23:07
openstackgerritMerged openstack/requirements: Bump upper-constraint for heatclient to 1.0.0
*** sridhar_ram has quit IRC23:07
*** marcusvrn_ has quit IRC23:07
*** baoli has quit IRC23:08
*** ybathia has quit IRC23:08
*** baoli has joined #openstack-infra23:09
*** gildub has joined #openstack-infra23:09
*** xyang1 has quit IRC23:09
*** geekinutah has joined #openstack-infra23:10
*** sdake has quit IRC23:11
*** pradk has quit IRC23:11
mtreinishnibalizer: so we should be good to start the worker (if you didn't already):
nibalizermtreinish: so just service something23:12
mtreinishyeah, it's service subunit-gearman(-worker)(-A) start (or something like that)23:13
*** zeih has joined #openstack-infra23:13
mtreinishpleia2: heh, it's jenkins?23:13
pleia2mtreinish: yeah, not confusing at all!23:13
mtreinishoh I guess I just used whatever naming scheme clarkb used for the log processor worker23:14
pleia2probably so23:14
nibalizermtreinish: we still have errors23:14
nibalizeroh you know what23:14
fungiles cultes des cargo?23:14
nibalizeransible cached23:15
nibalizerso it didn't pick it up23:15
nibalizerit wouldn't have cached except that I ran launch-node as nonroot23:15
*** amotoki has quit IRC23:16
*** sridhar_ram has joined #openstack-infra23:16
anteayafungi: is it better if it is French?23:16
*** sdake has joined #openstack-infra23:17
fungianteaya: maybe. that was probably also not really french23:17
*** markvan has quit IRC23:17
*** zeih has quit IRC23:17
*** rkukura_ has joined #openstack-infra23:17
*** ayoung has joined #openstack-infra23:18
*** fawadkhaliq has quit IRC23:18
*** fawadkhaliq has joined #openstack-infra23:18
anteayayeah according to google translate cargo cult to french isn't really exciting23:19
anteayaenglish to zulu is really interesting23:19
anteayalwezinhlangano ezingathandwa ezibhekwa cargo23:19
*** rkukura has quit IRC23:19
*** rkukura_ is now known as rkukura23:19
anteayalook at that23:19
*** tobiash_ has quit IRC23:19
anteayaand cargo didn't even move23:19
anteayathat was just cult23:19
fungirather a verbose language23:20
*** sarob has quit IRC23:20
anteayaso it appears23:20
*** sarob has joined #openstack-infra23:20
*** kzaitsev_mb has joined #openstack-infra23:20
fungii can only imagine how time-consuming incidencal conversation must be23:20
anteayaha ha ha23:20
fungier, incidental23:20
*** tobiash has joined #openstack-infra23:20
*** annegentle has quit IRC23:21
*** fawadkhaliq has quit IRC23:21
*** weshay has quit IRC23:21
*** fawadkhaliq has joined #openstack-infra23:21
*** annegentle has joined #openstack-infra23:21
anteayakevinbenton: trying to find the switch on the cisco side now, going to wait a bit hoping they are successful23:22
anteayakevinbenton: thanks for saying something23:22
kevinbentonanteaya: cool. glad they are helping23:22
*** sdake has quit IRC23:22
anteayayeah me too23:22
*** Jeffrey4l has joined #openstack-infra23:23
*** markvan has joined #openstack-infra23:24
*** sarob has quit IRC23:24
*** baoli has quit IRC23:26
*** baoli has joined #openstack-infra23:27
*** Sukhdev has quit IRC23:27
*** ianw has quit IRC23:28
*** gordc has quit IRC23:29
*** tqtran has joined #openstack-infra23:30
*** dangers_` is now known as dangers_away23:32
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: WIP: test all jobs w/ NETISO_V4
*** annegentle has quit IRC23:32
*** annegentle has joined #openstack-infra23:33
*** ianw has joined #openstack-infra23:35
*** ccrouch1 has quit IRC23:35
*** yamahata has joined #openstack-infra23:35
*** ccrouch has joined #openstack-infra23:38
*** annegentle has quit IRC23:38
*** sdake has joined #openstack-infra23:41
*** zz_dimtruck is now known as dimtruck23:41
ianwdoes anyone use weechat / glowing bear?23:42
*** rhallisey has joined #openstack-infra23:42
ianwi'm wondering if it can be a real irc bouncer (like znc), or if you can only use glowing bear to it23:42
fungiianw: i do use weechat, but i just run it under tmux on a very stable shell server in the cloud and mosh to that from everywhere23:45
fungifamiliar for me, since i was doing the same thing via telnet to a gnu screen session running ircii (if i rewind this pattern far enough)23:46
*** Jeffrey4l has quit IRC23:47
Clintwith your arsenal of ircii warscripts23:47
*** ybathia has joined #openstack-infra23:47
*** sflanigan has joined #openstack-infra23:47
*** sflanigan has joined #openstack-infra23:47
*** jamesmcarthur has quit IRC23:47
*** baoli has quit IRC23:48
ianwfungi: i was looking for something with a half-decent phone client, but on my desktop, i also want a separate irc client connected over the vpn for internal irc23:48
*** jamesmcarthur has joined #openstack-infra23:48
ianwi guess i could have weechat for freenode externally hosted (and phone available), then my inside-vpn irc bouncer, and use weechat client on my desktop to talk to both ... hmm, that might work23:50
*** grue_pm has joined #openstack-infra23:51
fungiyeah, when i've needed to connect to internal work irc servers in the past, i just run a separate instance of the client somewhere with access to the vpn or internal network23:52
*** salv-orl_ has joined #openstack-infra23:52
fungiwhich usually meant i treated my workstation on my desk at hq as a shell server, and ssh'd into that over my work vpn from home/the road23:52
fungiran irssi or whatever was du jour at the time under screen and detached/reattached as needed23:53
anteayathe eyes in the glowing bear icon23:53
*** jamesmcarthur has quit IRC23:54
fungiit meant personal and work irc were always in separate xterms, but in some ways i preferred that23:54
anteayait is good to have work in a separate xterm23:55
*** salv-orlando has quit IRC23:55
*** abregman|nb has quit IRC23:56
crinklejeblair: fungi I can't reproduce the mirror issue locally, it doesn't seem to me like anything in beaker or puppet is changing pip.conf, at least for puppet-openstackci23:57
*** ybathia has quit IRC23:58
fungian infra-root might need to catch a node running that job, hold it and perform some local inspection to validate assumptions23:58
fungior write a throwaway change to echo various files to the console before and after the job23:58
fungior maybe enhance the job to collect configs we're unsure about23:59
*** ianw has quit IRC23:59
*** ddieterly has quit IRC23:59
*** sarob has joined #openstack-infra23:59
*** ianw has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!