| rcarrillocruz | lulz, groovy... | 00:00 |
|---|---|---|
| *** roxanaghe has quit IRC | 00:01 | |
| clarkb | I wonder if 2.0 will fix the slaves in config problem | 00:01 |
| *** sridhar_ram has joined #openstack-infra | 00:02 | |
| *** Sukhdev has quit IRC | 00:03 | |
| *** nelsnels_ has quit IRC | 00:04 | |
| *** ybathia has quit IRC | 00:05 | |
| *** Sukhdev has joined #openstack-infra | 00:05 | |
| *** sridhar_ram has quit IRC | 00:06 | |
| *** dims has joined #openstack-infra | 00:06 | |
| *** markvoelker has quit IRC | 00:06 | |
| *** nelsnelson has joined #openstack-infra | 00:06 | |
| *** gildub has joined #openstack-infra | 00:07 | |
| *** zz_dimtruck is now known as dimtruck | 00:07 | |
| *** sridhar_ram has joined #openstack-infra | 00:10 | |
| kevinbenton | can the graphite graphs for job failures fall behind like logstash? | 00:11 |
| *** amotoki has quit IRC | 00:11 | |
| *** sarob has joined #openstack-infra | 00:11 | |
| clarkb | kevinbenton: shouldn't those don't end up behind a processing queue other than the statsd server applying the info | 00:11 |
| *** mriedem has joined #openstack-infra | 00:12 | |
| kevinbenton | ack. thanks | 00:12 |
| *** rbrndt_ has quit IRC | 00:14 | |
| *** ybathia has joined #openstack-infra | 00:14 | |
| *** IlyaG has quit IRC | 00:14 | |
| lifeless | tchaypo: thats groovy | 00:17 |
| cj | oh hey! Finally! jenkins likes my patch :-) | 00:17 |
| cj | https://review.openstack.org/#/c/287056 | 00:17 |
| *** sdague has quit IRC | 00:18 | |
| *** annegentle has joined #openstack-infra | 00:19 | |
| *** rhallisey has quit IRC | 00:19 | |
| *** Sukhdev has quit IRC | 00:20 | |
| *** bpokorny has quit IRC | 00:21 | |
| *** ybathia has quit IRC | 00:22 | |
| *** annegentle has quit IRC | 00:24 | |
| *** yamamoto has joined #openstack-infra | 00:24 | |
| *** Jeffrey4l has quit IRC | 00:27 | |
| *** abregman has quit IRC | 00:31 | |
| *** yamamoto has quit IRC | 00:31 | |
| *** andreykurilin__ has quit IRC | 00:33 | |
| *** bpokorny has joined #openstack-infra | 00:33 | |
| clarkb | so far everything I see running on vexxhost seems happy | 00:35 |
| nibalizer | do we have nodes running there now? | 00:35 |
| nibalizer | thats awesome | 00:35 |
| clarkb | the neutron-full jobs have increased their run time (maybe we are our own noisy neighbors?) but I think its till within the time frame as our other clouds | 00:35 |
| clarkb | nibalizer: quota of 25 test instances | 00:35 |
| jhesketh | clarkb: awesome work :-) | 00:36 |
| nibalizer | http://grafana.openstack.org/dashboard/db/nodepool-vexxhost seems to be things | 00:36 |
| nibalizer | clarkb: yes great job? | 00:36 |
| clarkb | jhesketh: it was mostly mnaser, they made it very easy for us | 00:36 |
| *** dimtruck is now known as zz_dimtruck | 00:36 | |
| *** xarses has quit IRC | 00:36 | |
| nibalizer | er great job! not ? | 00:36 |
| jhesketh | well kudos to mnaser too then :-) | 00:36 |
| clarkb | it is very vanilla and other than the glance image upload hiccup everything on our side just worked with their side | 00:37 |
| mnaser | thanks :) clarkb carried it out super quick | 00:38 |
| clarkb | mnaser: and do feel free to ask for more instance information if tracking down hiccups and odd behavior. Happy to dig it up | 00:39 |
| mnaser | clarkb: i'll probably bug about it every once in a while instead of every single time a single instance has something go on | 00:40 |
| *** amotoki has joined #openstack-infra | 00:40 | |
| jhesketh | clarkb: what was that hiccup btw? | 00:41 |
| *** sdake_ has quit IRC | 00:41 | |
| *** david_cu has joined #openstack-infra | 00:41 | |
| clarkb | jhesketh: nginx in front of glance was proxying in front of slower glance backend | 00:41 |
| clarkb | jhesketh: this caused glance clien tto think upload was done well before it got confirmation of that fact causing "connectivity" problems in glance client | 00:41 |
| clarkb | jhesketh: mnaser updated their proxy settings and everything worked just fine | 00:42 |
| jhesketh | ah cool | 00:42 |
| jhesketh | nice work | 00:42 |
| mnaser | yeah, we use nginx to do all of our ssl termination | 00:43 |
| mnaser | and the request buffering was confusin glanceclient | 00:43 |
| openstackgerrit | Travis Tripp proposed openstack/requirements: Add python-searchlightclient>=0.2.0 https://review.openstack.org/268394 | 00:44 |
| *** thorst has joined #openstack-infra | 00:44 | |
| *** sdake has joined #openstack-infra | 00:45 | |
| openstackgerrit | Merged openstack-infra/project-config: mostly standardize dsvm tempest jobs on 180 minute backstop https://review.openstack.org/287407 | 00:45 |
| *** ZZelle_ has quit IRC | 00:45 | |
| *** resker has joined #openstack-infra | 00:45 | |
| mgagne | would be great to mention those details somewhere (nginx vs glance) so others can benefit from your finding/experience | 00:46 |
| *** esker has quit IRC | 00:46 | |
| mnaser | mgagne: good idea, i'll see where i can put it out (though i'm not sure how many people run nginx infront of openstack services for api termination) | 00:46 |
| clarkb | mgagne: speaking of details any chance you are willing to tl;dr the gre + security groups thing for us again so I can respond on the neutron security groups thread or maybe you want to respond yourself? | 00:47 |
| mgagne | mnaser: neither do I know where to store that info | 00:47 |
| mgagne | clarkb: load gre conntrack helper, be happy | 00:47 |
| clarkb | mgagne: mnaser might make sense in the deployment guides? | 00:47 |
| clarkb | mgagne: mnaser under a "using http proxies" section or something | 00:47 |
| mnaser | perhaps, though none of the deployment guides say anything about using another serviec for SSL termination :p | 00:48 |
| mgagne | clarkb: I can dig the exact details as to why you should load it | 00:48 |
| mnaser | filing a bug would be a good first step till i have sometime to document it but it's just 3 nginx lines | 00:48 |
| clarkb | mgagne: that would be great, I was trying to explain that security groups don't always honor the rules that are present and GRE not working was my example but I didn't have all the details handy | 00:48 |
| mgagne | clarkb: could be similar to FTP passive mode I guess | 00:48 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing gate-{name}-ansible-functional-ubuntu-trusty-nv https://review.openstack.org/287358 | 00:53 |
| *** roxanaghe has joined #openstack-infra | 00:54 | |
| mgagne | clarkb: I don't know much about GRE and its inner working but I suspect that it requires RELATED rules in iptables and RELATED requires a conntrack helper to work. https://home.regit.org/netfilter-en/secure-use-of-helpers/ | 00:56 |
| *** sarob has quit IRC | 00:56 | |
| *** amotoki has quit IRC | 00:57 | |
| mgagne | clarkb: the coworker that helped us debug the issue is on PTO so won't be able to have details before next week | 00:57 |
| clarkb | mgagne: ok no rush as long as I/we can follow up on that | 00:57 |
| *** bpokorny_ has joined #openstack-infra | 00:58 | |
| mgagne | https://ask.openstack.org/en/question/65845/why-instances-gre-tunnel-packets-into-openstackneutron-are-dropped/ | 00:58 |
| mgagne | "Ok find it, must load the nf_conntrack_proto_gre kernel module and everything is fine." :-/ | 00:58 |
| *** ashleighfarnham has joined #openstack-infra | 00:58 | |
| *** rhallisey has joined #openstack-infra | 01:01 | |
| *** zz_dimtruck is now known as dimtruck | 01:01 | |
| *** bpokorny has quit IRC | 01:01 | |
| *** xarses has joined #openstack-infra | 01:03 | |
| *** larrie_kong is now known as kong | 01:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish renderspec docs https://review.openstack.org/287105 | 01:06 |
| *** xarses_ has joined #openstack-infra | 01:07 | |
| *** ddieterly has joined #openstack-infra | 01:07 | |
| *** xarses_ has quit IRC | 01:07 | |
| *** dims has quit IRC | 01:08 | |
| *** xarses_ has joined #openstack-infra | 01:08 | |
| *** xarses has quit IRC | 01:08 | |
| *** zhurong has joined #openstack-infra | 01:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental Manila LVM job with minimal services https://review.openstack.org/285828 | 01:08 |
| *** kzaitsev_mb has quit IRC | 01:10 | |
| *** markvoelker has joined #openstack-infra | 01:10 | |
| *** sarob has joined #openstack-infra | 01:11 | |
| *** geekinutah has quit IRC | 01:11 | |
| *** keedya has quit IRC | 01:12 | |
| *** doug-fish has joined #openstack-infra | 01:12 | |
| openstackgerrit | Merged openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty https://review.openstack.org/286161 | 01:18 |
| *** xarses_ has quit IRC | 01:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests https://review.openstack.org/284693 | 01:19 |
| openstackgerrit | Merged openstack-infra/project-config: gate-rally-install-devstack-centos7 job is ready for check & gate queue https://review.openstack.org/266253 | 01:20 |
| *** Qiming has quit IRC | 01:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Make lbaasv2-minimal job voting in check and gate https://review.openstack.org/286328 | 01:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add tacker liberty to gerrit bot https://review.openstack.org/284470 | 01:21 |
| *** amotoki has joined #openstack-infra | 01:23 | |
| *** gildub has quit IRC | 01:23 | |
| *** Sukhdev has joined #openstack-infra | 01:29 | |
| *** sdake_ has joined #openstack-infra | 01:29 | |
| *** sdake_ has quit IRC | 01:30 | |
| *** xarses_ has joined #openstack-infra | 01:30 | |
| *** sdake has quit IRC | 01:30 | |
| *** sdake_ has joined #openstack-infra | 01:30 | |
| *** Sukhdev has quit IRC | 01:31 | |
| fungi | kudos mnaser and clarkb! from zero to nodepool in 24 hours flat. our fastest provider donation addition to production ever | 01:31 |
| *** andreww has joined #openstack-infra | 01:32 | |
| *** sc68cal_ has quit IRC | 01:33 | |
| *** dizquierdo has quit IRC | 01:33 | |
| *** gildub has joined #openstack-infra | 01:33 | |
| *** xarses_ has quit IRC | 01:36 | |
| *** annegentle has joined #openstack-infra | 01:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/nodepool: Add an error log with the server fault message https://review.openstack.org/287481 | 01:36 |
| fungi | mordred: ^ minor fix to that change | 01:37 |
| *** gongysh has joined #openstack-infra | 01:37 | |
| *** cznewt has quit IRC | 01:38 | |
| *** cznewt has joined #openstack-infra | 01:38 | |
| clarkb | my local dib build VM is all kinds of unhappy updating its cache :/ | 01:38 |
| clarkb | haven't been able to get anyhwere on that today | 01:39 |
| clarkb | may use cloud VM | 01:39 |
| clarkb | I want to try removing the extra/different elements one by one until I can see the delta in file size go away | 01:39 |
| clarkb | first build without growroot, then without simple-init, then s/ubuntu-minimal/ubuntu/ | 01:40 |
| *** amit213 has joined #openstack-infra | 01:41 | |
| clarkb | they don't need to be functional if we can identify where the file size is differnet | 01:41 |
| *** sam_wan has joined #openstack-infra | 01:41 | |
| *** pvaneck has quit IRC | 01:41 | |
| *** bpokorny_ has quit IRC | 01:41 | |
| *** bpokorny has joined #openstack-infra | 01:42 | |
| *** annegentle has quit IRC | 01:43 | |
| *** rfolco has quit IRC | 01:44 | |
| *** gongysh has quit IRC | 01:45 | |
| mtreinish | clarkb: you might like: https://review.openstack.org/#/c/287463/ | 01:45 |
| *** annegentle has joined #openstack-infra | 01:45 | |
| fungi | yep, whether it's bootable or not is irrelevant for this experiment | 01:47 |
| *** Swami has quit IRC | 01:47 | |
| *** openstackgerrit has quit IRC | 01:47 | |
| *** openstackgerrit has joined #openstack-infra | 01:48 | |
| bkero | Time to build every permutation of the dib elements in parallel | 01:48 |
| *** tiswanso has joined #openstack-infra | 01:48 | |
| greghaynes | clarkb: hrm? | 01:49 |
| fungi | bkero: dib element matrixing as a service! our latest openstack project team | 01:49 |
| greghaynes | clarkb: dib caching is broken? | 01:49 |
| *** tiswanso has quit IRC | 01:49 | |
| fungi | greghaynes: our ubuntu-trusty images are ~2x the size of our devstack-trusty images, while using basically the same set of elements | 01:49 |
| *** Sukhdev has joined #openstack-infra | 01:49 | |
| *** tiswanso has joined #openstack-infra | 01:50 | |
| greghaynes | fungi: ah, right, I think clarkb mentioned that to me a bit ago | 01:50 |
| clarkb | fungi: one thing I noticed is we still cache stackforge | 01:50 |
| greghaynes | fungi: I think one of those is -minimal and one is not? | 01:50 |
| clarkb | fungi: and we probably don't need to | 01:50 |
| fungi | clarkb: meh, it's tiny | 01:50 |
| clarkb | greghaynes: ya its ubuntu vs ubuntu-minimal + growroot + simple-init | 01:50 |
| fungi | i mean, i'm fine if we make a filter list of repo patterns to skip from caching, but that's verging on premature optimization | 01:50 |
| greghaynes | ubuntu vs ubuntu-minimal seems like the only real difference | 01:51 |
| clarkb | fungi: sure it won't affect the 11GB vs 5.7GB problem | 01:51 |
| clarkb | fungi: but it will affect general image size and we upload them a lot and across oceans | 01:51 |
| *** sdake_ has quit IRC | 01:51 | |
| greghaynes | and -minimal will definitely have more stuff in the apt cache since it will be installing everything from apt rather than just copying a cloudimage rootfs | 01:51 |
| *** Sukhdev has quit IRC | 01:52 | |
| *** esikache1 has joined #openstack-infra | 01:52 | |
| clarkb | greghaynes: hrm is it doubled up packages, once installed and once in cache? | 01:52 |
| fungi | clarkb: yep, but fixing the extra 5gb of ubuntu-trusty will likely buy us a few years of waste caching stackforge repos to break even | 01:52 |
| greghaynes | clarkb: IIRC we copy in the deb cache in to the images we build, so yes | 01:52 |
| clarkb | in theory its the same cache on both though | 01:52 |
| clarkb | because the build host is shared | 01:53 |
| greghaynes | clarkb: ah | 01:53 |
| fungi | mmm, i thought we disabled the local apt cache via envvars | 01:53 |
| greghaynes | oh, thats possible | 01:53 |
| clarkb | oh! | 01:53 |
| clarkb | that would explain it then | 01:53 |
| clarkb | so I guess the experiment to do is without the apt cache | 01:53 |
| *** fawadkhaliq has quit IRC | 01:53 | |
| fungi | also, the apt cache would show up as eating space in /var/cache | 01:53 |
| greghaynes | oh, you think one of them has the apt cache and one does not? | 01:53 |
| clarkb | greghaynes: well I think fungi was saying we don't share the cache across the builds | 01:54 |
| clarkb | so both are caching but minimal may be caching a lot more | 01:54 |
| greghaynes | ah | 01:54 |
| greghaynes | yes, I would expect minimal to do that | 01:54 |
| clarkb | http://paste.openstack.org/show/489033/ | 01:54 |
| clarkb | var is less than a gig | 01:54 |
| bkero | weird, posting pastebins in the future | 01:54 |
| fungi | we had to do some gyrations to get dib to not clean the apt cache out of the images, and to not dirty the apt cache in the images with things we weren't explicitly caching (e.g. old cache cruft) | 01:55 |
| bkero | also yeah, didn't we find out it's in /opt? | 01:55 |
| clarkb | bkero: we found that /opt is the biggest disk consumer but its the same on both | 01:55 |
| clarkb | because that cache is shared | 01:55 |
| *** roxanaghe has quit IRC | 01:55 | |
| greghaynes | clarkb: do you have a similar du for the other image? | 01:55 |
| bkero | clarkb: But the discrepency is 5gb? | 01:55 |
| clarkb | greghaynes: no but I can get one | 01:55 |
| fungi | could there be unlinked inodes wasting space on one and not the other? i'm stabbing in the dark now | 01:56 |
| *** claudiub|2 has joined #openstack-infra | 01:56 | |
| greghaynes | only if tar is somehow picking them up | 01:56 |
| greghaynes | or cp | 01:56 |
| fungi | because your df said something like 10gb of space used in the rootfs on an ubuntu-trusty instance, so it's gotta be something in there | 01:56 |
| bkero | So these are qcow2s, right? | 01:56 |
| clarkb | bkero: yes | 01:56 |
| bkero | qcow2s won't shrink | 01:56 |
| *** esikache1 has quit IRC | 01:56 | |
| clarkb | the raw image is 17GB | 01:56 |
| clarkb | so we start both from the same disk usage | 01:57 |
| bkero | clarkb: What if that extra 5gb is deleted stuff? | 01:57 |
| *** taron1 has quit IRC | 01:57 | |
| greghaynes | oh, let me remember how we calculate the partition size | 01:57 |
| bkero | qcow2 will balloon, but it won't shrink unless you do a conversion | 01:57 |
| clarkb | We don't really delete anything | 01:57 |
| clarkb | its all additive | 01:57 |
| bkero | You don't delete debs from /var/cache/apt/archive? | 01:58 |
| clarkb | no | 01:58 |
| bkero | Huh | 01:58 |
| clarkb | we want them in our images | 01:58 |
| fungi | we want them in there because we only download them but don't install them | 01:58 |
| clarkb | so that installing mysql is fast | 01:58 |
| bkero | right | 01:58 |
| fungi | jobs install packages if they need them, and having them in a warm cache on the instance is the fastest solution to that | 01:58 |
| *** claudiub has quit IRC | 01:58 | |
| bkero | You want the debs you're going to use for local cache, but not hte ones you aren't | 01:58 |
| fungi | yep | 01:59 |
| clarkb | http://paste.openstack.org/show/489076/ | 01:59 |
| greghaynes | hah, so I just noticed something interesting in dib | 01:59 |
| *** dims has joined #openstack-infra | 01:59 | |
| greghaynes | we determine the root partition size by doign a du on the built directory | 01:59 |
| greghaynes | that happens *before* cleanup phase runs | 01:59 |
| bkero | clarkb: That's 500mb of difference in du size | 01:59 |
| clarkb | ya nowhere near ~5.3 gb | 02:00 |
| fungi | huh, so the 6.8gb on disk in the devstack-trusty instance is smaller than the entire qcow2 it was booted from? | 02:00 |
| bkero | So some extra non-file padding in the image? | 02:00 |
| greghaynes | ok, so I wonder if thats what is going on - dib is calculating a larger root fs size than we need | 02:00 |
| fungi | 6.8gb on disk in /opt i mean | 02:00 |
| clarkb | fungi: yes it compresses | 02:00 |
| fungi | okay | 02:00 |
| clarkb | fungi: 17GB is raw | 02:00 |
| clarkb | greghaynes: maybe | 02:00 |
| *** mtanino has quit IRC | 02:01 | |
| greghaynes | clarkb: can you build this image again? | 02:01 |
| fungi | so maybe the ubuntu-trusty image isn't compressing as efficiently? | 02:01 |
| bkero | clarkb: what's the block device size on both of them? | 02:01 |
| greghaynes | or that | 02:01 |
| bkero | fungi: I think so | 02:01 |
| *** taron has joined #openstack-infra | 02:01 | |
| greghaynes | I was going to suggest adding a set -x at the top of disk-image-create, we can then see what du_size ends up getting set to | 02:01 |
| greghaynes | but you could also just check the partition size for both the images partition tables | 02:01 |
| clarkb | bkero: isnt that determined by where you boot not image? | 02:02 |
| clarkb | partition size is broken by grow root | 02:02 |
| fungi | yeah, block device size of the booted instances should be the same | 02:03 |
| greghaynes | clarkb: yea, youd have to look at it by converting the image and mounting on loopback | 02:03 |
| greghaynes | clarkb: before boot | 02:03 |
| clarkb | ok I just left computer for kids but maybe fungi can check that | 02:03 |
| fungi | i can give it a shot | 02:03 |
| greghaynes | dont let the kids get at that computer! | 02:03 |
| bkero | If the kids get ahold of those images they'll probably be 99999gb | 02:04 |
| fungi | convert them from qcow2 to raw? or do we keep raw copies of these around already | 02:04 |
| fungi | ? | 02:04 |
| greghaynes | fungi: Unsure if we keep raws, if we dont then yes they need to be converted to raw | 02:04 |
| bkero | can qemu-nbd loopback mount, then fdisk -l the loopback device | 02:04 |
| clarkb | we keep raws | 02:04 |
| fungi | trying to remember what directory it holds them in | 02:04 |
| clarkb | basically you need to nodepool dib image list to get filename mapping | 02:05 |
| *** aeng has quit IRC | 02:05 | |
| fungi | aha | 02:05 |
| clarkb | they are in /opt/nodepool_din | 02:05 |
| clarkb | er _dib | 02:05 |
| fungi | though it lies about the filenames | 02:06 |
| clarkb | oh its the id num | 02:07 |
| fungi | yep | 02:07 |
| fungi | let's call that a cosmetic bug for now | 02:07 |
| greghaynes | I may have to run in a few to deal with inlaws, but when we see that the partition sizes are significantly different the next question I'd have is how much of that space is used, and then id want to drop a bash in disk-image-create righ after it calculates du_size to see what is going on | 02:07 |
| *** Qiming has joined #openstack-infra | 02:07 | |
| *** annegentle has quit IRC | 02:07 | |
| *** annegentle has joined #openstack-infra | 02:08 | |
| *** yamamoto_ has joined #openstack-infra | 02:09 | |
| greghaynes | I think its possible that -minimal does more cleanup after a build, but we do cleanup after calculating root partition size so we overestimate what we need | 02:09 |
| bkero | Is that cleanup going to result in lower disk size though? | 02:09 |
| bkero | The dusage reported by the filesystem is the same | 02:09 |
| bkero | s/dusage/usage/ | 02:09 |
| greghaynes | bkero: itll result in a lower used space but the actual fs and partition size will not be smaller - we calculate the size of that before cleanup | 02:10 |
| *** thorst has quit IRC | 02:10 | |
| *** dimtruck is now known as zz_dimtruck | 02:10 | |
| *** thorst has joined #openstack-infra | 02:10 | |
| bkero | What I mean can disk compression reap that now-free space? | 02:11 |
| greghaynes | oh, thats up to qcow2 in this case, and I have no idea how effective that is | 02:12 |
| bkero | That's what I'm suspecting is causing the discrepency | 02:12 |
| *** annegentle has quit IRC | 02:12 | |
| fungi | i'm flailing mounting these raw images. mount wants a filesystem type... is there an mbr i need to offset past? | 02:12 |
| bkero | If I have a 5gb ubuntu qcow2, then dd 50gb of crap to it, then delete the crap, the resulting qcow2 will still be > 5gb | 02:13 |
| bkero | fungi: if the images have partitions, then kpartx -a $image, then look in /dev/mapper for the partitions | 02:13 |
| fungi | obvious filesystem types are erroring | 02:13 |
| greghaynes | fungi: usually once I make a loopback devide for that file the different partitions show up in /dev/loopXN | 02:13 |
| greghaynes | er, loopback device | 02:13 |
| fungi | oh, losetup it | 02:13 |
| greghaynes | yep | 02:14 |
| fungi | gotcha | 02:14 |
| bkero | (can losetup /dev/loop0 $FILE; kpartx -a /dev/loop0; mount /dev/mapper/loop0p1 /mnt) | 02:14 |
| fungi | mount -o loop not gonna work for raw disk. derp | 02:14 |
| bkero | Huh, I've never gotten loopback devices to make partition nodes by themselves before | 02:14 |
| greghaynes | Yea, its a pretty annoying series of hoops to jump through | 02:14 |
| bkero | libguestfs was supposed to help with that. Nerp. | 02:15 |
| *** yuanying_ has joined #openstack-infra | 02:15 | |
| *** lezbar__ has joined #openstack-infra | 02:15 | |
| *** doug-fis_ has joined #openstack-infra | 02:16 | |
| *** markmcd__ has joined #openstack-infra | 02:16 | |
| *** armax_ has joined #openstack-infra | 02:16 | |
| *** hrybacki_ has joined #openstack-infra | 02:16 | |
| fungi | they're basically the same: http://paste.openstack.org/show/489077/ | 02:17 |
| fungi | and no extra partitions in either one | 02:18 |
| bkero | yup | 02:18 |
| greghaynes | interesting | 02:18 |
| bkero | qcow2's just doing a better job shrinking or one has some deleted bloat during the install that qcow2 isn't compressing | 02:18 |
| greghaynes | so that points to compression simply being less effective for one | 02:18 |
| *** IlyaG has joined #openstack-infra | 02:18 | |
| fungi | strangely, ubuntu-trusty is using some 300mb more even though it should in theory be more minimal, but whatevs, these are not the bytes you're looking for | 02:18 |
| *** thorst has quit IRC | 02:18 | |
| *** SotK_ has joined #openstack-infra | 02:18 | |
| *** dosaboy_ has joined #openstack-infra | 02:18 | |
| bkero | were both images made at the same time on the same system? Maybe different versions of qemu-img resulted in different vresions of qcow2? | 02:19 |
| *** dangers_` has joined #openstack-infra | 02:19 | |
| *** jokke__ has joined #openstack-infra | 02:19 | |
| bkero | fungi: what does 'file' show for them? | 02:19 |
| *** elmiko_ has joined #openstack-infra | 02:19 | |
| *** aloga_ has joined #openstack-infra | 02:19 | |
| *** icey_ has joined #openstack-infra | 02:19 | |
| greghaynes | fungi: can you mount those partitions and see how much free space is in each? | 02:19 |
| fungi | bkero: "x86 boot sector" for both | 02:19 |
| bkero | fungi: I mean the qcow2 files | 02:20 |
| fungi | oh | 02:20 |
| bkero | They list versions in the magic file string | 02:20 |
| bkero | thing: QEMU QCOW Image (v3), 1073741824 bytes | 02:20 |
| fungi | /opt/nodepool_dib/176350.qcow2: QEMU QCOW Image (v2), 17681350656 bytes | 02:20 |
| fungi | /opt/nodepool_dib/176353.qcow2: QEMU QCOW Image (v2), 18188664832 bytes | 02:20 |
| bkero | Huh | 02:20 |
| greghaynes | I also wonder if we leave a lot more garbage in the fs for the -minimal image, possibly by using more space then deleting files at which point qcow2 would be less effective at compression becuase itll be random data rather then 0's | 02:20 |
| fungi | the first is the devstack-trusty i mounted, the second is the ubuntu-trusty i mounted | 02:20 |
| *** eliqiao_ has joined #openstack-infra | 02:20 | |
| *** dolpher1 has joined #openstack-infra | 02:20 | |
| fungi | well, the qcow2 versions of the raws i mounted anyway | 02:21 |
| *** alex_xu_ has joined #openstack-infra | 02:21 | |
| fungi | greghaynes: yeah, that's where i've been leaning | 02:21 |
| bkero | Yeah, that's what I've been saying. The FS won't zero deleted data, so it's not compressible. The deleted data will just sit there. | 02:21 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services https://review.openstack.org/271128 | 02:22 |
| bswartz | you can force the FS to zero empty blocks | 02:22 |
| *** claudiub|2 has quit IRC | 02:22 | |
| *** stevebak` has joined #openstack-infra | 02:22 | |
| *** sgordon` has joined #openstack-infra | 02:22 | |
| fungi | we can hackishly `cp /dev/zero /foo;rm foo` but is there a better way? | 02:22 |
| greghaynes | TIL there is a tool called zerofree | 02:22 |
| bkero | bswartz: do you know the tool for that? | 02:22 |
| bswartz | what fungi said | 02:22 |
| bkero | or the tune2fs option? | 02:22 |
| greghaynes | http://manpages.ubuntu.com/manpages/natty/man8/zerofree.8.html | 02:22 |
| bkero | lol | 02:22 |
| greghaynes | I wonder if we should try that | 02:22 |
| *** odyssey4me_ has joined #openstack-infra | 02:22 | |
| bswartz | dd if=/dev/zero of=/delete_me bs=1M | 02:22 |
| *** andreww has quit IRC | 02:23 | |
| bkero | Even then you might not be able to shrink it unless you convert to raw then back to qcow2 again | 02:23 |
| fungi | worth adding a zerofree element in dib (or just calling it directly in each minimal element?) | 02:23 |
| *** manfred has joined #openstack-infra | 02:23 | |
| bkero | Let me try that | 02:23 |
| *** xarses has joined #openstack-infra | 02:23 | |
| greghaynes | fungi: element would be great | 02:23 |
| *** xarses has quit IRC | 02:23 | |
| *** bhunter__ has joined #openstack-infra | 02:23 | |
| *** dosaboy_ has quit IRC | 02:23 | |
| bkero | zerofree only works on ext* fs fwiw | 02:23 |
| fungi | bkero: afaik we generate raw and then convert to qcow2 | 02:23 |
| *** zhurong has quit IRC | 02:23 | |
| bkero | right | 02:23 |
| greghaynes | bkero: yea, but thats what we are creating here | 02:23 |
| *** stevebaker has quit IRC | 02:23 | |
| *** mkoderer__ has quit IRC | 02:23 | |
| *** sgordon has quit IRC | 02:23 | |
| *** dangers_away has quit IRC | 02:23 | |
| *** therve has quit IRC | 02:23 | |
| *** odyssey4me has quit IRC | 02:23 | |
| *** gtmanfred has quit IRC | 02:23 | |
| *** dosaboy has quit IRC | 02:23 | |
| *** SotK has quit IRC | 02:23 | |
| *** armax has quit IRC | 02:23 | |
| *** yuanying has quit IRC | 02:23 | |
| *** beisner has quit IRC | 02:23 | |
| *** stevemar has quit IRC | 02:23 | |
| *** calebb has quit IRC | 02:23 | |
| *** jokke_ has quit IRC | 02:23 | |
| *** aloga has quit IRC | 02:23 | |
| *** markmcd1 has quit IRC | 02:23 | |
| *** hrybacki has quit IRC | 02:23 | |
| *** alex_xu has quit IRC | 02:23 | |
| *** eliqiao has quit IRC | 02:23 | |
| *** dolpher has quit IRC | 02:23 | |
| *** elmiko has quit IRC | 02:23 | |
| *** doug-fish has quit IRC | 02:23 | |
| *** bhunter_ has quit IRC | 02:23 | |
| *** icey has quit IRC | 02:23 | |
| *** lezbar has quit IRC | 02:23 | |
| *** manfred is now known as gtmanfred | 02:23 | |
| *** mkoderer___ has joined #openstack-infra | 02:23 | |
| *** armax_ is now known as armax | 02:23 | |
| *** calebb1 has joined #openstack-infra | 02:23 | |
| *** dolpher1 is now known as dolpher | 02:23 | |
| *** therve has joined #openstack-infra | 02:23 | |
| fungi | so adding a step to zero free space on the filesystem before finalizing the raw ought to do it if this is the culprit | 02:24 |
| *** beisner has joined #openstack-infra | 02:24 | |
| *** stevemar has joined #openstack-infra | 02:24 | |
| *** zhurong has joined #openstack-infra | 02:24 | |
| *** dosaboy has joined #openstack-infra | 02:24 | |
| fungi | however it's getting late here so this is an exercise better left for tomorrow unless someone else wants to take up the torch | 02:24 |
| *** doug-fis_ has quit IRC | 02:24 | |
| *** yamahata has quit IRC | 02:24 | |
| greghaynes | fungi: yep, and even if thats not the fix I would love that feature in dib :) | 02:24 |
| bkero | sounds like a good dib finalize step | 02:24 |
| *** hrybacki_ is now known as hrybacki | 02:24 | |
| *** roxanagh_ has joined #openstack-infra | 02:24 | |
| *** xarses has joined #openstack-infra | 02:24 | |
| * greghaynes needs to family so will wait for tomorrow | 02:25 | |
| *** xarses has quit IRC | 02:25 | |
| fungi | most of the distros do this by default in their official images already and i used to use the same copy /dev/zero trick then delete when i made my xen base images years ago, so sort of assumed this was already something dib did too | 02:25 |
| *** jcooley_ has quit IRC | 02:26 | |
| fungi | actually surprised to learn it does not | 02:26 |
| fungi | and definitely worth fixing | 02:26 |
| *** xarses has joined #openstack-infra | 02:26 | |
| greghaynes | Yea, AFAIK it does not | 02:26 |
| *** doug-fish has joined #openstack-infra | 02:27 | |
| bswartz | guys I actually have a question about images and disk image builder -- hoping to learn from your wisdom | 02:27 |
| *** xarses has quit IRC | 02:27 | |
| bswartz | manila builds some images -- we have a project called manila-image-elements | 02:27 |
| *** xarses_ has joined #openstack-infra | 02:27 | |
| bswartz | however the output from those builds doesn't get saved anywhere official | 02:27 |
| fungi | but yeah, if the difference between the ubuntu and ubuntu-minimal elements is "mount a distro provided image" vs "run debootstrap" then yes they've done most of the cleanup for us in the first scenario | 02:27 |
| *** xarses_ has quit IRC | 02:27 | |
| bswartz | is there an official repository where we could be uploading images? | 02:28 |
| fungi | bswartz: closest thing we have to that is https://tarballs.openstack.org/ | 02:28 |
| bswartz | ideally something similar to the tarballs site that just take the output of the build after every commit and saves it? | 02:28 |
| fungi | (it has more than tarballs, but not much more) | 02:28 |
| *** xarses_ has joined #openstack-infra | 02:28 | |
| *** sarob has quit IRC | 02:28 | |
| bswartz | okay how do I figure out how to make a job that uploads disk image builder qcows to the tarballs site? | 02:29 |
| bswartz | is there an example I can follow? | 02:29 |
| fungi | bswartz: what size are the images? how many commits do you merge? what do you expect the year-over-year growth would be for storage? | 02:29 |
| fungi | we're sort of limited since we're stuck using a filesystem on a logical volume spread across cinder devices right now, so max out around 14tb | 02:30 |
| bswartz | well the project was started 9 months ago and we have 36 commits... https://github.com/openstack/manila-image-elements | 02:30 |
| *** sputnik13 has quit IRC | 02:30 | |
| bswartz | so not a ton of traffic | 02:30 |
| bswartz | let me check on size | 02:30 |
| *** andreww has joined #openstack-infra | 02:30 | |
| fungi | but as far as images, for example ironic-python-agent builds disk images it uses in its own testing and stashes them on tarballs.o.o, if you're looking for an example | 02:31 |
| bswartz | looks like the images we're building weigh in at 300MB currently | 02:31 |
| bswartz | fungi: thanks | 02:32 |
| bkero | Phew, zerofree takes a while to run. | 02:32 |
| fungi | bswartz: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/ironic-python-agent-jobs.yaml | 02:32 |
| fungi | bswartz: though they don't keep them all, they just replace each time http://tarballs.openstack.org/ironic-python-agent/coreos/ | 02:33 |
| *** yamamoto_ has quit IRC | 02:33 | |
| bkero | So I created a 10gb qcow2 volume, exported with qemu-nbd, mkfs.ext4'd, mounted, then dd if=/dev/unrandom size=1G to it. The size rose from 133mb (ext metadata) to 1300mb. After running zerofree the size is still 1300mb. If I use qemu-img to convert from qcow2 to qcow2 the resulting image is 896k. | 02:33 |
| *** xarses_ has quit IRC | 02:33 | |
| bswartz | well I'd like tagged commits to get saved -- similar to how actual tarballs work | 02:34 |
| bswartz | but ordinary commits could get clobbered | 02:34 |
| clarkb | bkero so zerofree in diblikely would help | 02:34 |
| bswartz | or reaped after some max number | 02:34 |
| fungi | bswartz: yep, that's doable... just base the image name off something like $ZUUL_REFNAME and only run the versioned builds in the release pipeline | 02:34 |
| bkero | clarkb: try this: qemu-img convert -O qcow2 -c oldimage newimage | 02:34 |
| bkero | See if the filesize changes | 02:35 |
| fungi | clarkb: yeah, if greghaynes is right that we don't zero deleted garbage in the fs, starting from canonical's image (where they most certainly have done that already) vs running debootstrap (which does a ton of file deletion) will result in significantly different compressed sizes | 02:36 |
| *** rhallisey has quit IRC | 02:36 | |
| fungi | but even freeing in the devstack-trusty images will probably get us back some compressed image size there too | 02:37 |
| *** eliqiao_ is now known as eliqiao | 02:37 | |
| fungi | i'm sure some of the elements we're running besides ubuntu-minimal are deleting at least some files they temporarily create | 02:38 |
| fungi | so this needs to be a very generic element run in dib's finalizing steps for pretty much any image it creates ever | 02:39 |
| bswartz | how many zeros are on these disk images? are they like 10% full of data? 50%? 90%? | 02:39 |
| fungi | call it 25% in the devstack-trusty image and 50% in the ubuntu-trusty image | 02:40 |
| bkero | clarkb: so I created a 10gb raw image, mkfs.ext4'd it, mounted it, dd'd a random 1gb file to it. Resulting image size was 1.2GB. Did a qemu-img convert to qcow2 on it. Resulting qcow2 was 1001MB. Unmounted the raw image, ran zerofree. Ran a qemu-convert to qcow2 again. resulting size was 896k. | 02:40 |
| bswartz | for disk images with tons of free space it could be more efficient to "cp -a" the whole filesystem to a new empty filesystem of the same size rather than zeroing the free space | 02:40 |
| bkero | So the proof of concept shows that zerofree-ing a raw image, then converting to qcow2 will recover all deleted file space \o/ | 02:40 |
| fungi | bswartz: agreed, you get more contiguousness that way | 02:41 |
| fungi | fragmentation makes zeroing and then compressing less effective | 02:41 |
| bkero | That's true | 02:41 |
| clarkb | bkero: you going to write the dib patch? not sure if it makes sense in an element or in dib itself | 02:41 |
| fungi | for some reason i thought dib did that (via an intermediate tarball) | 02:41 |
| fungi | but my information is likely quite outdated/inaccurate | 02:42 |
| bkero | clarkb: Likely a package-install somewhere, then a finalize | 02:42 |
| bkero | I can do it | 02:42 |
| clarkb | no dib is raw to its outputs | 02:42 |
| clarkb | well its fancy chroot to raw to else | 02:42 |
| fungi | i may even be remembering some of the preliminary design discussions in this case, from before dib was really a thing | 02:43 |
| clarkb | if you ask for tarball you do chroot to tsrball | 02:43 |
| *** rhallisey has joined #openstack-infra | 02:44 | |
| *** rhallisey has quit IRC | 02:44 | |
| fungi | but bswartz has a great point, we're better off copying the content to a fresh filesystem and then compressing instead of trying to zero the cruft | 02:44 |
| mnaser | maybe im poking too much at things but | 02:44 |
| *** Shiv_ has joined #openstack-infra | 02:45 | |
| mnaser | http://grafana.openstack.org/dashboard/db/nodepool-vexxhost | 02:45 |
| *** gongysh has joined #openstack-infra | 02:45 | |
| *** Shiv_ has left #openstack-infra | 02:45 | |
| mnaser | is the queue just idle so it's running at 22 machines only? | 02:45 |
| bswartz | if it's a whole disk image though, you need to worry about duplicating the partition table, boot sector, etc | 02:45 |
| bkero | That doesn't sound fun | 02:45 |
| clarkb | mnaser: ya if demand falls use will fall and its proportional to quota | 02:46 |
| mnaser | gotcha | 02:46 |
| bswartz | it's not as bad as you think | 02:46 |
| clarkb | so dib does do the copy already | 02:46 |
| clarkb | dib works by building in a chroot | 02:46 |
| bswartz | do these disk images have swap partitions by any chance? | 02:46 |
| *** tphummel has quit IRC | 02:46 | |
| fungi | oh, it works in a chroot then copies to a loopback at the end? | 02:47 |
| clarkb | then towards the end it copies everything into a raw disk image loopback mounted | 02:47 |
| clarkb | then it converts to qcow2 or whatever | 02:47 |
| greghaynes | fungi: bswartz oh, dib does do a cp of the built image, I need to remember where though | 02:47 |
| greghaynes | which might be why we dont zero the fs | 02:47 |
| fungi | hrm, then this may be not the problem | 02:47 |
| *** resker has quit IRC | 02:48 | |
| greghaynes | yea, so it does an mv of the built artifacts right before finalise.d runs | 02:48 |
| greghaynes | So I think that theory isnt valid :( | 02:49 |
| bswartz | maybe all the extra space is taken up by the NSA rootkit | 02:49 |
| greghaynes | haha | 02:49 |
| * bswartz dons tinfoil hat | 02:49 | |
| fungi | well, the nsa _is_ inefficient, but i don't think even their rootkits take up 5gb | 02:50 |
| greghaynes | it is possible that stuff happens in finalise.d which creates this issue, but I would hope there isn't much going on there | 02:50 |
| bkero | greghaynes: around line 352 in disk-image-create? | 02:50 |
| greghaynes | bkero: 348 | 02:51 |
| bkero | yup | 02:51 |
| greghaynes | haha | 02:52 |
| *** esker has joined #openstack-infra | 02:52 | |
| greghaynes | so, apt-get clean runs in finalise.d | 02:52 |
| fungi | oh | 02:52 |
| bkero | gg | 02:52 |
| fungi | though we're disabling that i think | 02:52 |
| bkero | I'm assuming finalise.d is run after things are copied to the image | 02:52 |
| greghaynes | bkero: yea, see line 350 | 02:53 |
| *** sobersabre has quit IRC | 02:53 | |
| fungi | because we set an envvar to tell it not to clear the apt cache | 02:53 |
| greghaynes | fungi: ah, ok | 02:53 |
| greghaynes | Maybe a good test would be to run zerofree on one of the images after the fact and see what happens when converted back to qcow2 | 02:53 |
| fungi | since building up the apt cache (via --download-only) is one of the main reasons we make out own images | 02:54 |
| greghaynes | just to rule out that theory or not | 02:54 |
| fungi | greghaynes: yep, i concur, just not me tonight since i don't feel like figuring out the qcow2 conversion command line this late at night | 02:54 |
| bkero | qemu-nbd -c /dev/nbd0 oldimage.qcow2; kpartx -a /dev/nbd0; zerofree /dev/mapper/nbd0p1; qemu-nbd -d /dev/nbd0; qemu-img convert -O qcow2 -c oldimage.qcow2 newimage.qcow2 | 02:54 |
| bkero | compare filesizes | 02:54 |
| *** gongysh has quit IRC | 02:54 | |
| greghaynes | fungi: hah, totally understand that :) | 02:55 |
| fungi | though apparently bkero has it all memorized anyway | 02:55 |
| bkero | :) | 02:55 |
| bkero | I didn't think those life skills would ever come in handy. | 02:55 |
| *** camunoz has quit IRC | 02:55 | |
| fungi | amuse your friends. fun at parties | 02:56 |
| bkero | A blast at parties, let me tell you | 02:56 |
| fungi | disk image conversions in between keg stands | 02:56 |
| greghaynes | I need to find better parties | 02:57 |
| * fungi remembers all the keg stands he built in college. they have to be sturdy to support all that beer, you know | 02:58 | |
| bkero | Maybe I'm failing at life, but I don't remember building any. | 02:59 |
| clarkb | greghaynes: the only parties you get to go to now involve small children | 02:59 |
| *** sam_wan has quit IRC | 03:00 | |
| bkero | clarkb: sometimes they involve brisket | 03:00 |
| fungi | corralling toddlers and trying not to look zippy the clown directly in the eyes | 03:00 |
| Clint | zippy | 03:01 |
| * Clint shakes fist | 03:01 | |
| cj | hey folks | 03:01 |
| bkero | howdy | 03:02 |
| greghaynes | clarkb: wait until the kids learn to convert diskimages | 03:02 |
| cj | http://docs.openstack.org/developer/pbr/ <- under "entry_points" here, I'm confused as to what I should put... | 03:02 |
| cj | I'm trying to create a setup.cfg for python-irclib | 03:02 |
| fungi | greghaynes: wait until the kids learn to do keg stands | 03:02 |
| cj | the first for the 'irccat' script... | 03:03 |
| * bkero gives up on computer and goes to find a dinner. | 03:03 | |
| cj | there's no main entrypoint there, since it executes the code directly | 03:03 |
| *** sridhar_ram has quit IRC | 03:03 | |
| fungi | cj: so no main() you can call into? | 03:03 |
| cj | what value do I use in this case? I don't want pbr to create a new script for me, I just want it to take the one that's there and put it in /usr/local/bin or whatever the prefix is set to | 03:03 |
| greghaynes | fungi: haha, I'm not sure which is scarier | 03:04 |
| cj | fungi: https://github.com/jbalogh/python-irclib/blob/master/irccat | 03:04 |
| fungi | mordred: ^ if you're around, do you recall how pbr-ized packages are expected to deal with directly installing executable scripts? | 03:06 |
| fungi | is it as simple as using a file declaration for them, or is more magic required? | 03:06 |
| *** apoorvad has quit IRC | 03:07 | |
| *** rguillebert has quit IRC | 03:07 | |
| *** camunoz has joined #openstack-infra | 03:08 | |
| *** bpokorny has quit IRC | 03:08 | |
| *** sobersabre has joined #openstack-infra | 03:08 | |
| fungi | cj: my first guess would be in the [files] section do "datafiles = irccat = bin/irccat" or something along those lines. i'm not immediately finding any really great examples since we tend to rely heavily on entrypoints | 03:10 |
| *** doug-fish has quit IRC | 03:10 | |
| fungi | lifeless: it's possible you're more awake than mordred this time of day, perhaps you happen to know the answer to cj's pbr question? | 03:10 |
| mordred | yeah. we pretty much don't do that | 03:11 |
| mordred | I'm sure it's possible some how | 03:11 |
| mordred | but I'm not really sure why you'd do it | 03:11 |
| *** achanda has joined #openstack-infra | 03:11 | |
| mordred | :) | 03:11 |
| fungi | right, pbr's design expects heavily libbish packages where the only executable scripts are very lightweight wrappers calling into library functions | 03:12 |
| *** ashleighfarnham has quit IRC | 03:12 | |
| fungi | so you're likely stuck relying on whatever raw file copying solutions it has if you need that | 03:13 |
| fungi | and files.data_files is the closest thing that springs to mind | 03:14 |
| *** thorst has joined #openstack-infra | 03:16 | |
| craige | heya infra root people, is there somewhere I can go to get an eye on the state of phabricator.openstack.org? | 03:16 |
| *** sam_wan has joined #openstack-infra | 03:17 | |
| *** fedexo has joined #openstack-infra | 03:20 | |
| *** jswarren_ has joined #openstack-infra | 03:21 | |
| *** MCoLo___ has joined #openstack-infra | 03:21 | |
| *** beisner- has joined #openstack-infra | 03:21 | |
| *** yuanying has joined #openstack-infra | 03:21 | |
| *** beisner has quit IRC | 03:22 | |
| *** beisner- is now known as beisner | 03:22 | |
| *** markmcd1 has joined #openstack-infra | 03:22 | |
| *** mwhahaha_ has joined #openstack-infra | 03:23 | |
| *** jraim_ has joined #openstack-infra | 03:23 | |
| *** kmartin__ has joined #openstack-infra | 03:23 | |
| *** thorst has quit IRC | 03:24 | |
| *** nijaba_ has joined #openstack-infra | 03:24 | |
| *** ddieterl_ has joined #openstack-infra | 03:24 | |
| *** sulo_ has joined #openstack-infra | 03:24 | |
| *** yamamoto_ has joined #openstack-infra | 03:25 | |
| *** bodepd_ has joined #openstack-infra | 03:25 | |
| *** dansmith_ has joined #openstack-infra | 03:25 | |
| *** mikap has joined #openstack-infra | 03:25 | |
| *** f1ller has joined #openstack-infra | 03:25 | |
| *** markmcclain1 has joined #openstack-infra | 03:25 | |
| *** sambetts has joined #openstack-infra | 03:26 | |
| *** jvrbanac_ has joined #openstack-infra | 03:26 | |
| *** dtantsur has joined #openstack-infra | 03:26 | |
| *** mestery_ has joined #openstack-infra | 03:26 | |
| *** NikitaKonovalov2 has joined #openstack-infra | 03:26 | |
| *** bradm_ has joined #openstack-infra | 03:26 | |
| *** Odd_Blok1 has joined #openstack-infra | 03:26 | |
| *** jraim has quit IRC | 03:26 | |
| *** laurelm has quit IRC | 03:26 | |
| *** dansmith has quit IRC | 03:26 | |
| *** MCoLo__ has quit IRC | 03:26 | |
| *** mwhahaha has quit IRC | 03:26 | |
| *** sambetts|afk has quit IRC | 03:26 | |
| *** mrunge has quit IRC | 03:26 | |
| *** igormarnat has quit IRC | 03:26 | |
| *** dtantsur|afk has quit IRC | 03:26 | |
| *** camunoz has quit IRC | 03:26 | |
| *** markmcd__ has quit IRC | 03:26 | |
| *** vponomaryov has quit IRC | 03:26 | |
| *** LinuxJedi has quit IRC | 03:26 | |
| *** jvrbanac has quit IRC | 03:26 | |
| *** sulo has quit IRC | 03:26 | |
| *** markmcclain has quit IRC | 03:26 | |
| *** Odd_Bloke has quit IRC | 03:26 | |
| *** filler has quit IRC | 03:26 | |
| *** jswarren has quit IRC | 03:26 | |
| *** nijaba has quit IRC | 03:26 | |
| *** NikitaKonovalov has quit IRC | 03:26 | |
| *** ZZelle has quit IRC | 03:26 | |
| *** kmartin_ has quit IRC | 03:26 | |
| *** Mmike has quit IRC | 03:26 | |
| *** gibi has quit IRC | 03:26 | |
| *** mnaser has quit IRC | 03:26 | |
| *** vdrok_ has quit IRC | 03:26 | |
| *** clayton has quit IRC | 03:26 | |
| *** dolpher has quit IRC | 03:26 | |
| *** IlyaG has quit IRC | 03:26 | |
| *** yuanying_ has quit IRC | 03:26 | |
| *** ddieterly has quit IRC | 03:26 | |
| *** msuriar has quit IRC | 03:26 | |
| *** ildikov has quit IRC | 03:26 | |
| *** lxsli has quit IRC | 03:26 | |
| *** lennyb has quit IRC | 03:26 | |
| *** ajo has quit IRC | 03:26 | |
| *** HenryG has quit IRC | 03:26 | |
| *** hamzy has quit IRC | 03:26 | |
| *** Roamer` has quit IRC | 03:26 | |
| *** mika has quit IRC | 03:26 | |
| *** bodepd has quit IRC | 03:26 | |
| *** briancline has quit IRC | 03:26 | |
| *** IlyaG has joined #openstack-infra | 03:26 | |
| *** Roamer` has joined #openstack-infra | 03:26 | |
| *** leifmadsen has quit IRC | 03:26 | |
| *** zaro has quit IRC | 03:26 | |
| *** hamzy has joined #openstack-infra | 03:26 | |
| *** mestery has quit IRC | 03:26 | |
| *** briancline has joined #openstack-infra | 03:26 | |
| *** bradm has quit IRC | 03:26 | |
| *** mrunge_ has joined #openstack-infra | 03:26 | |
| *** igormarnat_ has joined #openstack-infra | 03:26 | |
| *** yamahata has joined #openstack-infra | 03:26 | |
| *** NikitaKonovalov2 is now known as NikitaKonovalov | 03:26 | |
| *** mikap is now known as mika | 03:26 | |
| *** LinuxJedi_ has joined #openstack-infra | 03:26 | |
| *** markmcclain1 is now known as markmcclain | 03:26 | |
| *** HenryG has joined #openstack-infra | 03:26 | |
| *** msuriar has joined #openstack-infra | 03:26 | |
| *** igormarnat_ is now known as igormarnat | 03:26 | |
| *** ajo has joined #openstack-infra | 03:26 | |
| *** jraim_ is now known as jraim | 03:26 | |
| *** vdrok_ has joined #openstack-infra | 03:27 | |
| *** LinuxJedi_ is now known as LinuxJedi | 03:27 | |
| *** dansmith_ is now known as dansmith | 03:27 | |
| *** ildikov has joined #openstack-infra | 03:27 | |
| *** woodster_ has quit IRC | 03:27 | |
| *** vponomaryov has joined #openstack-infra | 03:27 | |
| *** camunoz has joined #openstack-infra | 03:27 | |
| *** laurelm has joined #openstack-infra | 03:27 | |
| *** mwhahaha_ is now known as mwhahaha | 03:27 | |
| *** lennyb has joined #openstack-infra | 03:27 | |
| *** Mmike has joined #openstack-infra | 03:27 | |
| *** ZZelle has joined #openstack-infra | 03:27 | |
| *** dolpher has joined #openstack-infra | 03:27 | |
| *** dansmith is now known as Guest60497 | 03:27 | |
| *** leifmadsen has joined #openstack-infra | 03:27 | |
| *** clayton has joined #openstack-infra | 03:28 | |
| *** mestery_ is now known as mestery | 03:28 | |
| *** gibi has joined #openstack-infra | 03:28 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 03:29 |
| *** links has joined #openstack-infra | 03:29 | |
| *** mnaser has joined #openstack-infra | 03:30 | |
| *** ansiwen has quit IRC | 03:31 | |
| *** ansiwen has joined #openstack-infra | 03:31 | |
| *** christiana has quit IRC | 03:31 | |
| *** aeng has joined #openstack-infra | 03:34 | |
| *** lennyb__ has joined #openstack-infra | 03:35 | |
| *** tojuvone_ has joined #openstack-infra | 03:36 | |
| *** pfallenop has quit IRC | 03:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:37 | |
| *** [1]Thelo has joined #openstack-infra | 03:39 | |
| *** mrunge_ has quit IRC | 03:39 | |
| *** lennyb has quit IRC | 03:40 | |
| *** tojuvone has quit IRC | 03:40 | |
| *** Thelo has quit IRC | 03:41 | |
| *** [1]Thelo is now known as Thelo | 03:41 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 03:43 |
| *** pfallenop has joined #openstack-infra | 03:44 | |
| *** baoli has joined #openstack-infra | 03:45 | |
| *** mrunge has joined #openstack-infra | 03:45 | |
| *** baoli has quit IRC | 03:47 | |
| *** dims has quit IRC | 03:48 | |
| openstackgerrit | Bob Haddleton proposed openstack/requirements: Add heat-translator>=0.4.0 https://review.openstack.org/287581 | 03:48 |
| *** bradm_ is now known as bradm | 03:49 | |
| *** keedya has joined #openstack-infra | 03:51 | |
| *** lxsli has joined #openstack-infra | 03:52 | |
| *** eric_lopez has quit IRC | 03:56 | |
| *** eric_lopez has joined #openstack-infra | 03:56 | |
| *** camunoz has quit IRC | 03:58 | |
| mnaser | interesting metric of seeing how going from 1 to 25 servers brings the list servers response time from ~0.25s to ~0.5s | 03:59 |
| *** doug-fish has joined #openstack-infra | 04:00 | |
| *** doug-fish has quit IRC | 04:05 | |
| *** kzaitsev_mb has quit IRC | 04:05 | |
| *** camunoz has joined #openstack-infra | 04:11 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 04:15 |
| *** Mmike has quit IRC | 04:15 | |
| *** Mmike has joined #openstack-infra | 04:15 | |
| *** mriedem has quit IRC | 04:16 | |
| *** IlyaG has quit IRC | 04:17 | |
| *** sam_wan has quit IRC | 04:18 | |
| *** amrith is now known as _amrith_ | 04:19 | |
| *** woodster_ has joined #openstack-infra | 04:20 | |
| *** marg7175_ has quit IRC | 04:21 | |
| *** thorst has joined #openstack-infra | 04:21 | |
| *** sam_wan has joined #openstack-infra | 04:22 | |
| *** harlowja_at_home has quit IRC | 04:27 | |
| *** Jeffrey4l has joined #openstack-infra | 04:27 | |
| *** achanda has quit IRC | 04:28 | |
| *** Daisy has joined #openstack-infra | 04:28 | |
| *** thorst has quit IRC | 04:29 | |
| *** bhunter__ has quit IRC | 04:30 | |
| *** tiswanso has quit IRC | 04:33 | |
| *** Daisy has quit IRC | 04:34 | |
| *** Daisy has joined #openstack-infra | 04:34 | |
| *** denisra has quit IRC | 04:38 | |
| *** denisra has joined #openstack-infra | 04:38 | |
| *** yonglihe has quit IRC | 04:42 | |
| *** roxanagh_ has quit IRC | 04:44 | |
| *** resker has joined #openstack-infra | 04:45 | |
| *** esker has quit IRC | 04:47 | |
| *** esikache1 has joined #openstack-infra | 04:51 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 04:52 |
| *** rossella_s has quit IRC | 04:53 | |
| *** rossella_s has joined #openstack-infra | 04:54 | |
| *** esikache1 has quit IRC | 04:56 | |
| *** Jeffrey4l has quit IRC | 04:59 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:02 | |
| *** jamespage has quit IRC | 05:05 | |
| *** gnuoy has quit IRC | 05:05 | |
| *** gnuoy has joined #openstack-infra | 05:06 | |
| *** jamespage has joined #openstack-infra | 05:06 | |
| *** kzaitsev_mb has quit IRC | 05:07 | |
| lifeless | fungi: what was cj's pbr question ? | 05:10 |
| lifeless | cj: ^ | 05:10 |
| *** achanda has joined #openstack-infra | 05:11 | |
| *** sridhar_ram has joined #openstack-infra | 05:13 | |
| *** eliqiao_ has joined #openstack-infra | 05:13 | |
| *** eliqiao has quit IRC | 05:16 | |
| *** alex_xu has joined #openstack-infra | 05:18 | |
| *** salv-orl_ has joined #openstack-infra | 05:19 | |
| *** roxanaghe has joined #openstack-infra | 05:20 | |
| *** maishsk has joined #openstack-infra | 05:20 | |
| *** alex_xu_ has quit IRC | 05:21 | |
| *** oanson has joined #openstack-infra | 05:21 | |
| fungi | lifeless: he wanted to know if pbr had a mechanism for directly installing an executable script rather than relying on an entrypoint | 05:22 |
| *** salv-orlando has quit IRC | 05:22 | |
| lifeless | we do; its just he underlying setuptools stuff. | 05:23 |
| lifeless | Don't use it. | 05:23 |
| *** zaro has joined #openstack-infra | 05:25 | |
| *** thorst has joined #openstack-infra | 05:27 | |
| *** claudiub|2 has joined #openstack-infra | 05:28 | |
| *** noorul has joined #openstack-infra | 05:28 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-translation_checksite: Puppet module to provide a translation checksite https://review.openstack.org/274701 | 05:29 |
| *** sarob has joined #openstack-infra | 05:29 | |
| *** Douhet has quit IRC | 05:31 | |
| *** sam_wan has quit IRC | 05:31 | |
| *** Douhet has joined #openstack-infra | 05:32 | |
| *** thorst has quit IRC | 05:33 | |
| *** sarob has quit IRC | 05:33 | |
| *** claudiub|2 has quit IRC | 05:34 | |
| *** kiran-r has joined #openstack-infra | 05:37 | |
| *** maishsk has quit IRC | 05:39 | |
| *** sobersabre has quit IRC | 05:43 | |
| *** Daisy has quit IRC | 05:44 | |
| *** Daisy has joined #openstack-infra | 05:44 | |
| *** abregman has joined #openstack-infra | 05:46 | |
| *** Daisy has quit IRC | 05:46 | |
| *** Daisy has joined #openstack-infra | 05:47 | |
| *** sarob has joined #openstack-infra | 05:50 | |
| *** alex_xu_ has joined #openstack-infra | 05:50 | |
| *** _sarob has joined #openstack-infra | 05:51 | |
| *** Daisy has quit IRC | 05:51 | |
| *** Daisy has joined #openstack-infra | 05:52 | |
| *** alex_xu has quit IRC | 05:53 | |
| *** sarob has quit IRC | 05:54 | |
| *** Daisy has quit IRC | 05:55 | |
| *** Daisy has joined #openstack-infra | 05:55 | |
| *** Daisy has quit IRC | 05:56 | |
| *** Daisy has joined #openstack-infra | 05:56 | |
| *** gildub has quit IRC | 05:56 | |
| *** gildub has joined #openstack-infra | 05:57 | |
| *** baoli has joined #openstack-infra | 05:59 | |
| *** sdake has joined #openstack-infra | 05:59 | |
| *** resker has quit IRC | 06:01 | |
| prometheanfire | could someone tell me why this isn't getting workflowed? https://review.openstack.org/#/c/281960/ | 06:02 |
| prometheanfire | SpamapS seems to be absent | 06:02 |
| *** kzaitsev_mb has joined #openstack-infra | 06:03 | |
| *** baoli has quit IRC | 06:04 | |
| *** esker has joined #openstack-infra | 06:05 | |
| greghaynes | prometheanfire: done | 06:05 |
| greghaynes | prometheanfire: he is on leave ATM | 06:05 |
| prometheanfire | ah, thanks | 06:06 |
| prometheanfire | at the time it was behind another, dependant commit | 06:07 |
| openstackgerrit | Merged openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 06:07 |
| *** armax has quit IRC | 06:07 | |
| prometheanfire | huh, that was quick | 06:08 |
| greghaynes | Yea, for doc-only changes we short-circuit some of the testing | 06:08 |
| *** kzaitsev_mb has quit IRC | 06:09 | |
| prometheanfire | nice | 06:09 |
| prometheanfire | wish we did that in rpc-openstack | 06:10 |
| prometheanfire | waiting 1.5hr for jenkins to fail isn't fun | 06:10 |
| *** gongysh has joined #openstack-infra | 06:10 | |
| *** sam_wan has joined #openstack-infra | 06:13 | |
| *** yonglihe has joined #openstack-infra | 06:14 | |
| *** sdake has quit IRC | 06:16 | |
| *** Thelo has quit IRC | 06:23 | |
| *** woodster_ has quit IRC | 06:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/287613 | 06:29 |
| *** thorst has joined #openstack-infra | 06:31 | |
| *** gongysh has quit IRC | 06:32 | |
| *** amit213 has quit IRC | 06:33 | |
| *** jtomasek has joined #openstack-infra | 06:33 | |
| *** x00350071 is now known as xiangxinyong | 06:35 | |
| *** sridhar_ram has quit IRC | 06:35 | |
| *** cfarquhar has quit IRC | 06:37 | |
| *** eliqiao_ is now known as eliqiao | 06:39 | |
| *** thorst has quit IRC | 06:39 | |
| *** cfarquhar has joined #openstack-infra | 06:40 | |
| *** cfarquhar has quit IRC | 06:40 | |
| *** cfarquhar has joined #openstack-infra | 06:40 | |
| *** tojuvone_ is now known as tojuvone | 06:44 | |
| *** gildub has quit IRC | 06:53 | |
| *** gildub has joined #openstack-infra | 06:53 | |
| *** roxanaghe has quit IRC | 06:53 | |
| *** esikache1 has joined #openstack-infra | 06:54 | |
| *** camunoz has quit IRC | 06:54 | |
| *** roxanaghe has joined #openstack-infra | 06:55 | |
| *** esikache1 has quit IRC | 06:58 | |
| *** esikachev has joined #openstack-infra | 06:59 | |
| *** aeng has quit IRC | 07:00 | |
| *** sam_wan has quit IRC | 07:02 | |
| *** gnuoy_ has joined #openstack-infra | 07:03 | |
| *** rlandy has quit IRC | 07:04 | |
| *** amit213 has joined #openstack-infra | 07:04 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:04 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287620 | 07:05 |
| *** gnuoy_ has quit IRC | 07:06 | |
| *** amit213 has quit IRC | 07:06 | |
| *** amit213 has joined #openstack-infra | 07:08 | |
| *** kzaitsev_mb has quit IRC | 07:09 | |
| *** amit213 has quit IRC | 07:10 | |
| *** Daisy has quit IRC | 07:13 | |
| *** Daisy has joined #openstack-infra | 07:13 | |
| *** jaosorior has joined #openstack-infra | 07:15 | |
| *** amit213 has joined #openstack-infra | 07:15 | |
| *** Daisy has quit IRC | 07:18 | |
| *** amit213 has quit IRC | 07:18 | |
| *** scheuran has joined #openstack-infra | 07:20 | |
| openstackgerrit | Merged openstack-infra/project-config: midonet: Rename rally job so that it matches regex for doc-only changes https://review.openstack.org/286956 | 07:23 |
| AJaeger | Nakato: are you still around? Problem with post jobs and constraints is a limitation of zuul-cloner, zuul-cloner will not setup the repository correctly. We need to enhance zuul-cloner first | 07:24 |
| *** nmagnezi has joined #openstack-infra | 07:26 | |
| *** claudiub|2 has joined #openstack-infra | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable pep8 job on tripleo-heat-templates https://review.openstack.org/258122 | 07:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add freezer-api job for tempest tests https://review.openstack.org/287486 | 07:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add total singlestats for nodepool dashboard https://review.openstack.org/287469 | 07:27 |
| *** claudiub|2 has quit IRC | 07:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Make Packstack integration jobs voting, add to gate https://review.openstack.org/287461 | 07:30 |
| *** achanda has quit IRC | 07:31 | |
| *** zhurong has quit IRC | 07:32 | |
| *** zhurong has joined #openstack-infra | 07:34 | |
| *** fedexo has quit IRC | 07:34 | |
| *** boris-42 has quit IRC | 07:34 | |
| openstackgerrit | Sergey Kraynev proposed openstack/requirements: Bump upper-constraint for heatclient to 1.0.0 https://review.openstack.org/287351 | 07:35 |
| *** boris-42 has joined #openstack-infra | 07:35 | |
| eliqiao | hello, does anyone know why my in PYTHON, /usr/lib appears before /usr/local/lib ? | 07:35 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Only run jjb if that is not already running https://review.openstack.org/284462 | 07:35 |
| *** korzen has joined #openstack-infra | 07:37 | |
| *** thorst has joined #openstack-infra | 07:37 | |
| *** maishsk has joined #openstack-infra | 07:39 | |
| *** amit213 has joined #openstack-infra | 07:40 | |
| openstackgerrit | Csaba Henk proposed openstack-infra/project-config: Add Manila with GlusterFS driver and Heketi layout CI job https://review.openstack.org/287632 | 07:40 |
| *** amit213 has quit IRC | 07:42 | |
| *** jcoufal has joined #openstack-infra | 07:42 | |
| *** thorst has quit IRC | 07:43 | |
| *** andymaier has joined #openstack-infra | 07:44 | |
| *** _sarob has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-infra | 07:45 | |
| *** k4n0 has joined #openstack-infra | 07:46 | |
| *** roxanaghe has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Add openstack client command overlap checking https://review.openstack.org/284497 | 07:47 |
| *** k4n0 has quit IRC | 07:48 | |
| *** achanda has joined #openstack-infra | 07:49 | |
| *** kushal has joined #openstack-infra | 07:53 | |
| *** mrmartin has joined #openstack-infra | 07:53 | |
| *** e0ne has joined #openstack-infra | 07:55 | |
| *** e0ne has quit IRC | 07:56 | |
| *** jed56 has joined #openstack-infra | 07:57 | |
| *** mikelk has joined #openstack-infra | 07:59 | |
| *** resker has joined #openstack-infra | 08:01 | |
| *** esker has quit IRC | 08:01 | |
| *** eliqiao has quit IRC | 08:01 | |
| *** sorantis has joined #openstack-infra | 08:01 | |
| *** amotoki has quit IRC | 08:02 | |
| *** markvoelker has quit IRC | 08:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack/requirements: Bump upper constraint on python-novaclient to 3.3.0 https://review.openstack.org/287355 | 08:06 |
| *** eliqiao has joined #openstack-infra | 08:07 | |
| *** amit213 has joined #openstack-infra | 08:09 | |
| *** andymaier has quit IRC | 08:09 | |
| *** kzaitsev_mb has quit IRC | 08:10 | |
| *** amit213 has quit IRC | 08:13 | |
| *** [HeOS] has joined #openstack-infra | 08:15 | |
| *** gildub has quit IRC | 08:15 | |
| *** _nadya_ has joined #openstack-infra | 08:16 | |
| *** amit213 has joined #openstack-infra | 08:16 | |
| *** claudiub|2 has joined #openstack-infra | 08:16 | |
| *** mrmartin has quit IRC | 08:18 | |
| *** sflanigan has quit IRC | 08:18 | |
| *** amit213 has quit IRC | 08:18 | |
| *** HeOS has quit IRC | 08:19 | |
| *** yaume has joined #openstack-infra | 08:20 | |
| claudiub|2 | AJaeger: hello. Do you know if the pypi jobs were down last night? I pushed a signed tag to gerrit, it exists on tarballs.openstack.org, it exists on github, but not on pypi. project name: os-win | 08:22 |
| *** sobersabre has joined #openstack-infra | 08:22 | |
| sobersabre | hi. Is there a way to retrigger CI on a patch? | 08:22 |
| *** mrmartin has joined #openstack-infra | 08:22 | |
| claudiub|2 | sobersabre: recheck | 08:23 |
| *** esikachev has quit IRC | 08:23 | |
| claudiub|2 | sobersabre: or if it's a 3rd party ci, it usually says how to in the comment it posts. | 08:23 |
| *** amit213 has joined #openstack-infra | 08:24 | |
| *** andymaier has joined #openstack-infra | 08:25 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: do not propose a new keystone config file if already approved https://review.openstack.org/287647 | 08:28 |
| *** vincentll has joined #openstack-infra | 08:29 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: do not propose a new keystone config file if already approved https://review.openstack.org/287647 | 08:29 |
| sobersabre | claudiub|2: thanks, I added "recheck". How long does it usually take until there is something I can identify that CI picked this up ? | 08:32 |
| *** ihrachys has joined #openstack-infra | 08:32 | |
| *** achanda has quit IRC | 08:32 | |
| claudiub|2 | sobersabre: if it's jenkins, check http://status.openstack.org/ | 08:32 |
| *** dingyichen has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Create DVR multinode grenade job for Neutron upgrade tests https://review.openstack.org/250215 | 08:32 |
| *** esikachev has joined #openstack-infra | 08:33 | |
| claudiub|2 | sobersabre: or this: http://status.openstack.org//zuul/ you'll see all the jobs currently in queue. | 08:34 |
| *** matrohon has joined #openstack-infra | 08:34 | |
| *** eliqiao_ has joined #openstack-infra | 08:35 | |
| *** dolpher has quit IRC | 08:35 | |
| *** eliqiao has quit IRC | 08:35 | |
| *** e0ne has joined #openstack-infra | 08:36 | |
| AJaeger | claudiub|2: run git show-ref YOUR_TAG and check the log files on logs.openstack.org | 08:37 |
| *** shardy has joined #openstack-infra | 08:37 | |
| AJaeger | If your tag is a0b... look at logs.openstack.org/a0/a0b... for it | 08:37 |
| *** jlanoux has joined #openstack-infra | 08:38 | |
| *** e0ne has quit IRC | 08:38 | |
| claudiub|2 | ok thanks. :) | 08:38 |
| *** arxcruz has joined #openstack-infra | 08:40 | |
| *** vgridnev has joined #openstack-infra | 08:40 | |
| *** thorst has joined #openstack-infra | 08:41 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Setup translation for networking-midonet https://review.openstack.org/287654 | 08:46 |
| *** sorantis has quit IRC | 08:49 | |
| *** thorst has quit IRC | 08:49 | |
| AJaeger | yolanda, system-config cores: for mitaka the German translators need a place to discuss and want an email list, could you review https://review.openstack.org/#/c/287221/ , please? | 08:50 |
| AJaeger | claudiub|2: What do the logs say? Where's the problem? | 08:51 |
| claudiub|2 | AJaeger: honestly, can't find them.. not really sure where to look. the pushed tag is 0.2.3: git show-ref 0.2.3 -> 800d261a12fde9cdd6c5cedc150aea8965da7752 refs/tags/0.2.3 | 08:52 |
| AJaeger | logs.openstack.org/80/800d261a12fde9cdd6c5cedc150aea8965da7752 | 08:52 |
| claudiub|2 | oh, i see. | 08:53 |
| AJaeger | Looking at the logfile, it looks like a problem on pypi - ask an infra-root to reupload, please | 08:53 |
| claudiub|2 | ok, sure. thanks for the help. I appreciate it. :) | 08:54 |
| *** vgridnev has quit IRC | 08:54 | |
| *** vgridnev has joined #openstack-infra | 08:55 | |
| claudiub|2 | fungi: hello | 08:55 |
| AJaeger | claudiub|2: please don't wake him ;) He should be around 15:00 UTC | 08:56 |
| claudiub|2 | ok. :D | 08:57 |
| *** hashar has joined #openstack-infra | 09:00 | |
| *** BobBall_AWOL is now known as BobBall | 09:01 | |
| *** resker has quit IRC | 09:01 | |
| *** fhubik has joined #openstack-infra | 09:02 | |
| *** andymaier has quit IRC | 09:03 | |
| *** markvoelker has joined #openstack-infra | 09:03 | |
| *** kushal has quit IRC | 09:05 | |
| *** esker has joined #openstack-infra | 09:05 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:06 | |
| sobersabre | claudiub|2: thanks, I got it, and the builds passed ok. | 09:06 |
| *** eliqiao_ is now known as eliqiao | 09:07 | |
| *** rraja has joined #openstack-infra | 09:10 | |
| *** kzaitsev_mb has quit IRC | 09:11 | |
| *** asettle has joined #openstack-infra | 09:11 | |
| openstackgerrit | Max V.K. proposed openstack-infra/jenkins-job-builder: upd google-cloud-storage plugin support https://review.openstack.org/287497 | 09:12 |
| *** fhubik is now known as fhubik_brb | 09:12 | |
| *** jordanP has joined #openstack-infra | 09:14 | |
| *** andymaier has joined #openstack-infra | 09:15 | |
| claudiub|2 | sobersabre: no problem. :) | 09:15 |
| *** jistr has joined #openstack-infra | 09:17 | |
| *** lucas-dinner is now known as lucasagomes | 09:17 | |
| *** daemontool_ has joined #openstack-infra | 09:19 | |
| *** zhurong has quit IRC | 09:22 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:24 | |
| *** e0ne has joined #openstack-infra | 09:24 | |
| *** Daisy has joined #openstack-infra | 09:26 | |
| *** Daisy has quit IRC | 09:27 | |
| *** Daisy has joined #openstack-infra | 09:28 | |
| *** Daisy has quit IRC | 09:29 | |
| *** Daisy has joined #openstack-infra | 09:29 | |
| *** Daisy has quit IRC | 09:30 | |
| *** Daisy has joined #openstack-infra | 09:30 | |
| *** mrmartin has quit IRC | 09:31 | |
| abregman | AJaeger: hi. can you take a look?: https://review.openstack.org/#/c/287239 | 09:31 |
| *** Daisy has quit IRC | 09:32 | |
| *** sorantis has joined #openstack-infra | 09:32 | |
| *** Daisy has joined #openstack-infra | 09:33 | |
| *** jokke__ is now known as jokke_ | 09:33 | |
| *** Daisy has quit IRC | 09:33 | |
| *** Daisy has joined #openstack-infra | 09:33 | |
| *** Daisy has quit IRC | 09:35 | |
| *** Daisy has joined #openstack-infra | 09:36 | |
| *** Daisy has quit IRC | 09:36 | |
| *** Daisy has joined #openstack-infra | 09:37 | |
| ttx | Job at top of gate stuck for the last 2 hours | 09:41 |
| ttx | AJaeger: any idea what the current job timeout is ? | 09:43 |
| ttx | https://jenkins02.openstack.org/job/gate-grenade-dsvm-multinode/3180/ | 09:43 |
| *** gongysh has joined #openstack-infra | 09:44 | |
| AJaeger | ttx, let me check... | 09:46 |
| AJaeger | ttx: 180 mins | 09:46 |
| openstackgerrit | James Page proposed openstack-infra/project-config: Update groups for charm projects https://review.openstack.org/287676 | 09:46 |
| *** thorst has joined #openstack-infra | 09:46 | |
| ttx | oh well, I guess it's simpler to just wait for it to hit then | 09:47 |
| AJaeger | abregman: why is that job run with tempest? | 09:48 |
| abregman | AJaeger: what do you mean? why not? | 09:48 |
| AJaeger | I mean part of devstack - it should test neutron, shouldn't it? | 09:49 |
| AJaeger | I'll give it a -1 on naming - jobs start with gate. And sdague should review further... | 09:50 |
| *** kushal has joined #openstack-infra | 09:51 | |
| *** openstackgerrit_ has joined #openstack-infra | 09:53 | |
| *** thorst has quit IRC | 09:54 | |
| abregman | AJaeger: not sure I'm following. "we do not use experimental"? there are other jobs there that using it. also, the job is not testing neutron, it installs it instead of using nova network | 09:58 |
| AJaeger | ttx, job timed out finally ;( | 09:59 |
| AJaeger | abregman: the name should be "gate-XXX" not "experimental-XXX" | 09:59 |
| abregman | AJaeger: why? this is an experimental job | 09:59 |
| AJaeger | abregman: a convention we have. | 09:59 |
| AJaeger | abregman: you will move it to check/gate later and should not need to rename it... | 10:00 |
| *** e0ne has quit IRC | 10:00 | |
| abregman | AJaeger: but look on line 1273, 1274 here: https://review.openstack.org/#/c/287239/3/jenkins/jobs/projects.yaml | 10:01 |
| *** kukacz has joined #openstack-infra | 10:01 | |
| abregman | AJaeger: it's also using experimental. is it a mistake? | 10:01 |
| *** kzaitsev_mb has quit IRC | 10:01 | |
| *** Odd_Blok1 is now known as Odd_Bloke | 10:02 | |
| *** openstackgerrit_ has quit IRC | 10:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename experimental-kolla-dsvm-deploy-multinode-ubuntu-source https://review.openstack.org/287683 | 10:02 |
| *** Odd_Bloke has quit IRC | 10:02 | |
| *** Odd_Bloke has joined #openstack-infra | 10:02 | |
| AJaeger | abregman: yes, a mistake | 10:02 |
| abregman | heh..ack | 10:02 |
| *** Jeffrey4l has joined #openstack-infra | 10:03 | |
| *** _degorenko|afk is now known as degorenko | 10:03 | |
| *** boris-42 has quit IRC | 10:04 | |
| *** gildub has joined #openstack-infra | 10:04 | |
| openstackgerrit | Qiming Teng proposed openstack/requirements: Bump python-senlinclient version to 0.3.0 https://review.openstack.org/263598 | 10:05 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename experimental jobs https://review.openstack.org/287683 | 10:05 |
| AJaeger | abregman: fixed ^ | 10:06 |
| *** rguillebert has joined #openstack-infra | 10:06 | |
| *** Daisy has quit IRC | 10:06 | |
| *** Daisy has joined #openstack-infra | 10:07 | |
| *** odyssey4me_ is now known as odyssey4me | 10:07 | |
| openstackgerrit | abregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 10:08 |
| *** fhubik_brb is now known as fhubik | 10:09 | |
| AJaeger | abregman: change looks fine on syntax/convention. But I don't understand the reason why this is needed and where this should go, so adding some more explanation to it would be great. | 10:09 |
| *** mrmartin has joined #openstack-infra | 10:09 | |
| abregman | AJaeger: ack. adding. | 10:10 |
| *** e0ne has joined #openstack-infra | 10:10 | |
| *** dizquierdo has joined #openstack-infra | 10:10 | |
| *** Daisy has quit IRC | 10:11 | |
| *** Qiming has quit IRC | 10:11 | |
| AJaeger | abregman: I think I'm getting what you want finally and thus adding to devstack is proper way. Just one more question: Why do we need both centos7 and centos7-neutron? Shouldn't one be enough? | 10:12 |
| abregman | AJaeger: when centos7-network will be stable, I'll make sure to remove centos7. At this point it might be buggy. we'll have to see. | 10:13 |
| abregman | centos7-neutron* | 10:13 |
| *** sfinucan has joined #openstack-infra | 10:13 | |
| AJaeger | abregman: then mention that plan in the commit message, please | 10:14 |
| AJaeger | That's a perfectly fine way forward and with that information I'm happy to +2 ;) | 10:14 |
| AJaeger | Ah, I see ianw commenting as well... | 10:15 |
| ianw | AJaeger: yeah, sorry, I asked abregman to look at this :) | 10:15 |
| AJaeger | abregman: I'll wait for jenkins tests and then +2A... The comments give enough information | 10:15 |
| *** amotoki has joined #openstack-infra | 10:16 | |
| *** amotoki has quit IRC | 10:16 | |
| *** jordan_ has joined #openstack-infra | 10:16 | |
| AJaeger | ianw: which is perfectly fine - just the information why this was needed and what your plans are belong into a commit message. | 10:16 |
| openstackgerrit | abregman proposed openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 10:16 |
| AJaeger | I can see what is done but I need to understand the why sometimes as well ;) | 10:16 |
| abregman | AJaeger: fair enough :) extended commit message. | 10:17 |
| AJaeger | Thanks, abregman for the update! | 10:17 |
| abregman | thank you :) | 10:17 |
| AJaeger | +2 | 10:17 |
| *** jordanP has quit IRC | 10:20 | |
| *** abregman is now known as abregman|afk | 10:22 | |
| *** SotK_ is now known as SotK | 10:22 | |
| *** sulo_ is now known as sulo | 10:22 | |
| *** |-paul-| has joined #openstack-infra | 10:24 | |
| wznoinsk | hi all | 10:26 |
| wznoinsk | anteaya, are you around? | 10:26 |
| wznoinsk | actually that's a general infra question I have here... | 10:27 |
| wznoinsk | do you anticipate moving CI/jenkins to Kolla in next some time? | 10:27 |
| openstackgerrit | Merged openstack-infra/project-config: Add new expermintal job - centos with neutron https://review.openstack.org/287239 | 10:35 |
| *** sgordon` is now known as sgordon | 10:37 | |
| *** Daisy has joined #openstack-infra | 10:38 | |
| *** sfinucan has quit IRC | 10:40 | |
| *** Daisy has quit IRC | 10:42 | |
| *** Daisy has joined #openstack-infra | 10:42 | |
| *** Daisy has quit IRC | 10:47 | |
| yolanda | hi AJaeger, i've been out all morning, reading the backlog now | 10:50 |
| *** thorst has joined #openstack-infra | 10:51 | |
| dguitarbite | AJaeger: I realized that the package "zip" is missing on the infra hosts | 10:53 |
| dguitarbite | AJaeger: Is there anything I can do to add/install "zip" to create zip files? | 10:53 |
| *** tpsilva has joined #openstack-infra | 10:55 | |
| *** mrmartin has quit IRC | 10:56 | |
| *** gongysh has quit IRC | 10:57 | |
| *** thorst has quit IRC | 10:59 | |
| *** doug-fish has joined #openstack-infra | 11:00 | |
| *** kushal has quit IRC | 11:00 | |
| *** alex_xu_ has quit IRC | 11:01 | |
| *** abregman|afk is now known as abregman | 11:01 | |
| *** salv-orl_ has quit IRC | 11:01 | |
| *** salv-orlando has joined #openstack-infra | 11:02 | |
| *** alex_xu has joined #openstack-infra | 11:02 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:02 | |
| *** christiana has joined #openstack-infra | 11:02 | |
| *** fhubik is now known as fhubik_brb | 11:02 | |
| *** jed56 has quit IRC | 11:03 | |
| *** doug-fish has quit IRC | 11:05 | |
| *** ociuhandu has joined #openstack-infra | 11:06 | |
| *** lezbar__ has quit IRC | 11:06 | |
| *** lezbar has joined #openstack-infra | 11:07 | |
| *** fhubik_brb is now known as fhubik | 11:07 | |
| *** rraja has quit IRC | 11:08 | |
| claudiub|2 | yolanda: hello. I need a bit of help. I've pushed the tag 0.2.3 for os-win last night and the pypi jobs failed. the tag exists on github, as well as tarball.openstack.org. Logs: http://logs.openstack.org/80/800d261a12fde9cdd6c5cedc150aea8965da7752/release/os-win-pypi-both-upload/d6e1bf3/console.html | 11:08 |
| *** links has quit IRC | 11:08 | |
| yolanda | hi claudiub|2 i'll try to help | 11:08 |
| *** rraja has joined #openstack-infra | 11:08 | |
| claudiub|2 | yolanda: cool, thanks! let me know how it goes. :) | 11:09 |
| yolanda | i'll take a loko | 11:09 |
| *** kzaitsev_mb has quit IRC | 11:10 | |
| *** dmsimard has quit IRC | 11:10 | |
| *** links has joined #openstack-infra | 11:10 | |
| *** jpr has joined #openstack-infra | 11:12 | |
| *** dims has joined #openstack-infra | 11:14 | |
| *** hashar has quit IRC | 11:19 | |
| *** salv-orl_ has joined #openstack-infra | 11:19 | |
| jamespage | AJaeger, yolanda: after some digging into the update-bug command in jeepyb I think https://review.openstack.org/#/c/287676/ will target the right bug tasks for the charms... | 11:19 |
| jamespage | example of API - https://bugs.launchpad.net/api/1.0/charms/+source/ceph/+bug/1493549 | 11:20 |
| openstack | Launchpad bug 1 in Ubuntu Malaysia LoCo Team "Microsoft has a majority market share" [Critical,In progress] - Assigned to MFauzilkamil Zainuddin (apogee) | 11:20 |
| *** ikalnitsky has left #openstack-infra | 11:21 | |
| *** salv-orlando has quit IRC | 11:22 | |
| openstackgerrit | Sergey Nikitin proposed openstack-infra/project-config: Move job gate-tempest-dsvm-nova-wsgi-full to the check list https://review.openstack.org/287726 | 11:23 |
| yolanda | jamespage checking, i didn't have knowledge about that | 11:26 |
| *** dims has quit IRC | 11:26 | |
| jamespage | yolanda, took me a while to figure out digging about in jeepyb | 11:26 |
| yolanda | for sure, groups is not the best name for that setting | 11:27 |
| *** dims has joined #openstack-infra | 11:27 | |
| *** kashyap has joined #openstack-infra | 11:28 | |
| *** jed56 has joined #openstack-infra | 11:28 | |
| kashyap | AJaeger: Hi, any tool to locally validate rST (I'm using `rstcheck` but it isn't catching invalid links) | 11:29 |
| *** gildub has quit IRC | 11:31 | |
| *** keedya has quit IRC | 11:32 | |
| yolanda | AJaeger, do you know the procedure to retry a pypi upload? i tried to rebuild the job on jenkins but i don't have that option | 11:36 |
| yolanda | what i usually do downstream, is that there is a rebuild with parameters option so you can do manually on jenkins, passed the parameters you need, but i don't see this option upstream | 11:36 |
| *** electrofelix has joined #openstack-infra | 11:37 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack-infra/infra-manual: creators.rst: Fix broken internel hyperlinks https://review.openstack.org/287083 | 11:37 |
| AJaeger | yolanda: no worries, nothing urgent here ;) | 11:38 |
| *** kzaitsev_mb has joined #openstack-infra | 11:38 | |
| AJaeger | yolanda: I can't help with the pypi upload. I think fungi is just running the job again but I'm not sure | 11:39 |
| *** yamahata has quit IRC | 11:39 | |
| AJaeger | kashyap: there's doc8 that can catch some issues and sphinx has a linkcheck option | 11:39 |
| kashyap | AJaeger: Thank you. I clearly failed in rST test, eh | 11:40 |
| yolanda | i tried that but it needs to be run with the same parameter, and i miss that option in jenkins.openstack | 11:40 |
| AJaeger | dguitarbite: no zip? I thought that was on the images - better ask the experts later today. | 11:40 |
| *** fhubik is now known as fhubik_brb | 11:40 | |
| *** kushal has joined #openstack-infra | 11:41 | |
| AJaeger | dguitarbite: do you have a log file to look at? | 11:41 |
| AJaeger | yolanda: then best ask fungi later | 11:41 |
| *** tqtran has quit IRC | 11:42 | |
| *** sorantis has quit IRC | 11:42 | |
| yolanda | i'll do | 11:42 |
| AJaeger | dguitarbite: jenkins/data/bindep-fallback.txt in project-config includes zip - so that means it should be there... | 11:46 |
| AJaeger | (note that list is not used for the image your job run, just derived from it) | 11:47 |
| AJaeger | dguitarbite: http://tarballs.openstack.org/training-labs/ has a master tarball - isn't that what you need? | 11:49 |
| *** sdague has joined #openstack-infra | 11:50 | |
| *** gildub has joined #openstack-infra | 11:50 | |
| *** snikitin has joined #openstack-infra | 11:51 | |
| *** rfolco has joined #openstack-infra | 11:52 | |
| openstackgerrit | Merged openstack-infra/project-config: Change ACL for fuel projects https://review.openstack.org/284680 | 11:53 |
| openstackgerrit | Merged openstack-infra/project-config: Dedicated specs repo for Freezer https://review.openstack.org/280307 | 11:54 |
| *** asettle has quit IRC | 11:55 | |
| *** kdas__ has joined #openstack-infra | 11:55 | |
| *** kushal has quit IRC | 11:56 | |
| *** kdas__ has quit IRC | 11:56 | |
| *** kushal has joined #openstack-infra | 11:56 | |
| *** thorst has joined #openstack-infra | 11:56 | |
| openstackgerrit | Sergey Nikitin proposed openstack-infra/project-config: Move job gate-tempest-dsvm-nova-wsgi-full to the check list https://review.openstack.org/287726 | 11:57 |
| *** csaba has joined #openstack-infra | 11:59 | |
| csaba | AJaeger: ping | 12:01 |
| csaba | AJaeger: can you please merge https://review.openstack.org/#/c/287632/ for me? | 12:01 |
| *** ldnunes has joined #openstack-infra | 12:02 | |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** openstackgerrit has joined #openstack-infra | 12:03 | |
| *** thorst has quit IRC | 12:04 | |
| *** aysyd has joined #openstack-infra | 12:04 | |
| *** gildub has quit IRC | 12:06 | |
| *** aysyd has quit IRC | 12:09 | |
| *** _amrith_ is now known as amrith | 12:09 | |
| *** aysyd has joined #openstack-infra | 12:10 | |
| *** esikachev has quit IRC | 12:11 | |
| openstackgerrit | Erno Kuvaja proposed openstack/requirements: Bump python-glanceclient to 2.0.0 https://review.openstack.org/287751 | 12:12 |
| *** |-paul-| has quit IRC | 12:17 | |
| *** rhallisey has joined #openstack-infra | 12:17 | |
| *** esikachev has joined #openstack-infra | 12:17 | |
| AJaeger | csaba: please explain what your change is and why it's important. I'll review later... | 12:18 |
| *** rhallisey has quit IRC | 12:18 | |
| *** rhallisey has joined #openstack-infra | 12:18 | |
| *** doug-fish has joined #openstack-infra | 12:19 | |
| *** sorantis has joined #openstack-infra | 12:21 | |
| *** sshnaidm has quit IRC | 12:30 | |
| *** doug-fish has quit IRC | 12:31 | |
| *** NobodyCam has quit IRC | 12:31 | |
| *** NobodyCam has joined #openstack-infra | 12:31 | |
| *** fhubik_brb is now known as fhubik | 12:32 | |
| *** kushal has quit IRC | 12:33 | |
| *** doug-fish has joined #openstack-infra | 12:34 | |
| *** sshnaidm has joined #openstack-infra | 12:35 | |
| openstackgerrit | Merged openstack/diskimage-builder: Replace sfdisk partitioning with parted https://review.openstack.org/258721 | 12:35 |
| *** thorst has joined #openstack-infra | 12:37 | |
| *** yamamoto_ has quit IRC | 12:38 | |
| *** jaosorior has quit IRC | 12:39 | |
| *** jaosorior has joined #openstack-infra | 12:39 | |
| *** jaosorior has quit IRC | 12:39 | |
| *** esikachev has quit IRC | 12:39 | |
| *** jaosorior has joined #openstack-infra | 12:40 | |
| *** dprince has joined #openstack-infra | 12:40 | |
| *** gordc has joined #openstack-infra | 12:41 | |
| AJaeger | sdague: https://review.openstack.org/#/c/287726/ is a new review that adds a job for nova check queue | 12:43 |
| dguitarbite | AJaeger: Yes, but the zip command was failing in the jenkins jobs so I disabled it for the time being. Ill see if I can find the jenkins failures | 12:43 |
| AJaeger | dguitarbite: Ah, I see. Best ask fungi or clarkb later about this and reference a log file showing the failure. | 12:44 |
| *** dprince has quit IRC | 12:46 | |
| openstackgerrit | Csaba Henk proposed openstack-infra/project-config: Add jobs for Manila glusterfs, glusterfs-native drivers with Heketi layout https://review.openstack.org/287632 | 12:46 |
| dguitarbite | ok, thanks :) Finally I can rest in peace as the tar balls are automatically hosted :D | 12:46 |
| csaba | AJaeger: I submitted a new driver mode to Manila's glusterfs drivers which is eligible to merge only if it gets gateway tested. I'm adding the testcases which do that. (The particular urgency that made me venture to bug you with it is that Manila m3 is to be tagged today.) | 12:48 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: add scenario002 jobs for puppet-cinder https://review.openstack.org/287771 | 12:50 |
| flaper87 | Do jobs in the gate use `run_tests.sh` scripts? Or do they just use tox for individual projects? I'd expect the latter, AFAIK | 12:50 |
| *** dprince has joined #openstack-infra | 12:53 | |
| *** rossella_s has quit IRC | 12:54 | |
| *** rossella_s has joined #openstack-infra | 12:54 | |
| *** _nadya_ has quit IRC | 12:55 | |
| *** _nadya_ has joined #openstack-infra | 12:55 | |
| *** thiagop has joined #openstack-infra | 12:56 | |
| AJaeger | csaba: PLesae get a +1 by manila folks on that change first. | 12:56 |
| AJaeger | csaba: btw I cannot merge a change by myself - we always use two core reviewers... | 12:57 |
| AJaeger | flaper87: give me an example job name... | 12:57 |
| AJaeger | I expect it will use jenkins/scripts/run-tox.sh which runs tox -e ... - and looking at the log file should show that as well | 12:58 |
| *** jpr has quit IRC | 12:58 | |
| openstackgerrit | abregman proposed openstack-infra/devstack-gate: DO NOT MERGE - Testing Gates https://review.openstack.org/287780 | 12:59 |
| flaper87 | AJaeger: I'm interested in Glance's jobs in this case. I'd like to get rid of the old run_tests script, which none of us is using but I'm not sure if it's being used in the gate (which I doubt) | 12:59 |
| *** _nadya_ has quit IRC | 13:00 | |
| EmilienM | fungi: hello! we found out https://github.com/openstack-infra/system-config/commit/3894d932667de2d4dbbc89c58fc34b3d47294e6b makes Puppet OpenStack CI jobs failing quite often | 13:00 |
| EmilienM | fungi: because SYS_UID_MIN is missing and in our case, rabbitmq is stealing the UID of keystone | 13:00 |
| EmilienM | fungi: and boom. | 13:01 |
| AJaeger | flaper87: a quick grep shows no occurence in project-config. Remove it - your tests should fail if my grep is wrong | 13:01 |
| AJaeger | EmilienM: good catch | 13:01 |
| *** apevec has joined #openstack-infra | 13:02 | |
| flaper87 | AJaeger: did it, waiting for the gate! Thanks for looking into it | 13:02 |
| EmilienM | AJaeger: well, I think there is a reason of such a patch, so before sending a patch, I want to discuss with fungi / pabelanger about that | 13:02 |
| AJaeger | EmilienM: sure ;) | 13:02 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287784 | 13:03 |
| apevec | yeah, I'd like to understand the issue about "Declare a non-overlapping dynamic UID/GID range" | 13:03 |
| *** baoli has joined #openstack-infra | 13:03 | |
| apevec | and why replacing the whole file... | 13:03 |
| *** maishsk has quit IRC | 13:05 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Move search icon https://review.openstack.org/279204 | 13:05 |
| *** noorul has quit IRC | 13:07 | |
| *** fhubik has quit IRC | 13:08 | |
| csaba | AJaeger: thanks, I'm to do that. | 13:08 |
| *** dkranz has joined #openstack-infra | 13:08 | |
| *** placko has joined #openstack-infra | 13:09 | |
| *** amrith is now known as _amrith_ | 13:10 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI [WIP] https://review.openstack.org/287784 | 13:11 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Move search icon https://review.openstack.org/279204 | 13:11 |
| *** markvoelker has quit IRC | 13:11 | |
| *** markvoelker has joined #openstack-infra | 13:11 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Move search icon https://review.openstack.org/279204 | 13:11 |
| *** pradk has joined #openstack-infra | 13:13 | |
| *** sfinucan has joined #openstack-infra | 13:13 | |
| *** dmsimard has joined #openstack-infra | 13:14 | |
| *** sdake has joined #openstack-infra | 13:14 | |
| *** gongysh has joined #openstack-infra | 13:14 | |
| *** kgiusti has joined #openstack-infra | 13:15 | |
| openstackgerrit | Csaba Henk proposed openstack-infra/project-config: Add jobs for Manila glusterfs, glusterfs-native drivers with Heketi layout https://review.openstack.org/287632 | 13:15 |
| *** dizquierdo has quit IRC | 13:18 | |
| *** _oanson has joined #openstack-infra | 13:19 | |
| *** gongysh has quit IRC | 13:19 | |
| *** moravec has quit IRC | 13:19 | |
| *** __oanson has joined #openstack-infra | 13:20 | |
| snikitin | sdague: Hi Sean! I saw your comment about WSGI nova job https://review.openstack.org/#/c/287726/ So you propose to increase the number of tests for wsgi and then move this job to the list of check jobs? | 13:20 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Basic Task Links https://review.openstack.org/277882 | 13:22 |
| *** oanson has quit IRC | 13:23 | |
| *** _oanson has quit IRC | 13:23 | |
| *** esikachev has joined #openstack-infra | 13:24 | |
| *** yamamoto_ has joined #openstack-infra | 13:25 | |
| *** yamamoto_ has quit IRC | 13:26 | |
| *** rlandy has joined #openstack-infra | 13:28 | |
| *** yamamoto_ has joined #openstack-infra | 13:31 | |
| *** keedya has joined #openstack-infra | 13:31 | |
| sdague | snikitin: honestly, I don't know right now, but we're constrained on test resources, and I don't want to pile on a new full job that is 99% the same as the existing ones | 13:32 |
| *** bhunter__ has joined #openstack-infra | 13:32 | |
| *** links has quit IRC | 13:34 | |
| *** zhurong has joined #openstack-infra | 13:34 | |
| *** hashar has joined #openstack-infra | 13:35 | |
| *** yamamoto_ has quit IRC | 13:35 | |
| *** sbelous_ has joined #openstack-infra | 13:37 | |
| *** xyang1 has joined #openstack-infra | 13:37 | |
| *** maishsk has joined #openstack-infra | 13:39 | |
| *** kiran-r has quit IRC | 13:40 | |
| *** vmbrasseur has quit IRC | 13:40 | |
| *** eezhova has joined #openstack-infra | 13:41 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack-infra/infra-manual: creators.rst: Fix broken internal hyperlinks https://review.openstack.org/287083 | 13:42 |
| frickler | someone might want to cancel https://jenkins02.openstack.org/job/gate-ubuntu-trusty-cookbook-openstack-identity-chef-rake-integration-nv/1/ as it seems I created a loop there :D | 13:42 |
| frickler | on a side note, when and why did curl get removed from ubuntu instances? | 13:43 |
| pabelanger | frickler: it should get killed after 30mins | 13:43 |
| *** kashyap has left #openstack-infra | 13:43 | |
| pabelanger | frickler: we use DIBs for ubuntu-trusty so we likely need to ensure that package is installed by default | 13:44 |
| pabelanger | frickler: should be easy to do once we land -bindep jobs | 13:44 |
| *** larainema has quit IRC | 13:44 | |
| pabelanger | EmilienM: fungi: I am not sure the reason for the change. | 13:45 |
| frickler | pabelanger: the thing is, this worked two days ago: http://logs.openstack.org/19/286619/1/check/gate-cookbook-openstack-identity-chef-rake-integration-nv/678587f/console.html#_2016-03-01_14_30_55_464 now it fails: http://logs.openstack.org/31/287031/12/check/gate-ubuntu-trusty-cookbook-openstack-identity-chef-rake-integration-nv/31a6070/console.html#_2016-03-03_13_31_17_670 | 13:46 |
| pabelanger | frickler: look at the 3rd line of the output. We have changed base images | 13:46 |
| AJaeger | pabelanger: so, we miss curl on ubuntu-trusty? | 13:46 |
| pabelanger | bare-trusty vs ubuntu-trusty | 13:46 |
| pabelanger | AJaeger: likely | 13:47 |
| pabelanger | should be easy to fix | 13:47 |
| pabelanger | mostly need to figure out where to add it | 13:47 |
| AJaeger | bindep-fallback? | 13:47 |
| pabelanger | either dib element, puppet or bindep-fallback | 13:47 |
| AJaeger | Is there a guideline what goes where? | 13:48 |
| *** edmondsw has joined #openstack-infra | 13:48 | |
| pabelanger | not that I know off | 13:48 |
| pabelanger | we should likely define that | 13:48 |
| mnaser | woo not a single error node in the past 6 hours | 13:49 |
| pabelanger | So, if JJB requires it to run a test, I think having dib element or puppet do it is the place to be. Other wise, every projects other-requirements.txt will need to include curl | 13:49 |
| *** larainema has joined #openstack-infra | 13:49 | |
| *** weshay has joined #openstack-infra | 13:49 | |
| mordred | mnaser: nicely done! | 13:49 |
| *** yamamoto_ has joined #openstack-infra | 13:50 | |
| *** kiran-r has joined #openstack-infra | 13:50 | |
| mnaser | thanks mordred! this is a nice external health meter of the cloud | 13:50 |
| AJaeger | mnaser: This is great, thanks a lot for your contribution and help! | 13:52 |
| mnaser | no problem, we're doing a few things but we'll probably bump it up soon | 13:52 |
| mnaser | (the quota) | 13:53 |
| pabelanger | AJaeger: fungi: when you have a moment, 287488 is to enable the installation of bindep on fedora-23, but we still don't cache packages yet. | 13:54 |
| *** krtaylor has quit IRC | 13:54 | |
| pabelanger | we'll need the gate-bindep-fallback-fedora-23 experimental job to pass first | 13:54 |
| pabelanger | and moving it to the gate is an easy way for other to see the results of bindep | 13:55 |
| *** elmiko_ is now known as elmiko | 13:55 | |
| *** sfinucan has quit IRC | 13:56 | |
| *** placko has quit IRC | 13:57 | |
| *** regXboi has joined #openstack-infra | 13:57 | |
| AJaeger | pabelanger: did you see ianw 's comment on 287488? | 13:57 |
| *** vmbrasseur has joined #openstack-infra | 13:58 | |
| pabelanger | AJaeger: Yup, I left a comment | 13:58 |
| pabelanger | well, I thought I did | 13:58 |
| *** bhunter__ has quit IRC | 13:58 | |
| pabelanger | let me repost it | 13:59 |
| *** vgridnev has quit IRC | 14:00 | |
| *** ddieterl_ is now known as ddieterly | 14:01 | |
| pabelanger | AJaeger: update | 14:01 |
| pabelanger | updated* | 14:01 |
| *** vgridnev has joined #openstack-infra | 14:02 | |
| sdague | fungi / mordred - vexxhost is now lit? | 14:02 |
| sdague | has anyone confirmed if multinode works on it? | 14:02 |
| *** jed56 has quit IRC | 14:03 | |
| *** flepied has quit IRC | 14:04 | |
| *** flepied has joined #openstack-infra | 14:04 | |
| *** maishsk has quit IRC | 14:05 | |
| *** mrmartin has joined #openstack-infra | 14:05 | |
| *** yamamoto_ has quit IRC | 14:07 | |
| *** flepied1 has joined #openstack-infra | 14:08 | |
| *** zhurong has quit IRC | 14:08 | |
| *** flepied has quit IRC | 14:09 | |
| *** flepied1 is now known as flepied | 14:09 | |
| pabelanger | sdague: https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-multinode-full/2384/ is currently running on vexxhost | 14:10 |
| pabelanger | I haven't heard anybody say anything about failures | 14:10 |
| pabelanger | but only been running for the last 12 hours or so | 14:10 |
| sdague | http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_node%3A*vexx*%20AND%20(message%3A%5C%22Finished%20FAILURE%5C%22%20OR%20message%3A%5C%22Finished%20SUCCESS%5C%22)%20AND%20build_name%3A*multi* | 14:11 |
| sdague | right, vexxhost have never worked for a multinode job | 14:11 |
| sdague | can we pull it out of that job class? | 14:11 |
| *** jaypipes has joined #openstack-infra | 14:12 | |
| sdague | I think we should also have a policy of lighting new providers that we don't put them into the multinode pool by default | 14:12 |
| sdague | because the experience has been we need to tweak for every one of them | 14:12 |
| sdague | and this seems to also be true for vexxhost | 14:12 |
| *** jordan__ has joined #openstack-infra | 14:13 | |
| pabelanger | will defer to clarkb on that | 14:13 |
| *** kencjohnston has joined #openstack-infra | 14:13 | |
| sdague | ok, well right now we only have 2 data points, and they are both failures | 14:14 |
| anteaya | wznoinsk: what do you mean by moving CI/jenkins to Kolla? | 14:14 |
| sdague | and this created a 7 hour delay in the gate overnight. It would be better to err on the side of safety | 14:14 |
| *** links has joined #openstack-infra | 14:15 | |
| anteaya | wznoinsk: I don't know what you are asking | 14:15 |
| *** doug-fish has quit IRC | 14:17 | |
| *** amitgandhinz has joined #openstack-infra | 14:17 | |
| *** jordan_ has quit IRC | 14:17 | |
| *** alivigni has joined #openstack-infra | 14:19 | |
| *** _nadya_ has joined #openstack-infra | 14:20 | |
| *** rkukura has quit IRC | 14:20 | |
| mnaser | bummer re: failures | 14:20 |
| mnaser | let me know if there's anything that's on our side which we can do | 14:21 |
| *** mriedem has joined #openstack-infra | 14:21 | |
| anteaya | mnaser: thanks sdague is refering to this setup: http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt | 14:22 |
| *** doug-fish has joined #openstack-infra | 14:22 | |
| mnaser | im looking at failure logs | 14:22 |
| anteaya | note fungi has a patch up to change gre to vxlan in those diagrams | 14:22 |
| *** denisra has quit IRC | 14:23 | |
| anteaya | logs are best to be sure | 14:23 |
| anteaya | and good morning | 14:23 |
| sdague | mnaser: there is some kind of networking issue it appears, the failure I just saw is not yet in elastic search | 14:23 |
| *** tiswanso has joined #openstack-infra | 14:23 | |
| fungi | hrm, i wonder why we wouldn't be able to vxlan tunnel between instances across the provider network there | 14:24 |
| *** jpr has joined #openstack-infra | 14:24 | |
| *** tiswanso has quit IRC | 14:24 | |
| mnaser | http://logs.openstack.org/16/287316/4/check/gate-manila-tempest-dsvm-neutron-scenario-multibackend/9442e19/console.html | 14:24 |
| mnaser | failures here seem to be related to >AttributeError: 'NetworkClient' object has no attribute 'add_router_interface_with_subnet_id' | 14:24 |
| sdague | http://logs.openstack.org/81/283781/6/gate/gate-grenade-dsvm-multinode/3c1b54b/logs/grenade.sh.txt.gz#_2016-03-03_09_51_29_865 | 14:24 |
| sdague | the issue is that an ssh is hanging at a socket level for 15 minutes before a failure | 14:24 |
| mnaser | so that one failing doesnt seem to be related to us, im looking at the other failure | 14:24 |
| *** tiswanso has joined #openstack-infra | 14:24 | |
| *** doug-fish has quit IRC | 14:24 | |
| fungi | but yeah, i'll get a patch up to stop multi-noding there temporarily | 14:24 |
| *** doug-fish has joined #openstack-infra | 14:24 | |
| sdague | fungi: awesome, great | 14:25 |
| fungi | while investigation ensues | 14:25 |
| sdague | I think it's just a thing we need to be cautious of | 14:25 |
| *** dmsimard has quit IRC | 14:25 | |
| sdague | because with ovh, internap, and now vexxhost there have been bumps on the multinode jobs | 14:25 |
| *** mrmartin has quit IRC | 14:25 | |
| sdague | so probably just assume they won't work by default, then test them before we light new providers there | 14:25 |
| fungi | agreed, after finding two providers where we couldn't pass gre between instances (and so switching to vxlan) it's clearly a blind spot in our current evaluation process | 14:26 |
| *** geekinutah has joined #openstack-infra | 14:27 | |
| mnaser | its strange because i cant imagine any reason to cause, we don't block or filter anything either | 14:27 |
| sdague | mnaser: mtu things? | 14:28 |
| anteaya | AJaeger: for https://review.openstack.org/#/c/287464/2 I'm curious if you have any thoughts about why tox -e zuul is outputing nonvoting status on jobs that you have demonstrating are voting | 14:28 |
| anteaya | AJaeger: does your tox -e zuul output disagree with mine? | 14:28 |
| *** max_lobur has joined #openstack-infra | 14:28 | |
| mnaser | sdague: that's the only possibility. but even then, we push out a specific tested mtu via neutron dhcp | 14:28 |
| sdague | mnaser: right, but then we put our own overlays in | 14:29 |
| max_lobur | G'morning All | 14:29 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily stop using 2-node workers in vexxhost https://review.openstack.org/287826 | 14:29 |
| *** mrmartin has joined #openstack-infra | 14:29 | |
| mnaser | sdague: so if you're adjusting mtu, it's likely adjusting to the wrong number | 14:29 |
| sdague | anyway, grasping at straws, but the fail is definitely at some fundamental networking layer here | 14:29 |
| max_lobur | how do we add people to newly created group in gerrit ? | 14:29 |
| fungi | mnaser: yeah, we set an mtu inside the tunnel, but i think we set it to 1450 and not to some function of the interface's mtu | 14:30 |
| *** jsavak has joined #openstack-infra | 14:30 | |
| fungi | i'm checking now | 14:30 |
| mnaser | ah that can explain things | 14:30 |
| fungi | we may need to make devstack-gate calculate an mtu reduction for the tunnel based on the mtu of the interface | 14:30 |
| *** links has quit IRC | 14:30 | |
| *** eharney has quit IRC | 14:31 | |
| yolanda | hi fungi, how can i rebuild a failed pypi job? i tried to relaunch one that failed today, but i do not see a rebuild option in jenkins | 14:31 |
| mnaser | fungi: we push out 1394 over dhcp | 14:31 |
| fungi | `git grep 1450` in openstack-infra/devstack-gate confirms this hypothesis | 14:32 |
| *** moravec has joined #openstack-infra | 14:32 | |
| mnaser | so if anything 1394-50 | 14:32 |
| fungi | yolanda: it depends on how it failed | 14:32 |
| yolanda | fungi | 14:32 |
| *** Qiming has joined #openstack-infra | 14:32 | |
| yolanda | http://logs.openstack.org/80/800d261a12fde9cdd6c5cedc150aea8965da7752/release/os-win-pypi-both-upload/d6e1bf3/console.html | 14:32 |
| *** erlon has joined #openstack-infra | 14:33 | |
| fungi | yolanda: there are a variety of ways that series of release jobs can go wrong, and different failure modes require different sorts of cleanup as some steps are not idempotent | 14:33 |
| anteaya | a peek at grafana tells me all jenkinsii seem to be jenkinsing | 14:33 |
| *** kencjohnston has quit IRC | 14:33 | |
| yolanda | fungi if failed uploading the wheel file, with requests.exceptions.HTTPError: 503 Server Error: No healthy backends | 14:33 |
| fungi | mnaser: that's almost certainly the problem--thanks. i think this is the first provider we've ended up on with an mtu other than 1500 on the instances | 14:34 |
| mnaser | fungi: cross-dc connections kinda force us to lower our mtus, but soon we'll be able to bump it back up .. so maybe try $link_mtu - 50; | 14:34 |
| fungi | yolanda: right, there are also two ways that error can go. sometimes that means it didn't upload the wheel, sometimes pypi reports that error on wheel upload even though the wheel uploaded successfully | 14:34 |
| fungi | yolanda: if it's the former, then enqueue-ref can be used to rerun that set of release jobs | 14:35 |
| claudiub|2 | os-win 0.2.3 doesn't exit on pypi: https://pypi.python.org/pypi/os-win | 14:35 |
| *** annegentle has joined #openstack-infra | 14:35 | |
| fungi | yolanda: if the latter, then the sdist needs to be uploaded manually (because twine will fail if it tries to upload a wheel which is already there--pypi rejects it) and the trigger-job.py used to run the release announcement job specifically | 14:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions https://review.openstack.org/287835 | 14:36 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Increase heat timeout with addition of new lbaasv2 scenario tests https://review.openstack.org/287836 | 14:36 |
| mordred | fungi, yolanda: when you get a sec ^^ I wrote that to tryt o get some better logs to figure out what's going on with bluebo | 14:36 |
| fungi | yolanda: but of course, all this assumes that you've first checked that the project is even set up correctly to publish releases on pypi at all | 14:36 |
| *** Guest60497 is now known as dansmith | 14:36 | |
| max_lobur | Clarifying a little bit: we've recently added bareon-ironic project, with it's own ACL, currently the group is empty https://review.openstack.org/#/admin/groups/1308,members . Could anoyone tell me how do I add people there? | 14:37 |
| mordred | or, rather, I now know sort of what's going on - but looking for it was hard, so I wanted to update the logs to be friendlier | 14:37 |
| AJaeger | anteaya: I run tox -e zuul this morning on a clean tree with head - and it disagreed with you as I commented in the review. | 14:37 |
| yolanda | fungi it failed when a tag was created, previous publishes to pypi worked | 14:37 |
| anteaya | great, so somehow I must be stale | 14:37 |
| AJaeger | Could you clean your tree and double check, please? | 14:37 |
| *** mrmartin has quit IRC | 14:38 | |
| anteaya | yeah I can do that, thanks for checking me | 14:38 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 14:39 |
| anteaya | AJaeger: I have to run some errands this morning and will fix upon my return | 14:39 |
| *** yamamoto_ has joined #openstack-infra | 14:39 | |
| *** roxanaghe has joined #openstack-infra | 14:40 | |
| *** roxanaghe has quit IRC | 14:40 | |
| yolanda | fungi https://pypi.python.org/simple/os-win/, no 0.2.3 content uploaded | 14:41 |
| *** Jeffrey4l has quit IRC | 14:41 | |
| *** lykinsbd has joined #openstack-infra | 14:41 | |
| AJaeger | anteaya: no urgency on this. And thanks a lot for that patch, you found quite a few problems! But something is strange in your tree (or mine ;) | 14:41 |
| fungi | yolanda: so looking at https://pypi.python.org/pypi/os-win it appears this time the failure means it didn't actually upload the wheel at all, so zuul enqueue-ref should work | 14:42 |
| anteaya | AJaeger: likely mine | 14:42 |
| anteaya | AJaeger: thanks for reviewing | 14:42 |
| fungi | yolanda: yep, just finished tracing through it, sorry, several things going on at once | 14:42 |
| *** akshai has joined #openstack-infra | 14:42 | |
| *** mriedem has left #openstack-infra | 14:42 | |
| *** mriedem has joined #openstack-infra | 14:43 | |
| *** sfinucan has joined #openstack-infra | 14:43 | |
| *** cbader has joined #openstack-infra | 14:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily stop using 2-node workers in vexxhost https://review.openstack.org/287826 | 14:46 |
| *** rbrndt_ has joined #openstack-infra | 14:46 | |
| *** nelsnels_ has joined #openstack-infra | 14:46 | |
| AJaeger | mordred, fungi, could either of you create two German mailing lists, please? https://review.openstack.org/#/c/284297/ and https://review.openstack.org/#/c/287221/ | 14:47 |
| *** vgridnev has quit IRC | 14:48 | |
| *** Jeffrey4l has joined #openstack-infra | 14:48 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Reduce template name for ansible bindep jobs https://review.openstack.org/287850 | 14:48 |
| *** vgridnev has joined #openstack-infra | 14:49 | |
| pabelanger | AJaeger: odyssey4me: ^ When you have a moment, that will fix the interrupter issue with tox an 128+ char path names | 14:49 |
| AJaeger | pabelanger: yes, we should avoid tox explosions! | 14:49 |
| *** _amrith_ is now known as amrith | 14:49 | |
| *** nelsnelson has quit IRC | 14:49 | |
| fungi | AJaeger: i remember seeing some discussion on the i18n ml about adding other translation-oriented, locale-specific lists. are there more than just those two? | 14:49 |
| pabelanger | AJaeger: fungi so, once we fix tox path names, I think we are good for bindep on ansible. cc odyssey4me | 14:50 |
| pabelanger | AJaeger: fungi we did notice curl is not installed when you add other-requirements.txt so we should likely add it into a dib element or puppet by default. To avoid all projects having to add it | 14:51 |
| yolanda | fungi, i enqueued the ref again | 14:51 |
| AJaeger | fungi, there'S a french one I guess as well - let me check... | 14:52 |
| AJaeger | fungi, yolanda : https://review.openstack.org/287712 - french translators list, could you review as well, please? | 14:53 |
| *** mrmartin has joined #openstack-infra | 14:53 | |
| *** pradk_ has joined #openstack-infra | 14:53 | |
| AJaeger | fungi, those three are all open mailing list requests that I could find directly | 14:53 |
| *** pradk has quit IRC | 14:53 | |
| *** pradk_ is now known as pradk | 14:53 | |
| *** andreww has quit IRC | 14:54 | |
| *** andreww has joined #openstack-infra | 14:54 | |
| fungi | AJaeger: thanks, will have a look at those in a moment | 14:55 |
| *** korzen has quit IRC | 14:55 | |
| *** eharney has joined #openstack-infra | 14:55 | |
| claudiub|2 | yolanda: os-win 0.2.3 is now on pypi. :D thanks! :) | 14:55 |
| fungi | pabelanger: oh, wow, we exceeded a pathname max in tox? | 14:56 |
| yolanda | woot! | 14:56 |
| *** dizquierdo has joined #openstack-infra | 14:56 | |
| pabelanger | fungi: Ya, we've done that before with ansible jobs | 14:57 |
| pabelanger | we have some long project names | 14:57 |
| *** andreww has quit IRC | 14:57 | |
| *** andreww has joined #openstack-infra | 14:57 | |
| *** andreww has quit IRC | 14:57 | |
| pabelanger | fungi: If you are fine with the testing approach with 287488, we can land that and I should be able to work on fedora-23 bindep on the next round of dib builds | 14:59 |
| fungi | so it seems | 14:59 |
| markus_z | We get now bug reports in Nova based on the "DocImpact" keyword in a merged commit. That report also gets tagged with "nova", which is redundant. Can we get rid of that? | 14:59 |
| *** yamamoto_ has quit IRC | 14:59 | |
| *** yamamoto_ has joined #openstack-infra | 15:00 | |
| *** ddieterly has quit IRC | 15:01 | |
| AJaeger | fungi, yolanda, if you have some more time ;) - there's a vhost request by rossella_s: https://review.openstack.org/#/c/284281/ - needed for https://review.openstack.org/#/c/284284/ . Could you look at these as well, please? | 15:01 |
| *** esker has quit IRC | 15:02 | |
| AJaeger | markus_z: discuss with armax: Ic9d2bdffd4acca3bb093f5037e62582961919ebf# | 15:02 |
| *** kevinsho_ has joined #openstack-infra | 15:02 | |
| markus_z | AJaeger: thanks! I didn't find that. | 15:03 |
| AJaeger | markus_z: wrong change - but it's that area ... | 15:03 |
| rossella_s | AJaeger, thanks a lot!!! | 15:03 |
| AJaeger | markus_z: having nova there makes it easier to assign to openstack-manuals - it's tagged correct for that ;) | 15:03 |
| *** kevinsho_ has quit IRC | 15:04 | |
| *** rcernin has joined #openstack-infra | 15:04 | |
| markus_z | AJaeger: Ah, you mean this is a preparation if we move a bug to the manuals project? | 15:04 |
| *** rkukura has joined #openstack-infra | 15:04 | |
| markus_z | AJaeger: https://bugs.launchpad.net/nova/+bug/1551782 for example | 15:04 |
| openstack | Launchpad bug 1551782 in OpenStack Compute (nova) " API: Improve os-migrateLive input parameters" [Undecided,New] | 15:04 |
| AJaeger | markus_z: it would help in that case | 15:05 |
| * AJaeger has to attend a meeting now, be back later... | 15:05 | |
| *** amotoki has joined #openstack-infra | 15:06 | |
| *** esker has joined #openstack-infra | 15:07 | |
| mtreinish | fungi, clarkb: mriedem and I were talking about e-r in -qa and we're seeing a lot of timeouts on the bot in the -qa channel and limited reporting to gerrit | 15:08 |
| mtreinish | is there anything in the logs for the bot | 15:08 |
| mtreinish | also the logstash queue graph is in a zig zag pattern which is never a good sign | 15:09 |
| fungi | pabelanger: on 287850, a 6-character reduction is sufficient to get their longest job name under the limit for tox to be happy? | 15:10 |
| *** kingia has quit IRC | 15:10 | |
| *** rkukura has quit IRC | 15:12 | |
| *** zz_dimtruck is now known as dimtruck | 15:13 | |
| *** icey_ is now known as icey | 15:14 | |
| *** nmagnezi has quit IRC | 15:14 | |
| csaba | AJaeger: got the manila +1 to https://review.openstack.org/287632 | 15:14 |
| pabelanger | fungi: if I counted correctly yes, let me confirm again | 15:15 |
| fungi | clarkb: sdague: so thinking about the mtu challenge a little more, this is probably going to get complicated to make dynamic, since i think i remember that the test scenarios have some hard-coded minimum mtu assumptions baked in as well? | 15:16 |
| fungi | so even if we have d-g calculate the tunneled mtu at interface mtu - 50 or something, the tests are going to blackhole traffic because they're assuming their tunnels-in-tunnels can do 1400, right? | 15:17 |
| pabelanger | fungi: ya, should drop us to 97 chars | 15:17 |
| pabelanger | err wait | 15:17 |
| fungi | pabelanger: 128-6 != 97, so i'm missing something | 15:18 |
| *** bhunter__ has joined #openstack-infra | 15:18 | |
| pabelanger | fungi: 121 chars | 15:18 |
| fungi | is the missing piece len('/home/jenkins/workspace/')? | 15:18 |
| pabelanger | I forgot the .tox/functional/bin part | 15:18 |
| fungi | oh, yep | 15:18 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:18 | |
| fungi | okay, cool by me then | 15:18 |
| pabelanger | /home/jenkins/workspace/gate-openstack-ansible-lxc_container_create-ansible-func-ubuntu-trusty-nv/.tox/functional/bin/pip is what I am using | 15:19 |
| pabelanger | hopefully zuulv3 allows for custom dir names :) | 15:19 |
| pabelanger | or mktemp | 15:19 |
| fungi | mktemp for the job workspace would be nice | 15:19 |
| *** annegentle has quit IRC | 15:21 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Added get all events endpoints https://review.openstack.org/287893 | 15:22 |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/jeepyb: notify_impact: remove project name from the tags list https://review.openstack.org/287908 | 15:22 |
| *** krtaylor has joined #openstack-infra | 15:23 | |
| *** jpr has quit IRC | 15:23 | |
| *** asettle has joined #openstack-infra | 15:23 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Disable failing tests test_gerrit_comment.py https://review.openstack.org/245389 | 15:24 |
| *** geekinutah has quit IRC | 15:25 | |
| jeblair | mtreinish: i'm looking at erbot's log file | 15:25 |
| jeblair | mtreinish: it is outputting things. | 15:25 |
| jeblair | (a lot of things. our gerrit is busy) | 15:26 |
| mtreinish | jeblair: ok, heh yeah it really is | 15:27 |
| jeblair | 2016-03-03 15:26:01,762 DEBUG recheckwatchbot: Console logs not ready for gate-nova-pypy-bindep 280653,9,e37fe2d | 15:27 |
| mtreinish | jeblair: yeah, I'm guessing we're having a lot of timeouts like that | 15:27 |
| mtreinish | because something is up with logstash, or it just got slower | 15:27 |
| *** ddieterly has joined #openstack-infra | 15:27 | |
| *** asettle has quit IRC | 15:28 | |
| *** asettle has joined #openstack-infra | 15:28 | |
| jeblair | 2016-03-03 15:28:01,906 INFO recheckwatchbot: Sending "Console logs not available after 13:24s for gate-nova-pypy-bindep 280653,9,e37fe2d" to #openstack-qa | 15:28 |
| fungi | pabelanger: on 287488, that won't cause the image build to fail presumably? the calls to yum there work? | 15:28 |
| jeblair | mtreinish: so it looks like the bot is working? | 15:28 |
| *** mtanino has joined #openstack-infra | 15:29 | |
| mtreinish | jeblair: yeah the bot seems to be working, its just almost always reporting timeouts :) | 15:29 |
| pabelanger | fungi: no, because we have some logic to skip packaging caching when dnf is used | 15:29 |
| pabelanger | fungi: https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/cache-bindep/install.d/50-cache-pkgs#L24 | 15:30 |
| jeblair | curl -XGET 'http://logstash.openstack.org:80/elasticsearch/logstash-2016.03.03/_search?size=10' -d '{"sort": {"@timestamp": {"order": "desc"}}, "query": {"query_string": {"query": "filename:\"console.html\" AND (message:\"[SCP] Copying console log\" OR message:\"Grabbing consoleLog\") AND build_status:\"FAILURE\" AND build_change:\"128940\" AND build_patchset:\"122\" AND build_name:\"gate-nova-pypy-bindep\"AND ... | 15:30 |
| jeblair | ... build_short_uuid:feaad18"}}}' | 15:30 |
| pabelanger | fungi: so, bindep will be installed but bindep-fallback will not be used | 15:30 |
| jeblair | mtreinish: ^ there's a request it recently tried to run and got 0 hits for | 15:30 |
| *** vgridnev has quit IRC | 15:30 | |
| pabelanger | once bindep is installed, we can test fedora-23 in the gate | 15:30 |
| *** angdraug has joined #openstack-infra | 15:31 | |
| openstackgerrit | Merged openstack-infra/project-config: Reduce template name for ansible bindep jobs https://review.openstack.org/287850 | 15:32 |
| dhellmann | is there any way to filter the zuul status page (http://status.openstack.org/zuul/) by queue? When I'm running releases, I'd like to be able to watch the release-related queues regardless of the project | 15:33 |
| *** achanda has joined #openstack-infra | 15:33 | |
| mtreinish | jeblair: right, that's the query it does up front to check if a job's results are on logstash yet | 15:33 |
| mtreinish | it tries that in a loop for ~13min and reports timeout if there isn't anything | 15:33 |
| openstackgerrit | Marton Kiss proposed openstack-infra/puppet-askbot: Add missing venv python to cron scripts https://review.openstack.org/287923 | 15:39 |
| *** achanda has quit IRC | 15:39 | |
| mrmartin | yolanda ^^^ | 15:39 |
| yolanda | that was fast! | 15:39 |
| openstackgerrit | Merged openstack-infra/system-config: Add openstack-de mailinglist https://review.openstack.org/284297 | 15:39 |
| *** yamamoto_ has quit IRC | 15:40 | |
| *** jamesmcarthur has joined #openstack-infra | 15:40 | |
| openstackgerrit | Merged openstack-infra/system-config: Add mailinglist openstack-i18n-de https://review.openstack.org/287221 | 15:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions https://review.openstack.org/287835 | 15:41 |
| openstackgerrit | Merged openstack-infra/system-config: Add a mailing list for the French i18n local group. https://review.openstack.org/287712 | 15:42 |
| *** ccrouch has quit IRC | 15:42 | |
| *** esker has quit IRC | 15:45 | |
| openstackgerrit | Qiming Teng proposed openstack/requirements: Bump openstacksdk version to 0.8.1 https://review.openstack.org/285599 | 15:48 |
| fungi | pabelanger: thanks, perfect | 15:48 |
| openstackgerrit | Merged openstack-infra/nodepool: Add an error log with the server fault message https://review.openstack.org/287481 | 15:48 |
| clarkb | dguitarbite: AJaeger gzip should be on all of our machines | 15:50 |
| *** Daisy has joined #openstack-infra | 15:50 | |
| *** jcoufal has quit IRC | 15:51 | |
| AJaeger | clarkb: dguitarbite used "zip" instead of "gzip" | 15:51 |
| *** jlanoux has quit IRC | 15:52 | |
| *** ccrouch has joined #openstack-infra | 15:52 | |
| *** kevinsho_ has joined #openstack-infra | 15:52 | |
| *** marg7175 has joined #openstack-infra | 15:52 | |
| *** kevinsho_ has quit IRC | 15:52 | |
| *** kevinsho_ has joined #openstack-infra | 15:53 | |
| clarkb | mordred: so shade release on monday maybe? | 15:54 |
| mordred | clarkb: yah | 15:55 |
| *** esker has joined #openstack-infra | 15:55 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 15:55 |
| clarkb | cool, and that will have ianw's fixes? I can make sure to restart nodepool services when that gets in if so | 15:55 |
| mordred | clarkb: I want to restart nodepool when the next patch lands | 15:55 |
| mordred | clarkb: yah | 15:55 |
| mordred | clarkb: (new patches to improve logging around timeouts) | 15:55 |
| *** yamamoto_ has joined #openstack-infra | 15:56 | |
| clarkb | great and that should have the fix for volume waiting which my launch node script updates use too. So win all around | 15:56 |
| EmilienM | fungi: fungi: hello! apevec and I found out that https://github.com/openstack-infra/system-config/commit/3894d932667de2d4dbbc89c58fc34b3d47294e6b makes Puppet OpenStack CI jobs failing, because rabbitmq is stealing keystone's uid now, I think we're missing SYS_UID_MIN in this file | 15:57 |
| aarefiev | ianw: hi, could you elaborate on your comment https://review.openstack.org/#/c/277843/ please | 15:58 |
| *** yamahata has joined #openstack-infra | 15:58 | |
| *** esker has quit IRC | 15:58 | |
| *** amotoki has quit IRC | 15:59 | |
| *** annegentle has joined #openstack-infra | 16:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable bindep / devstack cache for fedora23 nodes https://review.openstack.org/287488 | 16:00 |
| *** baoli has quit IRC | 16:01 | |
| *** baoli has joined #openstack-infra | 16:02 | |
| *** ogelbukh has joined #openstack-infra | 16:02 | |
| *** Qiming has quit IRC | 16:02 | |
| *** e0ne has quit IRC | 16:02 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on routes https://review.openstack.org/287943 | 16:02 |
| fungi | EmilienM: interesting. did you diff that against the vanilla one shipped by ubuntu? | 16:02 |
| EmilienM | fungi: we're running centos7 | 16:03 |
| fungi | i don't think it set SYS_UID_MIN either, but i don't have time to check just yet... in a meeting | 16:03 |
| fungi | oh, different file, sorry | 16:03 |
| EmilienM | fungi: no problem, we can tlak later | 16:03 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on routes https://review.openstack.org/287943 | 16:03 |
| fungi | EmilienM: but same story. i think i copied it from centos 6, maybe that's a little dated and needs updating? | 16:03 |
| EmilienM | I'll check | 16:04 |
| fungi | if memory serves i simply set the SYS_UID_MAX to the old UID_MIN-1 and then upped the UID_MIN by a couple thousand to give us breathing room for static id assignments (same for gid) | 16:04 |
| *** jpr has joined #openstack-infra | 16:06 | |
| *** jsavak has quit IRC | 16:06 | |
| *** rcernin has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack/requirements: Adding python-solumclient https://review.openstack.org/287313 | 16:07 |
| *** jistr is now known as jistr|call | 16:08 | |
| *** dmsimard has joined #openstack-infra | 16:08 | |
| *** thorst is now known as thorst_afk | 16:09 | |
| *** jpr has quit IRC | 16:10 | |
| EmilienM | wq! | 16:14 |
| EmilienM | oops | 16:14 |
| *** BobBall is now known as BobBall_AWOL | 16:14 | |
| *** rkukura has joined #openstack-infra | 16:14 | |
| mgagne | EmilienM left #openstack-infra channel | 16:15 |
| EmilienM | mgagne: I'll never leave :-P sorry for that | 16:15 |
| *** esker has joined #openstack-infra | 16:16 | |
| *** Daisy has quit IRC | 16:17 | |
| *** Daisy has joined #openstack-infra | 16:17 | |
| *** arxcruz has quit IRC | 16:18 | |
| *** sobersabre has quit IRC | 16:19 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/requirements: Bump ironic-lib to 1.1.0 https://review.openstack.org/287955 | 16:20 |
| clarkb | sdague: fungi has someone written a change ot dynamically calculate the MTU? | 16:20 |
| clarkb | if not I can do that real quick | 16:20 |
| fungi | clarkb: i haven't. i started to look at d-g and then remembered that neutron had hard-set the mtu in devstack too i think? | 16:21 |
| *** ddieterly has quit IRC | 16:21 | |
| clarkb | fungi: not sure, but definitely worth checking | 16:21 |
| *** Daisy has quit IRC | 16:22 | |
| fungi | which will complicate matters for the jobs running there if they make assumptions about what mtu we have for them | 16:22 |
| *** jcoufal has joined #openstack-infra | 16:22 | |
| openstackgerrit | Merged openstack/requirements: bump ceilometermiddleware upper constraints https://review.openstack.org/287416 | 16:22 |
| *** ddieterly has joined #openstack-infra | 16:22 | |
| openstackgerrit | Paul Bourke proposed openstack-infra/project-config: Add new experimental jobs for Kolla oraclelinux https://review.openstack.org/287957 | 16:23 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Set the bridge MTU dynamically https://review.openstack.org/287958 | 16:23 |
| clarkb | fungi: sdague ^ thats one half of it at least | 16:23 |
| *** jlanoux has joined #openstack-infra | 16:24 | |
| *** jsavak has joined #openstack-infra | 16:25 | |
| *** armax has joined #openstack-infra | 16:25 | |
| *** jpr has joined #openstack-infra | 16:26 | |
| *** yamahata has quit IRC | 16:26 | |
| *** jistr|call is now known as jistr | 16:26 | |
| *** vgridnev has joined #openstack-infra | 16:29 | |
| *** ryanpetrello has quit IRC | 16:30 | |
| *** Swami has joined #openstack-infra | 16:30 | |
| *** ddieterl_ has joined #openstack-infra | 16:31 | |
| *** sorantis has quit IRC | 16:31 | |
| *** yamahata has joined #openstack-infra | 16:32 | |
| *** jsavak has quit IRC | 16:32 | |
| *** jsavak has joined #openstack-infra | 16:33 | |
| *** ddieterly has quit IRC | 16:33 | |
| *** rkukura has quit IRC | 16:33 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Update upper-constraints for barbianclient 4.0.0 https://review.openstack.org/287417 | 16:35 |
| *** moravec has quit IRC | 16:35 | |
| *** jaypipes has quit IRC | 16:36 | |
| *** Jeffrey4l has quit IRC | 16:36 | |
| clarkb | fungi: just confirmed we are setting a config setting in d-g for devstack with the lower mtu | 16:38 |
| clarkb | so that change above should be sufficient | 16:38 |
| openstackgerrit | John Dickinson proposed openstack/requirements: bump python-swiftclient version https://review.openstack.org/287963 | 16:38 |
| fungi | clarkb: awesome. i'll prep a revert of my removal change and set it dependent on that | 16:39 |
| *** moravec has joined #openstack-infra | 16:40 | |
| *** harlowja_at_home has joined #openstack-infra | 16:40 | |
| *** bhunter__ is now known as bhunter71 | 16:40 | |
| *** korzen has joined #openstack-infra | 16:40 | |
| *** baoli has quit IRC | 16:41 | |
| *** yamahata has quit IRC | 16:41 | |
| *** baoli has joined #openstack-infra | 16:41 | |
| *** ddieterl_ is now known as ddieterly | 16:41 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 16:42 |
| *** scheuran has quit IRC | 16:42 | |
| *** exploreshaifali has joined #openstack-infra | 16:45 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: WIP: Decouple gerrit processing and es queries https://review.openstack.org/287973 | 16:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: WIP: Decouple gerrit processing and es queries https://review.openstack.org/287973 | 16:48 |
| *** bpokorny has joined #openstack-infra | 16:49 | |
| *** armax has quit IRC | 16:49 | |
| *** ryanpetrello has joined #openstack-infra | 16:51 | |
| *** armax has joined #openstack-infra | 16:51 | |
| *** sdague has quit IRC | 16:52 | |
| *** mikelk has quit IRC | 16:53 | |
| *** rossella_s has quit IRC | 16:53 | |
| *** rossella_s has joined #openstack-infra | 16:54 | |
| *** woodster_ has joined #openstack-infra | 16:55 | |
| *** asettle has quit IRC | 16:56 | |
| *** dmsimard has quit IRC | 16:57 | |
| *** esker has quit IRC | 16:58 | |
| nibalizer | good morning | 16:58 |
| *** thorst_afk is now known as thorst_ | 16:59 | |
| *** yamahata has joined #openstack-infra | 17:00 | |
| *** hashar has quit IRC | 17:01 | |
| *** dtantsur is now known as dtantsur|afk | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| *** ashleighfarnham has joined #openstack-infra | 17:01 | |
| wznoinsk | anteaya, are there any intentions at moving jenkins integration/scenario builds to use Kolla instead of devstack? | 17:01 |
| *** doug-fish has quit IRC | 17:02 | |
| *** jaosorior has quit IRC | 17:04 | |
| *** exploreshaifali has quit IRC | 17:04 | |
| *** ryanpetrello has quit IRC | 17:05 | |
| *** vincentll has quit IRC | 17:05 | |
| clarkb | fungi: greghaynes bkero funny story, a local build of the dib image using the same elements ubuntu-trusty uses is 4.8GB large | 17:05 |
| clarkb | so now I really don't know what is going on | 17:05 |
| *** ryanpetrello has joined #openstack-infra | 17:05 | |
| bkero | clarkb: your build system is different than the one the previous image was built on? | 17:05 |
| bkero | clarkb: run 'file' on the qcow2 file. Is it v2 or v3? | 17:05 |
| clarkb | v2 | 17:06 |
| clarkb | test-image.qcow2: QEMU QCOW Image (v2), 12870418432 bytes | 17:06 |
| bkero | Huh, ok | 17:06 |
| bkero | I'm generating v3 files, but that's lolarch | 17:06 |
| clarkb | I use the same ubuntu release on the same cpu architecture and the same versiosn of elements | 17:06 |
| bkero | Hmm | 17:06 |
| clarkb | so in theory it is the "same" | 17:06 |
| *** nmagnezi has joined #openstack-infra | 17:06 | |
| fungi | given your local ubuntu-trusty build is smaller even than the production devstack-trusty build, we've clearly got some crufty something going on on our builders | 17:06 |
| clarkb | fungi: yup I think it may be more related to that than anything else | 17:07 |
| clarkb | like the cache growing cruft | 17:07 |
| fungi | smaller by a gigabyte | 17:07 |
| fungi | curious what part of the cache is leaking into the images | 17:07 |
| *** dmsimard has joined #openstack-infra | 17:07 | |
| fungi | oh, unless dib is copying a large and crufty cached something into the image and then deleting it | 17:08 |
| fungi | clarkb: anyway, one thing still to check is whether zeroing the free space in the production raw images and then converting them to qcow2 significantly alters their size compared to the production qcow2 images | 17:08 |
| *** sbelous_ has quit IRC | 17:08 | |
| clarkb | yup I think it is still something to check as that may be an easy workaround | 17:09 |
| *** Guest5714 has quit IRC | 17:09 | |
| *** Hal has joined #openstack-infra | 17:09 | |
| *** doug-fish has joined #openstack-infra | 17:09 | |
| *** Hal is now known as Guest63919 | 17:10 | |
| openstackgerrit | David Lyle proposed openstack/requirements: Bump upper constraint django_openstack_auth https://review.openstack.org/287991 | 17:11 |
| mgagne | would there be interest in maintaining smaller Nodepool nodes to run smaller jobs like pep8, lint or docs? This could allow better utilization of the resources given by provider and/or run more jobs. | 17:11 |
| *** sfinucan_ has joined #openstack-infra | 17:11 | |
| *** IlyaG has joined #openstack-infra | 17:13 | |
| lucasagomes | clarkb, morning, if you have a time today do you mind taking a look at https://review.openstack.org/#/c/284036/ ? | 17:13 |
| clarkb | mgagne: sort of, there are two thing swe would have ot solve with that to make it viable. The first is in many cases our quota limits the number of instances regardless of size so we can't actually be bigger until we fix that (this is the easy one, send some emails get quota tweaks). The other is the nodepool scheduler would probably need to learn how to do multi axis scheduling in order to do this | 17:13 |
| clarkb | effectively | 17:13 |
| lucasagomes | it's a small patch and very wanted for ironic since that enables we to run tests in gate with our refence driver (ipmi ones) | 17:13 |
| *** sbelous_ has joined #openstack-infra | 17:13 | |
| clarkb | mgagne: it is something that would probably be worth mocking up in the nodepool load simulator to see how it shakes out | 17:14 |
| mgagne | clarkb: would there be a way to set max-servers per images/label? | 17:14 |
| *** sfinucan has quit IRC | 17:14 | |
| clarkb | mgagne: not today, what you can do today is have two "different" provider images with different flavor settings. | 17:15 |
| mgagne | clarkb: true, nice workaround =) | 17:15 |
| clarkb | mgagne: then you would assign the pep8 jobs and docs jobs to the smaller provider image and devstack and unittests to the larger provider image | 17:15 |
| *** apoorvad has joined #openstack-infra | 17:16 | |
| clarkb | mgagne: I think the trouble is that nodepool won't proper factor the two different sizes in when making its assignments against the max servers which is what I think needs fixing with multi axis scheduling support | 17:16 |
| mgagne | clarkb: because I believe those jobs could benefit from smaller and less performant nodes. | 17:16 |
| clarkb | but it is possible that this would just work | 17:16 |
| *** ccrouch1 has joined #openstack-infra | 17:16 | |
| clarkb | and the load simulator thing that jeblair wrote up should in theory check? | 17:16 |
| mgagne | I know some of those words =) | 17:16 |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: Promote gate-neutron-fwaas-dsvm-tempest from experimental to non-voting https://review.openstack.org/287996 | 17:17 |
| *** asettle has joined #openstack-infra | 17:17 | |
| clarkb | mgagne: basically I think the nodepool scheduler only considers max-servers and not the actual flavor size | 17:18 |
| *** ccrouch has quit IRC | 17:18 | |
| *** sbelous_ has quit IRC | 17:18 | |
| mgagne | clarkb: ok, having 2 providers would "fix" that right? | 17:18 |
| clarkb | mgagne: so if we have max servers of 100 nodepool will run a less informed mix of flavors rather than say the 50 small and 50 large you want or whatever | 17:18 |
| clarkb | mgagne: oh yes, sorry we can trick it with multiple providers | 17:19 |
| mgagne | clarkb: but I guess this would need more work than it looks where a job would still be able to run on bigger nodes if they are available as not all providers will be able to host smaller nodes | 17:19 |
| mgagne | should* | 17:19 |
| *** salv-orlando has joined #openstack-infra | 17:20 | |
| *** MarkAtwood has joined #openstack-infra | 17:20 | |
| *** salv-orl_ has quit IRC | 17:22 | |
| mordred | mgagne, clarkb: so, this problem _is_ a thing that we're hoping to solve with zuul v3, btw | 17:22 |
| clarkb | mgagne: that too is "tricky". We can tell gearman to run a job on A or B but it won't prefer one over the other iirc. Instead gearman says "here have a job" to the next hting in line | 17:22 |
| clarkb | mordred: yup | 17:22 |
| mgagne | clarkb: true | 17:22 |
| mordred | like, it's one of the tricky problems that has led to the design of zuul v3 | 17:23 |
| mordred | an in which we hope to even be able to just run pep8 jobs in a container on a reusable machine perhaps | 17:23 |
| mordred | but it does require a fairly deep rework of the scheduler logic | 17:23 |
| mgagne | mordred: I was waiting for that container suggestion to happen =) | 17:23 |
| jeblair | the problem is, we can't run it in a container on a host that we might use for something we actually trust | 17:24 |
| mordred | yah | 17:24 |
| jeblair | because container security is lolwat? | 17:24 |
| mordred | but we could potentially have some hosts that only work as container sources ... | 17:24 |
| mgagne | yea | 17:24 |
| mgagne | don't know why it wouldn't be possible | 17:24 |
| mordred | in any case, it's a capability we need to grow so that we can talk abou how to use it | 17:24 |
| *** andymaier has quit IRC | 17:25 | |
| jeblair | ayup | 17:25 |
| *** jaosorior has joined #openstack-infra | 17:26 | |
| clarkb | jeblair: and if you want to start pretending containers are secure a great chunk of our jobs won't function within them (you have to strip away so many rights that stuff starts to break) | 17:26 |
| mgagne | mordred: in the very short term, wouldn't it be possible to duplicate providers and dedicate some resources to smaller jobs/nodes? And tell Jenkins to choose whatever nodes fits the requirements (large or small) | 17:26 |
| mgagne | mordred: wouldn't be the great with low usage but on high usage day, smaller jobs would still be able to run on smaller nodes and free the queue of them | 17:27 |
| *** SumitNaiksatam has joined #openstack-infra | 17:27 | |
| mgagne | mordred: assuming we can find new resources too otherwise I guess we don't gain much | 17:28 |
| *** jvrbanac_ is now known as jvrbanac | 17:28 | |
| jeblair | mgagne: if you configured it to run on multiple types of nodes, that will have an effect on nodepool's allocator; i'm not entirely sure what the effect will be | 17:28 |
| mgagne | jeblair: unless code changed, I once saw jobs with OR operator in its label | 17:29 |
| jeblair | hopefully it will do what you want, and try to allocate nodes appropriately | 17:29 |
| jeblair | mgagne: yes, it works -- i'm just saying that we haven't used the ability for this precise purpose before | 17:29 |
| jeblair | mgagne: usually when we do it, the job is allowed to run on multiple types but be select one of them at runtime, so nodepool always knows exactly which one it needs | 17:30 |
| jeblair | mgagne: but i think when nodepool doesn't know, it assigns the demand to the first node type in the list | 17:30 |
| jeblair | mgagne: so you might end up inadvertently favoring one type | 17:30 |
| mgagne | the smaller one? :D | 17:31 |
| *** annegentle has quit IRC | 17:32 | |
| mgagne | but I could be wrong about that OR operator or it could be from a time where slaves were not single-use | 17:32 |
| *** jistr has quit IRC | 17:33 | |
| jeblair | mgagne: you aren't wrong -- i'm just saying we didn't use it like you are suggesting. | 17:33 |
| *** korzen has quit IRC | 17:34 | |
| mgagne | why can't I pervert systems to fit my selfish idea? :D | 17:34 |
| *** mrmartin has quit IRC | 17:35 | |
| *** piet has joined #openstack-infra | 17:35 | |
| jeblair | mgagne: zuul can specify which type of node to run a job on, so when we have jobs defined in jenkins with an ||, we actually have zuul pick one before it launches, so nodepool never has to guess. sorry if i didn't explain that well before :) | 17:35 |
| mgagne | from custom python functions right? | 17:35 |
| jeblair | mgagne: yep | 17:36 |
| mgagne | so Zuul doesn't really know if Nodepool has capacity for that label? | 17:36 |
| *** jaypipes has joined #openstack-infra | 17:36 | |
| fungi | my other concern with it is that this could be a premature optimization. more performant nodes finish static analysis jobs faster, and turn over faster clearing the way for new workloads. we probably mostly only gain on the node boot/delete overhead by tweaking this | 17:36 |
| *** degorenko is now known as _degorenko|afk | 17:37 | |
| jeblair | mgagne: right, the choice is not made due to capacity, it's based on the job being run (eg, this branch runs on X, this other branch runs on Y) | 17:37 |
| mgagne | fungi: I'm sure sphinx doesn't use all 8 vcpus =) | 17:37 |
| fungi | at the expense of much additional complexity | 17:37 |
| anteaya | wznoinsk: there have been no conversations of which I am aware of using kolla for our testing rather than devstack | 17:37 |
| jeblair | fungi: yep | 17:37 |
| fungi | mgagne: but also how long is it wasting those other 7 vcpus? | 17:37 |
| fungi | compared to a devstack-based job that's eating all 8 vcpus for an hour or more | 17:37 |
| mgagne | fungi: long enough to not allow 3-4 others jobs to run? | 17:37 |
| fungi | 3-4 other very quick jobs perhaps | 17:38 |
| mgagne | fungi: that's the whole idea | 17:38 |
| fungi | but a comparatively small percentage of the overall workload | 17:38 |
| *** sc68cal has joined #openstack-infra | 17:38 | |
| mgagne | fungi: think about providers too, they don't have infinite capacity, they can provide small nodes in greater quantity than bigger ones | 17:38 |
| fungi | i have no solid data on it, but would be surprised if it even netted us 10% additional capacity given the breakdown of our current job workloads | 17:39 |
| jeblair | (also, many of our providers oversubscribe us on cpu, so our "idle" vcpus may not actually be idle -- they may be running devstack) | 17:39 |
| anteaya | I don't even know what kolla does to be honest | 17:39 |
| mgagne | jeblair: I find oversubscribe doesn't work well at all, ask sdague | 17:39 |
| *** matrohon has quit IRC | 17:39 | |
| asselin_ | fungi, we have a job in the check queue that is stuck. The jenkins jobs is done, but jenkins failed and now the job is in the queue and unable to complete. What's the best way to restart it? | 17:40 |
| clarkb | asselin_: often that happens when your job crashes the test slave | 17:40 |
| fungi | a vast majority of our quotas are spent on devstack-based jobs, so optimizing the utilization of the other faster/smaller jobs doesn't necessarily buy us much | 17:40 |
| clarkb | asselin_: if possible the best way to work with that is to stop crashing the test slave | 17:40 |
| wznoinsk | anteaya thanks, fungi, pleia2 would you know any possible leaning towards Kolla in Jenkins builds instead of devstack in next sometime (year?) | 17:40 |
| asselin_ | clarkb, in this case we think its caused by the jenkins VM (unexpected) reboot which is time correlated. | 17:40 |
| anteaya | wznoinsk: why are you asking | 17:40 |
| jeblair | mgagne: i don't either, but i'm responsible for the architecture of very few public clouds. | 17:41 |
| jeblair | almost none | 17:41 |
| anteaya | wznoinsk: do you want to advocate for us doing so? | 17:41 |
| anteaya | if yes, then do advocate | 17:41 |
| asselin_ | but other than restarting zuul....is there anything else we can do? re-enqueue? promote? other hack? | 17:41 |
| wznoinsk | anteaya, I want to see what's the sentiment for Kolla in the OS infra | 17:41 |
| *** korzen has joined #openstack-infra | 17:41 | |
| *** vgridnev has quit IRC | 17:41 | |
| anteaya | wznoinsk: well my sentiment is "what is Kolla?" | 17:41 |
| clarkb | asselin_: stop rebooting it then? why is it rebooting? | 17:42 |
| anteaya | I'm not sure if any others have given it much thought | 17:42 |
| anteaya | if you want to expouse the beneifits of us looking at kolla, do expouse | 17:42 |
| asselin_ | clarkb, seems there was a failure on the esx compute node that caused all of its VMs to reboot | 17:42 |
| clarkb | anteaya: wznoinsk keep in mind thats largely QAs arena aiui | 17:42 |
| *** vgridnev has joined #openstack-infra | 17:42 | |
| anteaya | clarkb: great, thank you | 17:43 |
| wznoinsk | anteaya, I'll be honest here, only because you don't know what it is it doesn't mean it's not an option for jenkins builds ;-) | 17:43 |
| clarkb | we run the jobs but what goes in them is up to the project(s) | 17:43 |
| clarkb | asselin_: in that case the job should rerun and report | 17:43 |
| anteaya | wznoinsk: I didn't give any opioninon about whether it was an option for jenkins builds | 17:43 |
| *** zeih has joined #openstack-infra | 17:43 | |
| anteaya | wznoinsk: but you seem to have a reason for asking and I'm curious to know what it is | 17:43 |
| wznoinsk | clarkb, thought you'd be managing what scripts you provide for projects to be used for their builds... was anyone asking about Kolla do you know? | 17:44 |
| asselin_ | clarkb, seems that didn't happen..... | 17:44 |
| clarkb | asselin_: its a fifo so it goes to the end of the queue | 17:44 |
| clarkb | once its gearman turn comrs around again it should run the job | 17:44 |
| wznoinsk | anteaya, we don't plan to dev in Kolla for now, if there would be sentiment in projects/infra to start using it in next months then maybe we'd build knowledge from features/CI point of view | 17:44 |
| *** sridhar_ram has joined #openstack-infra | 17:45 | |
| anteaya | who is we in your statement? | 17:45 |
| asselin_ | clarkb, how can I inspect the gearman queue? | 17:45 |
| wznoinsk | team I'm in, 10 engins | 17:45 |
| anteaya | I don't know what that is either | 17:45 |
| wznoinsk | which would be 1 developers involved in Kolla | 17:45 |
| wznoinsk | 10 engineers | 17:45 |
| anteaya | well so far there is no push in infra to use kolla that I am aware of | 17:45 |
| wznoinsk | ok, thanks | 17:46 |
| anteaya | welcome | 17:46 |
| *** claudiub|2 has quit IRC | 17:47 | |
| mgagne | jeblair: I sure would like to know the performance of running those smaller jobs on smaller nodes. While larger nodes are busy running them, capacity isn't free to run devstack. Same for Puppet unit jobs. if it still takes 10m to run the puppet job but I can run 4x times jobs, we can stop wasting 6 larger nodes, use the capacity of 1/2 to create ~8 smaller nodes. And have ~4 free nodes for devstack. | 17:48 |
| *** denisra has joined #openstack-infra | 17:48 | |
| mgagne | that's my take on it | 17:48 |
| *** salv-orl_ has joined #openstack-infra | 17:49 | |
| *** david_cu has left #openstack-infra | 17:50 | |
| jeblair | mgagne: i'd like to as well, which is why i want to change the nodepool allocator in zuulv3 | 17:51 |
| clarkb | asselin_: run status against the gearman socket | 17:51 |
| *** jsavak has quit IRC | 17:51 | |
| *** sc68cal_ has joined #openstack-infra | 17:51 | |
| *** rkukura has joined #openstack-infra | 17:51 | |
| *** sputnik13 has joined #openstack-infra | 17:51 | |
| clarkb | anteaya: wznoinsk again thats really a better question for the qa team | 17:52 |
| anteaya | clarkb: yup | 17:52 |
| clarkb | anteaya: wznoinsk devstack is a qa project so it gets wide use | 17:52 |
| *** salv-orlando has quit IRC | 17:52 | |
| therve | lifeless, In case you see this, https://github.com/testing-cabal/testtools/pull/220 fixes a pretty big performance hit on testr | 17:52 |
| therve | lifeless, Several minutes lost for each runs in the gate | 17:53 |
| *** aarefiev22 has joined #openstack-infra | 17:53 | |
| therve | Would be cool to push that | 17:53 |
| *** baoli has quit IRC | 17:53 | |
| anteaya | sounds like management at some location is using infra as a thymus test to decide what to put dev resourses on | 17:53 |
| *** kzaitsev_mb has quit IRC | 17:54 | |
| *** mrmartin has joined #openstack-infra | 17:54 | |
| anteaya | or what infra is doing as a thymus test | 17:54 |
| *** sc68cal has quit IRC | 17:54 | |
| openstackgerrit | Merged openstack-infra/puppet-askbot: Add missing venv python to cron scripts https://review.openstack.org/287923 | 17:55 |
| *** e0ne has joined #openstack-infra | 17:55 | |
| *** sfinucan_ has quit IRC | 17:55 | |
| *** baoli has joined #openstack-infra | 17:55 | |
| openstackgerrit | Merged openstack/requirements: Change global requirements forcing cliff!=1.17.0 https://review.openstack.org/282886 | 17:55 |
| *** dmsimard has quit IRC | 17:56 | |
| *** ybathia has joined #openstack-infra | 17:56 | |
| *** yamamoto_ has quit IRC | 17:57 | |
| anteaya | not thymus, litmus | 17:58 |
| *** marg7175 has quit IRC | 17:58 | |
| anteaya | wow, I can't even blame that one on poor spelling | 17:58 |
| openstackgerrit | Merged openstack-infra/puppet-elasticsearch: Make network.publish_host configurable https://review.openstack.org/279690 | 17:58 |
| asettle | Hey infra team - wondering if someone is able to help me. I posted a patch in openstack-manuals nearly 2 days ago now. The first patch had an issue with the gate so I posted up another patch, but now it will not at all go through the gate. The patch itself isn't even in zuul. Here's the patch: https://review.openstack.org/#/c/286530/5/ I'm stumped | 17:58 |
| bswartz | that was a weird autocorrect | 17:58 |
| anteaya | bswartz: that was all my fault | 17:59 |
| anteaya | obviously today is not my best language day | 17:59 |
| anteaya | asettle: it depends on a patch which is in merge conflict: https://review.openstack.org/#/c/252791/ | 18:00 |
| asettle | OHHHHHH *duh* | 18:00 |
| asettle | Wait | 18:00 |
| asettle | Why would it go through the first time though? | 18:00 |
| anteaya | I don't see that jenkins has ever commented on 286530 | 18:01 |
| fungi | asettle: because its dependency might not have been in a merge conflict with the state of the repo at the time it was initially tested | 18:01 |
| *** flepied1 has joined #openstack-infra | 18:01 | |
| *** marg7175 has joined #openstack-infra | 18:01 | |
| *** salv-orl_ has quit IRC | 18:02 | |
| fungi | changes are constantly merging to each repository so the state of changes in development will also shift over time | 18:02 |
| asettle | Haha thank you anteaya and fungi :) I thought I was going crazy... turns out I can't read my own commit messages :p | 18:03 |
| *** korzen has quit IRC | 18:03 | |
| asettle | #longweek | 18:03 |
| anteaya | welcome to the club | 18:03 |
| anteaya | yup | 18:03 |
| *** salv-orl_ has joined #openstack-infra | 18:03 | |
| *** flepied has quit IRC | 18:03 | |
| *** dizquierdo has quit IRC | 18:04 | |
| *** baoli has quit IRC | 18:04 | |
| *** zeih has quit IRC | 18:04 | |
| *** baoli has joined #openstack-infra | 18:04 | |
| *** sarob has joined #openstack-infra | 18:05 | |
| *** flepied has joined #openstack-infra | 18:05 | |
| fungi | okay, going to go find some lunch, then come back and start putting together the final batch of contributor invites for austin | 18:05 |
| anteaya | enjoy lunch | 18:06 |
| fungi | thanks | 18:06 |
| *** jsavak has joined #openstack-infra | 18:06 | |
| *** flepied1 has quit IRC | 18:08 | |
| thingee | clarkb: getting volume path based off volume uuid support in brick https://review.openstack.org/#/c/268405/4/brick_cinderclient_ext/client.py | 18:08 |
| *** lucasagomes is now known as lucas-afk | 18:09 | |
| thingee | clarkb: can ask hemna on #openstack-cinder more about it. | 18:09 |
| rm_work | is jenkins-job-builder in pypi or do I REALLY have to clone it and use setup.py to install? all the docs say the latter but it seems weird that it isn't packaged somewhere? | 18:09 |
| rm_work | nm found it in pypi :P | 18:10 |
| *** jlanoux has quit IRC | 18:10 | |
| rm_work | i am dumb | 18:10 |
| rm_work | docs could be updated though, as a note :) | 18:10 |
| *** mriedem has quit IRC | 18:10 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 18:10 |
| *** salv-orl_ has quit IRC | 18:10 | |
| *** salv-orlando has joined #openstack-infra | 18:11 | |
| *** nmagnezi has quit IRC | 18:12 | |
| *** _nadya_ has quit IRC | 18:13 | |
| *** sbalukoff has quit IRC | 18:15 | |
| AJaeger | anteaya, sdague, dhellmann, fungi, bswartz: Here's a relevant review for the tc discussion about limited resources in our CI system: https://review.openstack.org/287632 adds the *fourth* jobs for glusterfs, seems a second driver but still... | 18:15 |
| anteaya | rm_work: well you know best what doc is out of date and how it should read, it would be lovely if you were inclined to submit a patch | 18:16 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 18:16 |
| *** sbelous_ has joined #openstack-infra | 18:17 | |
| *** _nadya_ has joined #openstack-infra | 18:18 | |
| *** ihrachys has quit IRC | 18:18 | |
| pleia2 | not around much this morning, hope to be back in the afternoon | 18:18 |
| jeblair | AJaeger: i think sdague has raised an interesting point, but i'd really like it if we could avoid the suggestion that we have somehow come to a conclusion, or worse, that the conclusion is that people should not run ci jobs. | 18:18 |
| *** exploreshaifali has joined #openstack-infra | 18:19 | |
| *** sarob has quit IRC | 18:19 | |
| asselin_ | clarkb, I couldn't see anything unusual. Anyway, I tried zuul promote on the check queue and it restarted it at the bottom while keeping all its time info. That's what I needed. | 18:19 |
| AJaeger | jeblair: Yes, I understood there was no conclusion. But sometimes it's good to look at one example to understand issues. | 18:20 |
| *** asettle has quit IRC | 18:20 | |
| jeblair | AJaeger: sure, but you're singling something out and i don't see the reason why | 18:20 |
| *** mriedem has joined #openstack-infra | 18:20 | |
| AJaeger | jeblair: since it just came in as review | 18:20 |
| zaro | morning | 18:20 |
| *** annegentle has joined #openstack-infra | 18:20 | |
| *** _nadya_ has quit IRC | 18:21 | |
| jeblair | AJaeger: yes, but unless i'm grossly misunderstanding you, you seem to be suggesting that somehow this job is too much | 18:21 |
| AJaeger | jeblair: I'll continue reviewing it as usual... | 18:21 |
| *** tphummel has joined #openstack-infra | 18:21 | |
| *** mtanino__ has quit IRC | 18:22 | |
| AJaeger | jeblair: Yes, I'm wondering why there are four variants of a driver | 18:22 |
| dhellmann | AJaeger, jeblair : I tend to agree about continuing to add jobs, though I do also wonder why there are so many variants of what seems like the same driver. | 18:23 |
| jeblair | AJaeger: i think that's a fine question to ask | 18:23 |
| jeblair | i'm just really worried about us sending some kind of message that people should be afraid to add jobs to do the testing they need | 18:23 |
| dhellmann | I don't want anyone to not add a job, or remove a job, "just because" but I agree with sdague that it makes some sense to review our resource use to see if we can be more economical | 18:24 |
| jeblair | dhellmann: yes, i hope everyone does that more or less continuously | 18:24 |
| jeblair | and if we need the occasionaly reminder, that's cool too | 18:24 |
| dhellmann | jeblair : I think the fear is we're doing it less continuously than we ought, so -- right, a reminder | 18:25 |
| jeblair | it's easy to lose sight of that | 18:25 |
| * dhellmann nods | 18:25 | |
| jeblair | i just don't want a perfectly valid question which sdague asked to send us off on a witch hunt :) | 18:25 |
| *** apoorvad has quit IRC | 18:25 | |
| dhellmann | i concur | 18:25 |
| openstackgerrit | Adam Harwell proposed openstack-infra/jenkins-job-builder: Update docs to include installation via pypi https://review.openstack.org/288051 | 18:26 |
| *** annegentle has quit IRC | 18:26 | |
| *** asettle has joined #openstack-infra | 18:26 | |
| rm_work | anteaya: ^^ thanks for the callout on that :P | 18:27 |
| rm_work | "someone should update that" is such a cop-out, lol | 18:27 |
| *** zeih has joined #openstack-infra | 18:28 | |
| AJaeger | jeblair: Understood and agreed on. | 18:29 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Remove non-voting jobs from the gate pipeline. https://review.openstack.org/287464 | 18:30 |
| *** dizquierdo has joined #openstack-infra | 18:30 | |
| anteaya | rm_work: thanks for the patch :) | 18:30 |
| mordred | rm_work: woot. thanks! | 18:30 |
| *** marg7175_ has joined #openstack-infra | 18:30 | |
| anteaya | morning zaro | 18:30 |
| anteaya | pleia2: hope all is well, thanks for the status update | 18:30 |
| *** jaosorior has quit IRC | 18:31 | |
| *** dizquierdo has quit IRC | 18:31 | |
| *** geekinutah has joined #openstack-infra | 18:31 | |
| *** abregman has quit IRC | 18:32 | |
| *** sdague has joined #openstack-infra | 18:32 | |
| anteaya | AJaeger jeblair dhellmann I agree with the sentiments expressed in the conversation thus far and also am curious why we need to test four variants of a driver | 18:33 |
| *** marg7175 has quit IRC | 18:34 | |
| *** sambetts is now known as sambetts|afk | 18:35 | |
| *** daemontool_ has quit IRC | 18:38 | |
| *** kiran-r has quit IRC | 18:38 | |
| *** ccrouch1 has quit IRC | 18:38 | |
| *** _nadya_ has joined #openstack-infra | 18:39 | |
| *** tphummel has quit IRC | 18:40 | |
| *** baoli has quit IRC | 18:41 | |
| *** rkukura has quit IRC | 18:41 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Dont include plugin definition in cmd checks https://review.openstack.org/288061 | 18:41 |
| *** sbalukoff has joined #openstack-infra | 18:42 | |
| *** baoli has joined #openstack-infra | 18:43 | |
| *** baoli has quit IRC | 18:43 | |
| *** rcernin has joined #openstack-infra | 18:44 | |
| *** zeih has quit IRC | 18:44 | |
| *** piet has quit IRC | 18:45 | |
| *** gyee has quit IRC | 18:46 | |
| EmilienM | mgagne: fyi https://review.openstack.org/#/c/242718/ (see last comment) | 18:49 |
| *** geekinutah has quit IRC | 18:50 | |
| *** SumitNaiksatam has quit IRC | 18:50 | |
| mgagne | EmilienM: so it was fine 6 minutes ago but then failed? :-/ | 18:51 |
| EmilienM | mgagne: it's maybe random | 18:51 |
| mgagne | EmilienM: I can bring it to the attention of our netadmins | 18:52 |
| *** IlyaG_ has joined #openstack-infra | 18:52 | |
| mgagne | if infra tells me they didn't reload anything on their side | 18:52 |
| EmilienM | mgagne: thx, just an fyi | 18:52 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add config option for git clone URL https://review.openstack.org/169469 | 18:53 |
| *** sarob has joined #openstack-infra | 18:53 | |
| *** marg7175_ has quit IRC | 18:54 | |
| *** tphummel has joined #openstack-infra | 18:55 | |
| *** IlyaG has quit IRC | 18:55 | |
| *** shardy has quit IRC | 18:56 | |
| *** e0ne has quit IRC | 18:57 | |
| openstackgerrit | Merged openstack/requirements: Raise upper-constraint on python-cinderclient to 1.6.0 https://review.openstack.org/287226 | 18:58 |
| *** jcoufal has quit IRC | 18:58 | |
| *** yamamoto_ has joined #openstack-infra | 18:58 | |
| *** baoli has joined #openstack-infra | 18:58 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:58 | |
| *** marg7175 has joined #openstack-infra | 19:00 | |
| *** moravec has quit IRC | 19:01 | |
| *** moravec1 has joined #openstack-infra | 19:01 | |
| *** baoli has quit IRC | 19:02 | |
| *** andymaier has joined #openstack-infra | 19:02 | |
| *** baoli has joined #openstack-infra | 19:03 | |
| anteaya | AJaeger: oh good thank you | 19:04 |
| *** asettle has quit IRC | 19:04 | |
| *** tphummel has quit IRC | 19:04 | |
| anteaya | EmilienM: can you take another look at 287464 when you get a chance, my patch to remove non-voting jobs from the gate | 19:04 |
| *** apoorvad has joined #openstack-infra | 19:04 | |
| *** sridhar_ram has quit IRC | 19:04 | |
| EmilienM | anteaya: sure thing, doing it right now | 19:04 |
| *** sridhar_ram has joined #openstack-infra | 19:05 | |
| anteaya | EmilienM: thank you | 19:05 |
| *** yamamoto_ has quit IRC | 19:05 | |
| EmilienM | anteaya: done. +1. Do you know why it was wrong at first? | 19:05 |
| anteaya | thanks | 19:05 |
| *** jsavak has quit IRC | 19:05 | |
| openstackgerrit | Brian Demers proposed openstack-infra/project-config: Change networking-cisco stable all to use networking-cisco-release https://review.openstack.org/288066 | 19:06 |
| anteaya | my local copy must have been stale | 19:06 |
| *** thiagop has quit IRC | 19:06 | |
| EmilienM | ok no problem, good | 19:06 |
| anteaya | so I rm'd it and git cloned fresh and now it matches | 19:06 |
| *** jsavak has joined #openstack-infra | 19:06 | |
| anteaya | sorry for the confusion yesterday | 19:06 |
| anteaya | thank you | 19:06 |
| EmilienM | anteaya: no problem | 19:06 |
| anteaya | :) | 19:06 |
| AJaeger | anteaya: next time, just run: git reset --hard origin/master | 19:07 |
| AJaeger | no need to clone again | 19:08 |
| anteaya | thank you | 19:08 |
| *** austin81 has joined #openstack-infra | 19:08 | |
| anteaya | I'll try to remember to do so | 19:08 |
| AJaeger | anteaya: I see that your change was done on master - I really suggest to always work on branches | 19:09 |
| AJaeger | anteaya: sorry, wrong line | 19:09 |
| *** _nadya_ has quit IRC | 19:09 | |
| anteaya | I thought I had | 19:09 |
| AJaeger | anteaya: I'd like to take back that statement | 19:09 |
| anteaya | oh okay thanks | 19:09 |
| *** ociuhandu has quit IRC | 19:09 | |
| anteaya | I'm usually wrong as it is, so welcome correction | 19:10 |
| sdake | AJaeger any chance we can get some action on this https://review.openstack.org/#/c/287321/ | 19:10 |
| anteaya | but glad I did okay here this time | 19:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Exclude neutronclient 4.1.0 https://review.openstack.org/288067 | 19:10 |
| sdake | our centos gates fail during the US daytime because the US is working and the internet is slower | 19:10 |
| sdake | they fail on timeouts | 19:10 |
| sdake | at night the gate runs in 30 minutes | 19:10 |
| sdake | (as in 3am my time) | 19:10 |
| sdake | what would be ideal is a centos mirror | 19:10 |
| sdake | someone did this for ubuntu recently not sure who | 19:11 |
| *** rkukura has joined #openstack-infra | 19:11 | |
| sdake | but if we coud lget a temporary 90 minute gate until we have a proper centos mirror that would be helpful ;) | 19:11 |
| pabelanger | sdake: yes, I am working on rhel afs mirrors | 19:11 |
| pabelanger | hope to start back up on it next week | 19:11 |
| sdake | otherwise i have to keep typing recheck until everything is green | 19:11 |
| sdake | pabelanger will that be centos + epel ? | 19:11 |
| pabelanger | yes | 19:11 |
| sdake | nice | 19:11 |
| sdake | in the meantime, can we get a 90 minut egate ?: ) | 19:12 |
| *** moravec1 has quit IRC | 19:12 | |
| sdake | its really more abusive for me to type recheckc in then to have a 90 minut egate for a couple of our 10 jobs | 19:12 |
| apevec | pabelanger, rhel? | 19:12 |
| apevec | or centos? | 19:12 |
| pabelanger | sdake: moving to bindep will help too | 19:12 |
| *** sbelous_ has quit IRC | 19:12 | |
| pabelanger | sdake: because we cache packages on the image | 19:12 |
| sdake | pabelanger bindep is what? | 19:12 |
| *** piet has joined #openstack-infra | 19:12 | |
| pabelanger | apevec: yes, centos + fedora | 19:12 |
| sdake | pabelanger would you mind keeping me in the loop on the centos + epel mirrors being finished sos we can take advantage of them when they are done | 19:13 |
| pabelanger | sdake: https://git.openstack.org/cgit/openstack-infra/bindep | 19:13 |
| pabelanger | will be the new way of how we create DIBs | 19:13 |
| pabelanger | sdake: well, once they come online, they should be seemless for you. We'll likely send an email to ML about it | 19:14 |
| *** esikachev has quit IRC | 19:14 | |
| *** kzaitsev_mb has quit IRC | 19:14 | |
| *** asettle has joined #openstack-infra | 19:15 | |
| sdake | the ubuntu mirrors were not seamless | 19:15 |
| sdake | but i'll take your word for it ;) | 19:15 |
| sdake | we are using the fastestmirror plugin if that is what you mean by seamless | 19:15 |
| sdake | atm we override the ubuntu mirrors in our dockerfiles i htink | 19:16 |
| *** baoli has quit IRC | 19:16 | |
| openstackgerrit | Armando Migliaccio proposed openstack/requirements: OUT https://review.openstack.org/286587 | 19:16 |
| *** amitgandhinz has quit IRC | 19:16 | |
| *** ifarkas has quit IRC | 19:17 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:18 | |
| *** SumitNaiksatam has quit IRC | 19:18 | |
| *** asettle has quit IRC | 19:20 | |
| *** _nadya_ has joined #openstack-infra | 19:21 | |
| AJaeger | armax: is networking-cisco leaving the big neutron tent? Or why are they changing ACLs? https://review.openstack.org/#/c/288066 | 19:21 |
| AJaeger | sdake: reviewed | 19:21 |
| armax | AJaeger: not yet | 19:22 |
| AJaeger | armax: do you want to -1 that - or is it ok as is? | 19:22 |
| armax | AJaeger: I sent an update to the ML | 19:22 |
| armax | I just did | 19:22 |
| AJaeger | thanks | 19:22 |
| fungi | clarkb: reviewing your dynamic mtu change, is gate-tempest-dsvm-multinode-full expected to fail? (gate-grenade-dsvm-multinode and gate-tempest-dsvm-neutron-multinode-full passed fwiw) | 19:23 |
| *** ccrouch has joined #openstack-infra | 19:23 | |
| *** jtomasek_ has joined #openstack-infra | 19:23 | |
| clarkb | fungi: I think it is less reliable for other reasons | 19:24 |
| clarkb | fungi: I checked the mtu and it came out as 1450 so that worked | 19:24 |
| pabelanger | sdake: bindep won't do too much for your gate time actually, but the afs mirror will. I thought you were using bare nodes, but that is not the case | 19:24 |
| fungi | rm_work: what's most awesome of all is that someone _did_ update that ;) | 19:24 |
| rm_work | lol | 19:26 |
| *** _nadya_ has quit IRC | 19:27 | |
| openstackgerrit | Austin Clark proposed openstack-infra/infra-specs: Deploy-Stackviz spec https://review.openstack.org/287373 | 19:27 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Update docs to include installation via pypi https://review.openstack.org/288051 | 19:27 |
| austin81 | AJaeger: Fixed your comments on ^^ | 19:27 |
| austin81 | mtreinish, sdague: https://review.openstack.org/287373 | 19:28 |
| sdake | AJaeger thanks :) | 19:29 |
| *** amrith is now known as _amrith_ | 19:30 | |
| dims | fungi : clarkb : http://codesearch.openstack.org/ has been flaky for me for a couple of days now. is there an ongoing issue with that? | 19:31 |
| anteaya | dims: it restarts everytime a new project is added and has to be indexed | 19:32 |
| fungi | dims: as in sometimes refurns a 5xx error for a minute when you first go to use it? | 19:32 |
| fungi | if so, that's what anteaya just described | 19:32 |
| anteaya | taron: is working on an upstream patch so that reindexing doesn't require a restart | 19:32 |
| dims | fungi : that's one. i also see weird errors in the UI too | 19:32 |
| fungi | dims: what errors? | 19:32 |
| anteaya | but I haven't heard a progress update in a bit | 19:32 |
| *** electrofelix has quit IRC | 19:33 | |
| dims | anteaya : thanks. | 19:33 |
| anteaya | dims: I'm not familiar with the ui errors complaint | 19:33 |
| anteaya | dims: do expand | 19:33 |
| dims | anteaya : fungi : let me try a few times and ping back | 19:33 |
| anteaya | dims: sure, screenshots would be grand if you have them | 19:34 |
| dims | anteaya : ack | 19:34 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost" https://review.openstack.org/288080 | 19:35 |
| *** sdake_ has joined #openstack-infra | 19:35 | |
| *** sdake has quit IRC | 19:35 | |
| *** fungi is now known as backup_openstack | 19:36 | |
| anteaya | backup_openstack: see in a while | 19:36 |
| backup_openstack | Jeremy Stanley proposed openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost" https://review.openstack.org/288080 | 19:36 |
| *** backup_openstack is now known as fungi | 19:37 | |
| fungi | oh, it did echo | 19:37 |
| fungi | for some reason i didn't see it | 19:37 |
| * Clint squints. | 19:37 | |
| fungi | clarkb: ^ the thing | 19:37 |
| fungi | i approved your dynamic mtu solution | 19:38 |
| *** Sukhdev has joined #openstack-infra | 19:38 | |
| fungi | so once it merges we can try that again | 19:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Use isinstance not type for dict conversions https://review.openstack.org/288083 | 19:38 |
| clarkb | cool | 19:39 |
| *** pvaneck has joined #openstack-infra | 19:39 | |
| mordred | Shrews, rcarrillocruz: ^^ that | 19:41 |
| mordred | Shrews, rcarrillocruz: I think should fix the neutronclient 4.1.0 + shade breakage | 19:41 |
| *** ihrachys has joined #openstack-infra | 19:41 | |
| openstackgerrit | Merged openstack-infra/puppet-httpd: Add SSL Procotol and Cipher config to default vhost https://review.openstack.org/286610 | 19:43 |
| *** abregman has joined #openstack-infra | 19:44 | |
| openstackgerrit | Armando Migliaccio proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0 https://review.openstack.org/286587 | 19:45 |
| armax | dhellmann: ^ | 19:45 |
| *** _amrith_ is now known as amrith | 19:45 | |
| armax | this needed a rebase | 19:45 |
| *** amitgandhinz has joined #openstack-infra | 19:46 | |
| *** sgordon has quit IRC | 19:46 | |
| *** moravec has joined #openstack-infra | 19:47 | |
| *** jpr has quit IRC | 19:47 | |
| *** exploreshaifali has quit IRC | 19:48 | |
| *** marg7175 has quit IRC | 19:49 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 19:51 |
| rcarrillocruz | mordred: cool thx, i'll try in a bit | 19:52 |
| rcarrillocruz | so issue on shade side then | 19:52 |
| rcarrillocruz | ? | 19:52 |
| *** hashar has joined #openstack-infra | 19:52 | |
| rcarrillocruz | reading scrollback on neutron channel... | 19:52 |
| *** jordan__ has quit IRC | 19:52 | |
| *** vgridnev has quit IRC | 19:53 | |
| rcarrillocruz | i see | 19:54 |
| *** boris-42 has joined #openstack-infra | 19:55 | |
| *** annegentle has joined #openstack-infra | 19:55 | |
| mordred | infra-root: I may want to either cut a shade release sooner than next week, or figure out how to cut a release with only that patch above in it | 19:55 |
| mordred | otherwise, anyone using shade out in the world with neutron who got a neutronclient upgrade will be dead in the water | 19:55 |
| fungi | mordred: i think the ff rush is over anyway | 19:55 |
| fungi | the gate is mostly a ghost town | 19:56 |
| anteaya | yeah the graphs aren't working at full strength right now | 19:56 |
| fungi | i'd go for it | 19:56 |
| mordred | fungi: cool. good. then maybe once that lands we can cut the release | 19:56 |
| nibalizer | mordred: 287504 or a different patch? | 19:56 |
| mordred | well, once rcarrillocruz verifies it fixes the issue :) | 19:56 |
| dims | fungi : anteaya : https://www.evernote.com/l/AZx_jXOqMGVDk782dDREO___fujv9bEh0gE seems to happen when the UI is up and a click on the button hits a 503 | 19:56 |
| mordred | nibalizer: yah. 287504 | 19:56 |
| anteaya | I'm not sure I'd call 13 in the gate a ghost town | 19:56 |
| *** ayoung has quit IRC | 19:56 | |
| anteaya | but it is less that we have had all week | 19:56 |
| *** vgridnev has joined #openstack-infra | 19:56 | |
| fungi | dims: my gut says it started reindexing/restarting when apache tried to hand off the request there | 19:57 |
| mordred | neutron now returns a dict subclass, not a pure dict - so our bug there in how we detect a dict gets exposed and neutron functions stop working | 19:57 |
| nibalizer | mordred: this looks like it just adds somee keystone v3 features | 19:57 |
| *** angdraug has quit IRC | 19:57 | |
| anteaya | dims: the server broke down, that is a helpful error | 19:57 |
| mordred | nibalizer: oh. piddle. no, wrong patch | 19:57 |
| fungi | dims: though i still don't buy the theory that this is only happening when we add new repos, it's far too frequent | 19:57 |
| mordred | 288083 | 19:57 |
| dims | fungi : 503 is very perisistent | 19:57 |
| dims | fungi : yes | 19:58 |
| *** dtardivel has quit IRC | 19:58 | |
| dims | too frequent | 19:58 |
| anteaya | fungi: okay fair | 19:58 |
| dims | at least for 2 days straight now | 19:58 |
| *** ihrachys_ has joined #openstack-infra | 19:58 | |
| fungi | i'm getting a service unavailable error from codesearch.o.o right now, in fact | 19:58 |
| anteaya | perhaps restarts include when we add new repos | 19:58 |
| fungi | haven't seen a new repo added for some hours | 19:58 |
| *** jtomasek_ has quit IRC | 19:58 | |
| anteaya | is is restarting? | 19:58 |
| *** jcooley_ has joined #openstack-infra | 19:59 | |
| dims | fungi : right | 19:59 |
| dhellmann | armax : ack, +2a let's see how the jobs do this time | 19:59 |
| dims | almost feels like it comes back up and then falls right over again anteaya and fungi | 19:59 |
| dims | over and over | 19:59 |
| fungi | no hound in the process list on the server at the moment | 19:59 |
| *** jcooley_ has quit IRC | 20:00 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| *** ihrachys has quit IRC | 20:01 | |
| *** jcooley_ has joined #openstack-infra | 20:01 | |
| * dims stepping out. back in an hour | 20:01 | |
| *** nwkarsten has joined #openstack-infra | 20:01 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Subscribe to project_config for create-hound-config https://review.openstack.org/261065 | 20:01 |
| pabelanger | fungi: nibalizer: I think that will help code.o.o ^ | 20:01 |
| ianw | clarkb: OMFG!!!!!!!!!!!!!!!!!!!!! https://jenkins04.openstack.org/job/gate-tempest-dsvm-platform-fedora23-nv/31/consoleFull | 20:01 |
| pabelanger | if it is the same issue I looked at before | 20:01 |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| ianw | that is one of the first good fedora 23 runs | 20:02 |
| asselin_ | anyone want to review the simple patch that's been hanging around for a while? https://review.openstack.org/#/c/267250/1 | 20:02 |
| ianw | wow, that only took, like, 4 months :) | 20:02 |
| *** amitgandhinz has quit IRC | 20:02 | |
| fungi | so... i issued a `sudo service hound start` and now there's a houndd in the process table | 20:02 |
| anteaya | dims: thanks for letting us know | 20:02 |
| anteaya | fungi: yay? | 20:03 |
| fungi | and codesearch.o.o is working for me again | 20:03 |
| fungi | i have a feeling something is stopping houndd or causing it to die | 20:03 |
| *** maishsk has joined #openstack-infra | 20:03 | |
| anteaya | hmmmm | 20:03 |
| fungi | but i welcome others to dig deeper since i have other things i need to work on at the moment | 20:03 |
| *** jsavak has quit IRC | 20:04 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Clarify oscc_file_contents instructions https://review.openstack.org/282010 | 20:04 |
| *** jsavak has joined #openstack-infra | 20:04 | |
| *** ccrouch1 has joined #openstack-infra | 20:05 | |
| *** ccrouch has quit IRC | 20:06 | |
| *** Sukhdev has quit IRC | 20:07 | |
| mtreinish | pabelanger: on: https://review.openstack.org/#/c/287373/ you realize that there isn't anything being deployed right? it's static content that needs to be built periodically and put on the slaves | 20:08 |
| *** shardy has joined #openstack-infra | 20:10 | |
| pabelanger | mtreinish: like I said, my main concern is this would be the first time we are using dib elements to control this. Where we usually do it with puppet. Looking at the existing patchset, you are downloading npm and compiling things, but to be puppet would be a more consistent place to do that | 20:10 |
| *** |-paul-| has joined #openstack-infra | 20:11 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: 10 day count is too high https://review.openstack.org/267230 | 20:11 |
| mtreinish | pabelanger: it's jsut because it's the first time we're adding js content to the output | 20:11 |
| *** sdake_ has quit IRC | 20:11 | |
| *** sdake has joined #openstack-infra | 20:12 | |
| mtreinish | pabelanger: how would you propose doing it in puppet? we just need the js to end up on the slave image, puppet isn't there | 20:13 |
| jbernard_ | mriedem: hey matt, ive been looking at improving the ceph ci job and i have a few thoughts | 20:13 |
| pabelanger | mtreinish: the other issue, we don't really have a way to test dib elements in the gate. So moving to puppet-stackviz, will allow us to deploy using puppet-apply jobs or beaker to ensure things work as expected | 20:13 |
| jbernard_ | mriedem: since we're not voting at the moment, i would really like to switch away from the in-devstack lib/ code, and to the ceph-plugin | 20:14 |
| mriedem | jbernard_: the ceph plugin job was failing before trusty did that release on 2/24 | 20:14 |
| *** yamahata has quit IRC | 20:14 | |
| mriedem | as in the ceph plugin job was already unreliable, much more so than the in-tree ceph job | 20:15 |
| cj | thanks, fungi | 20:15 |
| cj | fungi: luckily, it looks like I was working on an orphaned fork of the library. The official one is much more up-to-date. | 20:16 |
| jbernard_ | mriedem: hmm, id like to look into improving that - if a reliable plugin could be considered as a switch | 20:16 |
| mtreinish | pabelanger: testing what that we can do npm build for stackviz? That should be tested by stackviz | 20:16 |
| pabelanger | mtreinish: so https://review.openstack.org/#/c/279317/8/nodepool/elements/stackviz/install.d/99-prepare-stackviz would be moved into puppet. For example, look at puppet-askbot as an example. it clones a git repo, downloads dependencies and install this self. I would see puppet-stackviz working the same | 20:16 |
| jbernard_ | mriedem: is that possible? | 20:16 |
| pabelanger | mtreinish: right, testing it builds and gets installed | 20:16 |
| mriedem | jbernard_: i guess, it doesn't really make a difference to me | 20:16 |
| mriedem | jbernard_: i have no idea why the ceph plugin job was failing worse than the devstack job | 20:16 |
| pabelanger | mtreinish: otherwise, we have no good way to ensure it wont break dibs | 20:16 |
| jbernard_ | mriedem: that i need to look at | 20:16 |
| mriedem | jbernard_: since i thought the ceph plugin job was just a fork lift of the same code as in devstack | 20:16 |
| mtreinish | pabelanger: is your only objection that it's js instead of python? because the steps in that dib element aren't any different from what we do for python stuff | 20:17 |
| mtreinish | it's just using the js tooling to do it | 20:17 |
| *** amitgandhinz has joined #openstack-infra | 20:17 | |
| mriedem | jbernard_: use sdague's failopotamus (sp?) repo to chart the failure rates between those 2 jobs | 20:17 |
| jbernard_ | mriedem: the plugin fetches packages from ceph.com, which would have avoided the problem we hit last week | 20:17 |
| mriedem | oooo | 20:17 |
| mriedem | but apparently probably introduces oehters | 20:17 |
| mriedem | *others | 20:17 |
| mriedem | jbernard_: stable packages or bleeding edge? | 20:17 |
| mtreinish | pabelanger: right, that's a known quantity. So you're proposing we don't anything new to the dib elements because it's only tested once it lands? | 20:17 |
| pabelanger | mtreinish: no, my objection is we are using dib to clone, compile and install stackviz, when we usually use puppet to do this | 20:17 |
| jbernard_ | if that's the case, then ill try to address them | 20:17 |
| sdague | the plugin fail rate has definitely been higher than in tree, I don't know why | 20:18 |
| jbernard_ | mriedem: the plugin would substantially increase are ability to iterate | 20:18 |
| pabelanger | mtreinish: having this a dib element also makes it difficult for somebody wanting to isntall stackviz downstream | 20:18 |
| pabelanger | if they wanted too | 20:18 |
| *** max_lobur has quit IRC | 20:18 | |
| sdague | if we can get those fail rates to match, flipping over would be fine, we just can't go backwards on fail rates | 20:18 |
| mtreinish | pabelanger: so it shouldn't need to clone anything, it'll already be cloned by the git cache | 20:18 |
| mtreinish | it should just compile and install it | 20:19 |
| jbernard_ | sdague: understood, ill work towards that | 20:19 |
| *** rraja has quit IRC | 20:19 | |
| jbernard_ | sdague: you'll need access to the plugin to revert/rollback in case of a problem, correct? | 20:19 |
| *** eharney has quit IRC | 20:19 | |
| jbernard_ | sdague: would i need to change anythign project-wise to make is considerable? | 20:20 |
| sdague | jbernard_: yeh, chatting with dansmith earlier, I think we might want to treat the ceph plugin a bit special and get the qa team to adopt it | 20:20 |
| sdague | having the current core team on it, plus add devstack-core | 20:20 |
| mtreinish | pabelanger: and that feels like a perfect fit for doing in a dib element. Just like we pip install a bunch of stuff. | 20:20 |
| sdague | mtreinish: opinions? | 20:20 |
| *** pradk has quit IRC | 20:20 | |
| *** maishsk has quit IRC | 20:20 | |
| dansmith | sdague: the plugin fail rate is at base right? | 20:20 |
| dansmith | sdague: it's not failing for the current thing, AFAICT | 20:21 |
| sdague | then we could keep the subject matter experts working on issues, but also let the qa generalists help when we need that | 20:21 |
| sdague | dansmith: ?? | 20:21 |
| jbernard_ | sdague: that would be great | 20:21 |
| dansmith | sdague: the graph for it is super low | 20:21 |
| rcarrillocruz | mordred: http://paste.openstack.org/show/489232/ | 20:21 |
| sdague | dansmith: I don't know where things currently stand. I was asked to approve a cut over 2 weeks ago, and I -1ed it because of failure rate differences | 20:21 |
| rcarrillocruz | so works | 20:21 |
| jbernard_ | dansmith: turns out, we were already pulling ceph.com packages in the plugin | 20:21 |
| dansmith | sdague: http://graphite.openstack.org/render/?from=-200hours&height=500&until=now&width=800&bgcolor=ffffff&fgcolor=000000&yMax=100&yMin=0&vtitle=Failure%20Rate%20in%20Percent&title=Test%20failure%20rates%20over%20last%20200%20hours%20%2812%20hour%20rolling%20average%29&drawNullAsZero=true&&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full.FAILURE),transformNull | 20:21 |
| dansmith | (sum(stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-tempest-dsvm-full%20%28check%29%27),%27ff0000%27),1)&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.gate.job.gate-tempest-dsvm-full.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.gate.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-tempest-dsvm-ful | 20:21 |
| dansmith | l%20%28gate%29%27),%27b00000%27),2)&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full-ceph.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full-ceph.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-tempest-dsvm-full-ceph%20%28check%29%27),%270000ff%27),1)&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_co | 20:21 |
| jbernard_ | dansmith: but not in devstack proper | 20:22 |
| dansmith | unts.zuul.pipeline.gate.job.gate-tempest-dsvm-full-ceph.FAILURE),transformNull(sum(stats_counts.zuul.pipeline.gate.job.gate-tempest-dsvm-full-ceph.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-tempest-dsvm-full-ceph%20%28gate%29%27),%270000b0%27),2)&target=lineWidth(color(alias(movingAverage(asPercent(transformNull(stats_counts.zuul.pipeline.check.job.gate-tempest-dsvm-full-devstack-plugin-ceph-nv.FAILURE),transformNull(sum(stats_count | 20:22 |
| dansmith | s.zuul.pipeline.check.job.gate-tempest-dsvm-full-devstack-plugin-ceph-nv.{SUCCESS,FAILURE}))),%2712hours%27),%20%27gate-tempest-dsvm-full-devstack-plugin-ceph-nv%20%28check%29%27),%2700ff00%27),1) | 20:22 |
| dansmith | whoa | 20:22 |
| dansmith | did not mean that at all, sorry | 20:22 |
| mriedem | sdague: jbernard_: dansmith: i'll generate a graph to compare | 20:22 |
| rcarrillocruz | if you sort out the test failures i think we are good | 20:22 |
| dstufft | that's a hell of an URL | 20:22 |
| pabelanger | mtreinish: I disagree. the only things we pip install are zuul and some other JJB stuff. everything else dependency wise, we use puppet to manage | 20:22 |
| sdague | dansmith: tinyurl please | 20:22 |
| pabelanger | just look to devstack dependencies for example | 20:22 |
| dansmith | completely wrong paste :/ | 20:22 |
| dansmith | http://tinyurl.com/zzs89p3 | 20:22 |
| mtreinish | pabelanger: but if you really think doing a npm build in a dib element is evil an alternative I see is just having a post job build a tarball and just have the dib element pull that instead of doing | 20:22 |
| mtreinish | pabelanger: that's not the only stuff we pip install | 20:23 |
| mtreinish | there is ostestr stuff and we pip install subunit2sql too | 20:23 |
| sdague | ok, this is a very different graph than I had 2 weeks ago | 20:23 |
| sdague | so, yes, that looks better | 20:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fixes for latest cinder and neutron clients https://review.openstack.org/288110 | 20:23 |
| mtreinish | pabelanger: which is used to create a cache of old subunit streams | 20:23 |
| *** ybathia has quit IRC | 20:23 | |
| mriedem | http://tinyurl.com/j2hg34g | 20:23 |
| mriedem | ^ removes devstack full | 20:23 |
| mriedem | oh i guess you can't | 20:23 |
| sdague | mriedem: well devstack full is useful | 20:23 |
| mriedem | since that's the baseline | 20:23 |
| sdague | because we want it to be baselining similarly | 20:24 |
| sdague | which, it looks like it is | 20:24 |
| *** IlyaG_ has quit IRC | 20:24 | |
| mriedem | yeah | 20:24 |
| *** matrohon has joined #openstack-infra | 20:24 | |
| dansmith | it's 0.8% above base right now | 20:24 |
| pabelanger | mtreinish: it is not evil, we have never done it before. I would rather keep using puppet or even ansible to do it. Having a dib element to manage it, does feel like the right place | 20:24 |
| mriedem | so the plugin looks better | 20:24 |
| pabelanger | mriedem: sorry I have to run right now, but we can continue when I get back | 20:24 |
| sdague | mtreinish: so, how do you feel about ceph plugin being in QA program? | 20:24 |
| jbernard_ | the plugin has a few improvements, most noabely official ceph.com packages for trusty | 20:24 |
| mriedem | pabelanger: no problemo! | 20:24 |
| sdague | jbernard_: yeh, super cool that it does that. And great it's now succeeding well. | 20:25 |
| mtreinish | sdague: I guess we can adopt it, is it an orphan? | 20:25 |
| mriedem | mtreinish: goto pabelanger | 20:25 |
| sdague | mtreinish: well, here is the concern | 20:25 |
| sdague | ceph is a pretty mainline configuration, so lots of projects want it in their main test configs | 20:25 |
| jbernard_ | if not, i can certainly backport to devstack, but this seems like a perfect time to swtich over | 20:25 |
| sdague | however, without qa team with approve on it, if it breaks, then we can be hosed all over the place | 20:26 |
| sdague | so having both the subject matter experts in the core team for it, as well as devstack generalists, would give us the best of both worlds | 20:26 |
| dansmith | iff mtreinish doesn't want to adopt, we can still give the appropriate folks commit on that project right? | 20:26 |
| *** dmsimard has joined #openstack-infra | 20:27 | |
| sdague | dansmith: we cold | 20:27 |
| sdague | we could | 20:27 |
| dansmith | er -> if | 20:27 |
| sdague | but as devstack is under qa, and ceph is such a big part of our user base, it seems like a thing we should consider just as the right thing for the qa team to do | 20:27 |
| mtreinish | sdague: sure I understand the concern and the desire here. I'm not opposed to bringing it into qa, I'm just wondering why it's a plugin if we view it as being so important | 20:28 |
| dansmith | sure if it's up for it, absolitely | 20:28 |
| dansmith | mtreinish: being a plugin lets the SMEs work on it easier at least | 20:28 |
| timothyb89 | mtreinish: re: stackviz deployment, how safe is it to be running npm install in dib, given we don't have a local cache? | 20:28 |
| dansmith | I don't really care where we end up, I just want more reliable testing of ceph, however that goes | 20:28 |
| clarkb | timothyb89: its relatively safe our other dib builds dont use our mirrors | 20:29 |
| *** palendae has joined #openstack-infra | 20:29 | |
| mtreinish | dansmith: is that the only reason? | 20:29 |
| mtreinish | I think it makes total sense to bring the ceph devstack stuff under qa | 20:29 |
| clarkb | timothyb89: mirrors aee more important where we run things over and over 25 thousand times per day | 20:29 |
| *** kzaitsev_mb has joined #openstack-infra | 20:29 | |
| mtreinish | I'm just wondering if keeping it as a plugin makes sense given that | 20:29 |
| sdague | mtreinish: I think architecturally have ceph as a plugin is good for making sure we can do that right as well | 20:29 |
| timothyb89 | clarkb: good to know, will a failed element kill the whole build? | 20:30 |
| sdague | mtreinish: and, more importantly, right now, they have a good working plugin | 20:30 |
| dansmith | mtreinish: the plugin is the more modern of the two implementations right now.. we could move it into devstack, or just delete the other and keep rolling | 20:30 |
| *** claudiub|2 has joined #openstack-infra | 20:30 | |
| timothyb89 | clarkb: we'd probably prefer it to ignore the failure or retry, but I'm not sure how dib behaves | 20:31 |
| sdague | so, I'd rather run with what's working, and if we really want to integrated it all back in newton, do that | 20:31 |
| clarkb | timothyb89: yes but we will try again the next day | 20:31 |
| mtreinish | sdague: sure, that sounds like a good plan | 20:31 |
| clarkb | just keep in mind trusty's ceph packages are or were broken | 20:31 |
| sdague | ok, so, then I think these are the actions | 20:32 |
| mtreinish | because I think it's just a bit weird we're saying that ceph is important enough that we as qa want to ensure it's always working | 20:32 |
| *** e0ne has joined #openstack-infra | 20:32 | |
| mtreinish | but not enough to have it be part of devstack | 20:32 |
| dansmith | clarkb: that's why we're nuking the other one.. the plugin uses ceph.com official builds for trusty | 20:32 |
| sdague | 1) jbernard_ can you add devstack-core to the approvers for the devstack plugin | 20:32 |
| clarkb | oh we dont mirror thosr | 20:32 |
| timothyb89 | clarkb: okay, thanks for info | 20:32 |
| *** scottda has joined #openstack-infra | 20:32 | |
| jbernard_ | sdague: as soon as i figure out how to do that, yes | 20:32 |
| clarkb | but that may be fine depending on volumr and how good their webservers are | 20:32 |
| sdague | 2) mtreinish can you propose governance for that change | 20:33 |
| *** maishsk has joined #openstack-infra | 20:33 | |
| jbernard_ | clarkb: the plugin pulls trusty packages from ceph.com, which are officially supported | 20:33 |
| mtreinish | jbernard_: https://review.openstack.org/#/admin/groups/1196,members | 20:33 |
| sdague | clarkb: the current fail rates look like it's not a huge issue | 20:33 |
| mtreinish | jbernard_: add devstack-core to the included groups | 20:33 |
| *** ayoung has joined #openstack-infra | 20:33 | |
| jbernard_ | clarkb: it allows allows us to push package updates more quickly if there is a problem | 20:33 |
| sdague | clarkb: do we have a facility to mirror other package repos except main mirrors? | 20:34 |
| clarkb | jbernard_: yup thats fine but package repos jave long been a source of frustration so we avoid them | 20:34 |
| clarkb | sdague: I think we can point reprepro at another source | 20:35 |
| sdague | jbernard_: the big concern is we've found that package repos outside our networks get weird outages | 20:35 |
| clarkb | but I am not 100% sure how reprepro treats third party package locations | 20:35 |
| fungi | clarkb: hey! e-mail from osic says tomorrow--might be time to mention fixing the keystone catalogue too? | 20:35 |
| jbernard_ | clarkb: as long as we get officially supported pacakegs from somewhere | 20:35 |
| fungi | or maybe you already did... | 20:35 |
| sdague | 3) jbernard_ can you work with clarkb / infra on getting ceph packages into our mirrors from ceph.com ? | 20:35 |
| clarkb | fungi: I think they mentioned it, its required for them to change the name | 20:35 |
| fungi | perfect | 20:35 |
| jbernard_ | sdague: can do | 20:35 |
| sdague | 4) switch over to voting on the the plugin job | 20:35 |
| sdague | and 4 can happend before 3 | 20:36 |
| sdague | but we should try to close on 3 over the next couple of weeks | 20:36 |
| jbernard_ | clarkb: what would you need from me beyond our repo url? | 20:36 |
| sdague | this all seem like a reasonable plan? mtreinish / dansmith / jbernard_ | 20:36 |
| clarkb | jbernard_: thats probably the big thing, then someone has to write a puppet change to add it to our reprepro config | 20:37 |
| sdague | I guess 5 - we should put this job on devstack as well | 20:37 |
| sdague | I don't remember if it is or not | 20:37 |
| fungi | yeah, including it in our mirrors is probably not hard. caching the packages on our images gets into much trickier trust issues though | 20:37 |
| * mtreinish waits for 5.5 | 20:37 | |
| fungi | since apt sort of just munges all sources together and highest matching version for the same package name wins | 20:38 |
| dansmith | sdague: yes, definitely | 20:38 |
| mtreinish | sdague: yeah, I'm fine with that as the short term plan | 20:38 |
| mtreinish | well except for the part where I have to do something :) | 20:39 |
| fungi | so we have to be able to cache sources which we don't make available in default configuration, or somehow assure ourselves that other repos will never be at risk of overriding packages from the main distro except the ones we want them to | 20:39 |
| dansmith | sdague: the ceph dev people are also promising to look over the job in a couple weeks to figure out if we can do anything more efficiently like run with the in-memory driver or something like that | 20:39 |
| jbernard_ | clarkb: right now, we're pulling packages from http://ceph.com/debian-hammer | 20:39 |
| sdague | dansmith: great | 20:39 |
| jbernard_ | clarkb: once we have the packaegs mirrored, ill need to update the plugin | 20:39 |
| jbernard_ | clarkb: so we fetch from the mirror over ceph.com | 20:39 |
| sdague | mtreinish: we might also want to have a dedicated QA session in Austin to sort out any further details here, including a long term plan | 20:39 |
| jbernard_ | clarkb: (for trusty) | 20:40 |
| clarkb | jbernard_: thats actually something we may need to sort out since we have per region mirrors | 20:40 |
| sdague | given that ceph deploy is such a huge part of our user base as a community | 20:40 |
| mtreinish | sdague: right, that's what I was thinking | 20:40 |
| mtreinish | I guess now's a good a time as any to start a summit brainstorming etherpad | 20:40 |
| clarkb | jbernard_: when we launch jobs in a region we update its apt/pypi/etc config | 20:41 |
| clarkb | jbernard_: maybe we add apt/sources.d/ceph and you can just check if that is present or not | 20:42 |
| jbernard_ | sdague, mtreinish: devstack-core added to approvers | 20:42 |
| jbernard_ | clarkb: sure, that'll work | 20:42 |
| clarkb | if its present do nothing, if it isnt add your thing there | 20:42 |
| jbernard_ | clarkb: easy change | 20:42 |
| fungi | yeah, that should provide a clean transition | 20:42 |
| *** pradk has joined #openstack-infra | 20:43 | |
| dims | fungi : anteaya : poof it's gone again | 20:43 |
| dims | so it's cycling up and down | 20:43 |
| fungi | dims: yep, it's not running again | 20:44 |
| anteaya | dims: poof :( | 20:44 |
| dims | :) | 20:44 |
| anteaya | bringing up a vm to see if I can replicate an instance | 20:44 |
| mtreinish | sdague: https://etherpad.openstack.org/p/newton-qa-summit-topics | 20:44 |
| dims | anteaya : nothing in the logs? | 20:44 |
| anteaya | I haven't done this a lot so I would give me a lot of time to do this | 20:44 |
| *** jsavak has quit IRC | 20:44 | |
| anteaya | dims: I don't have access to the production server | 20:44 |
| fungi | puppet finished running on the server at 20:38 utc | 20:45 |
| fungi | i wonder if it's stopping it for some reason | 20:45 |
| anteaya | and fungi has to do the summit passes | 20:45 |
| dims | anteaya : fungi : i can't live without it :) it's indispensible :) | 20:45 |
| anteaya | dims: it is that | 20:45 |
| *** mrmartin has quit IRC | 20:45 | |
| mtreinish | jbernard_: cool thanks | 20:45 |
| *** jsavak has joined #openstack-infra | 20:45 | |
| *** sdague has quit IRC | 20:45 | |
| *** ZZelle_ has joined #openstack-infra | 20:46 | |
| palendae | Can someone help me understand why the docs job on https://review.openstack.org/#/c/287438/ doesn't appear to be including a table generated by a script? using setup.py build_sphinx produces it locally, but not in jenkins jobs | 20:46 |
| *** alex_xu has quit IRC | 20:46 | |
| jbernard_ | clarkb: once you decide the exact file path and you're sure, let me know and ill submit the ceph-plugin patch to match | 20:46 |
| clarkb | jbernard_: ok | 20:46 |
| anteaya | dims: and I successfully launched an instance I can't ping | 20:46 |
| *** piet has quit IRC | 20:47 | |
| *** ihrachys has joined #openstack-infra | 20:47 | |
| fungi | anteaya: dims: so it looks like puppet triggers a refresh (restart) of the hound service every time it runs, as i suspected | 20:48 |
| fungi | Mar 3 20:38:14 codesearch puppet-user[11161]: (/Stage[main]/Jeepyb/Package[libxslt-dev]/ensure) ensure changed 'purged' to 'present' | 20:48 |
| dims | fungi : ouch | 20:48 |
| fungi | Mar 3 20:38:17 codesearch puppet-user[11161]: (/Stage[main]/Openstack_project::Codesearch/Exec[create-hound-config]/returns) executed successfully | 20:48 |
| fungi | Mar 3 20:38:18 codesearch puppet-user[11161]: (/Stage[main]/Hound/Service[hound]) Triggered 'refresh' from 1 events | 20:48 |
| *** ddieterl_ has joined #openstack-infra | 20:48 | |
| anteaya | wow | 20:48 |
| fungi | i have a feeling the restart when it's running causes it to stop but not start, and a restart when it's not running causes it to start, leaving it down ~50% of the time | 20:49 |
| fungi | all theory for the moment. anyway, going back to working on other things | 20:49 |
| *** alex_xu has joined #openstack-infra | 20:49 | |
| *** doug-fis_ has joined #openstack-infra | 20:49 | |
| *** doug-fish has quit IRC | 20:50 | |
| *** tqtran has joined #openstack-infra | 20:50 | |
| *** hashar has quit IRC | 20:50 | |
| *** ihrachys_ has quit IRC | 20:50 | |
| fungi | i've started it again, but if my theory's right, it'll be offline again in another 10-15 minutes | 20:50 |
| anteaya | again wow | 20:50 |
| *** ddieterly has quit IRC | 20:51 | |
| *** nwkarsten has quit IRC | 20:52 | |
| *** nwkarsten has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged openstack/os-testr: Treat xfail output like success https://review.openstack.org/283266 | 20:53 |
| *** rossella_s has quit IRC | 20:54 | |
| *** nwkarste_ has joined #openstack-infra | 20:54 | |
| *** nwkarste_ has quit IRC | 20:54 | |
| *** rossella_s has joined #openstack-infra | 20:54 | |
| *** e0ne has quit IRC | 20:54 | |
| *** esker has joined #openstack-infra | 20:54 | |
| *** salv-orlando has quit IRC | 20:55 | |
| *** nwkarste_ has joined #openstack-infra | 20:55 | |
| *** salv-orlando has joined #openstack-infra | 20:55 | |
| *** gyee has joined #openstack-infra | 20:55 | |
| dims | fungi : anteaya : here's another one - https://www.evernote.com/l/AZzVXJ3XUR9Jka3McpJ1ThFj7UGGx8SsyZs | 20:56 |
| dims | "too many open files" | 20:56 |
| anteaya | dims: I've never seen that error before | 20:57 |
| *** nwkarsten has quit IRC | 20:57 | |
| dims | anteaya : try right now when it's in a tizzy | 20:57 |
| *** ldnunes has quit IRC | 20:57 | |
| dims | oops too late server broke down now | 20:57 |
| *** asettle has joined #openstack-infra | 20:57 | |
| fungi | dims: that may indicate a need to make ulimit adjustments for the process (which looks like it's running as root, so that's a fun one) | 20:57 |
| dims | guess, that's the sequence i see, up and running, then too many files, then brakes down | 20:58 |
| fungi | dims: oh, yep, in the last few seconds houndd disappeared from the process table again | 20:58 |
| dims | breaks :) | 20:58 |
| fungi | puppet had already finished running at 20:52:51 utc, so i think it may not be puppet-triggered | 20:58 |
| dims | thanks fungi anteaya :) you have enough information to go by now | 20:58 |
| bswartz | ulimit -n | 20:59 |
| fungi | though puppet did refresh the service again, which suggests it is at least restarting it every ~15 minutes which is also not good, but the too many open files may actually be crashing the process | 20:59 |
| *** Sukhdev has joined #openstack-infra | 20:59 | |
| *** lykinsbd has quit IRC | 20:59 | |
| *** e0ne has joined #openstack-infra | 21:00 | |
| *** stevebak` is now known as stevebaker | 21:00 | |
| fungi | unfortunately hound's log is almost useless (empty except some startup messages with no associated timestamp) so hard to know for certain | 21:00 |
| anteaya | someone said hound truncates its log on every restart | 21:01 |
| anteaya | I do believe that was an observation last round of why isn't hound running | 21:01 |
| *** Sukhdev has quit IRC | 21:01 | |
| fungi | 283810 lines in /var/log/hound.log so i don't think it's truncating on restart | 21:01 |
| *** rfolco has quit IRC | 21:01 | |
| *** maishsk has quit IRC | 21:02 | |
| openstackgerrit | Tim Hinrichs proposed openstack/requirements: Raise upper-constraint on python-congressclient https://review.openstack.org/288124 | 21:02 |
| fungi | if it is truncating on restart, that's a bunch of lines for a process which only ran a couple of minutes | 21:03 |
| *** sarob has quit IRC | 21:03 | |
| fungi | but without timestamps on any of the lines i can't really tell | 21:03 |
| *** sarob has joined #openstack-infra | 21:03 | |
| fungi | oh, wow no this is a traceback | 21:03 |
| fungi | there are actually timestamps if i scroll back far enough | 21:04 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Dont include plugin definition in cmd checks https://review.openstack.org/288061 | 21:04 |
| fungi | i'm just not used to looking at tracebacks from go | 21:04 |
| *** zeih has joined #openstack-infra | 21:04 | |
| fungi | but yes, looks like it spewed a lot of "fork/exec /usr/bin/git: too many open files" errors, then fell over leaving a tens-of-thousands-of-lines traceback when it did | 21:05 |
| dims | aha | 21:05 |
| mtreinish | sdague, dansmith, jbernard_: https://review.openstack.org/#/c/288126/ | 21:05 |
| jbernard_ | mtreinish: radical | 21:06 |
| *** baoli has joined #openstack-infra | 21:06 | |
| *** esikachev has joined #openstack-infra | 21:06 | |
| fungi | goroutine 6159 [running]: runtime.panic(0x70e180, 0xb7230a) /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6 | 21:06 |
| fungi | whee! | 21:06 |
| openstackgerrit | Ben Swartzlander proposed openstack/requirements: Raise manilaclient upper to 1.8.0 https://review.openstack.org/288129 | 21:07 |
| *** amrith is now known as _amrith_ | 21:07 | |
| *** jpr has joined #openstack-infra | 21:07 | |
| fungi | so as best i can piece together from the hound log, it is restart related. it gets restarted on every puppet run, starts reindexing, and when it gets to the point where it tries to index openstack/nova it reaches an open file limit on the git subprocess | 21:08 |
| *** asettle has quit IRC | 21:08 | |
| anteaya | ha ha ha | 21:08 |
| *** baoli_ has joined #openstack-infra | 21:08 | |
| fungi | this explains the several minutes of delay from when puppet restarts the process until it dies again | 21:08 |
| mtreinish | fungi: do you think it makes sense to resize up subunit-worker01.o.o? It's kinda close to max ram: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2385&rra_id=all | 21:08 |
| fungi | mtreinish: yeah, even -/+ buffers/cache: it's hovering at 75% used at the moment | 21:09 |
| dims | nova is too big :) | 21:10 |
| anteaya | way to big | 21:10 |
| fungi | mtreinish: subunit-gearman VIRT: 6822m RES: 5.8g SHR: 3540 according to top | 21:10 |
| *** jamesmcarthur has quit IRC | 21:10 | |
| fungi | mtreinish: unless you have reason to believe there's a leak in subunit-gearman, i agree a larger instance may be in order | 21:11 |
| *** moravec has quit IRC | 21:11 | |
| mtreinish | fungi: yeah the way the worker is architected right now is very ram hungry (it stores all the streams in needs to process as an StringIO object in a queue) | 21:11 |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| mtreinish | fungi: I don't think it's a leak, just a poor design | 21:11 |
| *** baoli has quit IRC | 21:11 | |
| fungi | mtreinish: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2385&rra_id=all doesn't look especially leaky, though it does seem to top out | 21:12 |
| mtreinish | it can be rearchitected to be more efficient (and probably write streams to disk instead of keeping it all in memory) but it's a matter of finding the time to do it | 21:12 |
| fungi | mtreinish: and it's not dipping into swap at all | 21:12 |
| mtreinish | fungi: that's purely coincidence I think | 21:13 |
| fungi | i'll buy that | 21:13 |
| mtreinish | we've oomkillered twice in the past couple of weeks | 21:13 |
| *** ashleighfarnham has quit IRC | 21:13 | |
| fungi | so yeah, bigger instance i guess, though i don't have time to drive that right now so need other volunteers | 21:13 |
| mtreinish | also there are no tests for the worker script, so I'd get a little nervous rewriting it (since it mostly works today) | 21:14 |
| mtreinish | infra-root: ^^^ anyone else got a spare couple of cycles to help with the resize | 21:14 |
| fungi | shake the can harder | 21:15 |
| anteaya | mtreinish: pleia2 isn't feeling well today but perhaps she might be able to help tomorrow | 21:15 |
| anteaya | I know she has been trying to help you with this | 21:15 |
| mtreinish | fungi: heh | 21:15 |
| *** zeih has quit IRC | 21:16 | |
| mtreinish | anteaya: sure, yeah pleia2 has been helping a lot with the other recent subunit worker issue | 21:16 |
| anteaya | she is less than 100% today | 21:16 |
| anteaya | but maybe tomorrow | 21:16 |
| *** esker has quit IRC | 21:17 | |
| *** e0ne has quit IRC | 21:17 | |
| *** jpr has quit IRC | 21:17 | |
| *** [HeOS] has quit IRC | 21:17 | |
| *** e0ne has joined #openstack-infra | 21:17 | |
| *** kgiusti has left #openstack-infra | 21:18 | |
| *** bhunter71 has quit IRC | 21:19 | |
| *** angdraug has joined #openstack-infra | 21:21 | |
| nibalizer | resize? or just new node? | 21:22 |
| nibalizer | i didn't know rax supported resize | 21:22 |
| anteaya | well to the extent nova supports resize I believe | 21:23 |
| anteaya | and I think the extent to which nova supports resize is kind of a chat over beer kind of topic | 21:23 |
| anteaya | fungi tried resize once upon a time | 21:24 |
| *** abregman has quit IRC | 21:24 | |
| anteaya | I think he gave up after waiting for over 24 hours for it to finish | 21:24 |
| fungi | nibalizer: there is no resize in rackspace, except for trove instances | 21:24 |
| jeblair | there used to be | 21:24 |
| fungi | yep, there did | 21:24 |
| *** ayoung has quit IRC | 21:24 | |
| *** ihrachys_ has joined #openstack-infra | 21:24 | |
| fungi | they dropped it when they migrated to "new" nova-backed instances | 21:25 |
| fungi | you could resize legacy instances | 21:25 |
| anteaya | did old resize work? | 21:25 |
| jeblair | we used it often and it generally worked well aside from the fact that runtime was unpredictable (and seemingly exponentially related to instance size) | 21:25 |
| anteaya | wonderful, thanks | 21:25 |
| fungi | i don't know why they stopped supporting resize when they switched to real openstack, but they did | 21:25 |
| *** ihrachys has quit IRC | 21:25 | |
| fungi | i recall the point at which we had about half our instances in the legacy environment, all the legacy instances had a "resize" option but none of the ones in the new environment had it | 21:26 |
| nibalizer | so we're talking about booting a new node? | 21:27 |
| nibalizer | i can likely do this | 21:27 |
| jeblair | fungi: the webui says it's not available for servers "made from this flavor" | 21:27 |
| jeblair | fungi: so maybe it's a pvhvm thing... | 21:27 |
| nibalizer | in spare cycles found on the floor today | 21:27 |
| nibalizer | mtreinish: specs you need? | 21:27 |
| anteaya | nibalizer: yes, I think booting a new node is the best way forward | 21:27 |
| fungi | jeblair: seems likely, as we switched to pvhvm around the same timeframe | 21:27 |
| *** e0ne has quit IRC | 21:28 | |
| fungi | and there was no pvhvm in the legacy environment, so that would be a pretty reasonable explanation | 21:28 |
| *** ihrachys has joined #openstack-infra | 21:28 | |
| jeblair | the pvhvm is worth it, i think | 21:28 |
| nibalizer | mtreinish: this is the flavor list http://paste.openstack.org/show/489245/ | 21:28 |
| nibalizer | we generally get the performance nodes | 21:29 |
| *** ihrachys_ has quit IRC | 21:29 | |
| nibalizer | we have 8gb of ram on it right now | 21:29 |
| *** abregman has joined #openstack-infra | 21:30 | |
| *** tqtran has quit IRC | 21:31 | |
| *** dkranz has quit IRC | 21:32 | |
| *** esikachev has quit IRC | 21:33 | |
| *** ddieterl_ is now known as ddieterly | 21:35 | |
| *** annegentle has quit IRC | 21:35 | |
| *** annegentle has joined #openstack-infra | 21:35 | |
| *** vgridnev has quit IRC | 21:36 | |
| bswartz | how do I debug a job that runs in the "post" pipeline? where are the logs? | 21:36 |
| *** max_lobur has joined #openstack-infra | 21:36 | |
| *** e0ne has joined #openstack-infra | 21:37 | |
| fungi | bswartz: you need to know the hash of the commit which triggered in post (often the merge commit rather than the has of the commit which got merged) and then look at http://logs.openstack.org/XX/YYYY where XX is the first two characters of the hash (lower cae) and YYYY is the full hash (lower-case) | 21:38 |
| fungi | some day there will be a better dashboard indexing these | 21:38 |
| bswartz | the commit hash not the gerrit change num? | 21:39 |
| *** esikachev has joined #openstack-infra | 21:39 | |
| kevinbenton | any thoughts on using letsencrypt to get free valid certs for the jenkins nodes? | 21:39 |
| *** aeng has joined #openstack-infra | 21:39 | |
| kevinbenton | (it's fully automatable) | 21:39 |
| *** ybathia has joined #openstack-infra | 21:40 | |
| bswartz | okay that worked | 21:41 |
| *** abregman is now known as abregman|nb | 21:41 | |
| mtreinish | nibalizer: performance 15g then? | 21:42 |
| mtreinish | the other thing is the workers are horizontally scallable we could just spin up another 8 in parallel and it should work | 21:42 |
| *** kingia has joined #openstack-infra | 21:42 | |
| *** arxcruz has joined #openstack-infra | 21:43 | |
| *** ihrachys_ has joined #openstack-infra | 21:43 | |
| *** max_lobur1 has joined #openstack-infra | 21:44 | |
| nibalizer | whatever you want to do | 21:44 |
| nibalizer | id rather go parallel if we can | 21:44 |
| *** jpr has joined #openstack-infra | 21:44 | |
| nibalizer | clarkb: does our current launch_node work or are there tricks? | 21:45 |
| fungi | kevinbenton: why? | 21:45 |
| *** rcernin has quit IRC | 21:45 | |
| fungi | kevinbenton: are you storing or passing sensitive information to/from them? | 21:45 |
| *** max_lobur has quit IRC | 21:45 | |
| *** ihrachys has quit IRC | 21:45 | |
| fungi | kevinbenton: self-signed certs are also fully automatable | 21:45 |
| fungi | also we hope to be not running jenkins pretty soon | 21:47 |
| kevinbenton | fungi: to prevent self-signed cert errors every time i want to watch the jenkins progress on a job | 21:47 |
| fungi | kevinbenton: i use a pretty terrible web browser, and even it will let me permanently accept the self-signed cert for that server without risk that the cert could somehow be used to compromise other activities on the web | 21:48 |
| fungi | though if you're concerned that someone might be presenting you with fake job results masquerading as our ci system... i guess that's a risk | 21:48 |
| *** ashleighfarnham has joined #openstack-infra | 21:49 | |
| *** |-paul-| has quit IRC | 21:49 | |
| mtreinish | nibalizer: ok, parallel it is then. We've never tested that before :) | 21:49 |
| kevinbenton | fungi: chrome doesn't make it easy to do so, and it would have to be done for every jenkins node | 21:49 |
| kevinbenton | fungi: it's not that i'm concerned about an impersonation | 21:49 |
| *** jamesmcarthur has quit IRC | 21:49 | |
| fungi | kevinbenton: put another way, if i could turn off https for the webui on those servers and make them use http, i would | 21:49 |
| nibalizer | mtreinish: heeh | 21:50 |
| *** jtomasek_ has joined #openstack-infra | 21:50 | |
| kevinbenton | fungi: just an annoyance to deal with on every visit to jenkins | 21:50 |
| kevinbenton | fungi: not a huge deal | 21:50 |
| fungi | i still find it galling that modern web broswers have decided that encrypting with a self-signed cert and tofu model is somehow _less_ secure than not encrypting communication at all | 21:50 |
| mtreinish | nibalizer: I'll push up the system config patch to add the node | 21:50 |
| nibalizer | cool | 21:51 |
| *** max_lobur has joined #openstack-infra | 21:51 | |
| *** max_lobur1 has quit IRC | 21:51 | |
| kevinbenton | fungi: yes, but as long as they show an error for it. it's presented as a dangerous situation | 21:51 |
| mtreinish | nibalizer: oh, nm it's done as a regex so I don't need to do anything | 21:51 |
| nibalizer | ok | 21:52 |
| mtreinish | nibalizer: https://github.com/openstack-infra/system-config/blob/master/manifests/site.pp#L450-L461 | 21:52 |
| nibalizer | what hostname do we want | 21:52 |
| *** sc68cal_ is now known as sc68cal | 21:52 | |
| nibalizer | subunit-worker02.o.o ? | 21:52 |
| fungi | kevinbenton: yep. best solution to this problem is to help us get rid of jenkins http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 21:52 |
| mtreinish | nibalizer: yeah | 21:53 |
| fungi | i'd rather not spend time "throwing good money after bad" | 21:53 |
| nibalizer | mtreinish: okay it is chuggin | 21:54 |
| kevinbenton | fungi: (side note, it would be really difficult for browsers to adopt a mode of HTTPS that is still considered insecure for all intents an purposes. HSTS and a myriad of other things would have to special-case unvalidated HTTPS connections) | 21:54 |
| kevinbenton | fungi: ack | 21:54 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Add gate-tempest-dsvm-full-ceph to experimental queue for devstack https://review.openstack.org/288148 | 21:54 |
| *** thorst_ has quit IRC | 21:54 | |
| nibalizer | mordred: im reealizing there is an opportunity while launch-node is running for ansible to see the host and cause it to do things | 21:54 |
| mtreinish | nibalizer: ok cool, I think you'll have to manual start the worker daemon on subunit-worker02.o.o once it's all up and running | 21:54 |
| mriedem | mtreinish: https://review.openstack.org/#/c/288148/ | 21:54 |
| mriedem | jamespage: https://review.openstack.org/#/c/288148/ | 21:55 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Fix manila-image-elements publish job https://review.openstack.org/288149 | 21:55 |
| *** fawadkhaliq has joined #openstack-infra | 21:55 | |
| *** yaume_ has joined #openstack-infra | 21:55 | |
| fungi | kevinbenton: i consider https pretty insecure as it is (have you looked at some of the certificate authorities allowed to issue certs for any domain they want?), but could care less if browsers stopped showing pretty padlock icons for sites with manually accepted certificates | 21:56 |
| *** dingyichen has joined #openstack-infra | 21:57 | |
| kevinbenton | fungi: right, it's not the padlock icon. it's that the browser has accepted it as a valid HTTPS connection. There is a bunch of crap built on top of the assumption that the site has been validated or the user has manually allowed it once the site is HTTPS-secured | 21:57 |
| *** alivigni has quit IRC | 21:58 | |
| anteaya | stuff | 21:58 |
| cody-somerville | jeblair, fungi, mordred, other people interested in Gertty: I implemented breadcrumb footer bar for Gertty to help folks keep track of where they are while using the stack based navigation. Here is a screenshot: http://imagebin.ca/v/2Z1FdV0JZvzP -- I'm going to tidy up the code a little bit before submitting patch but wanted to see if folks thought that looked good. | 21:58 |
| fungi | kevinbenton: sounds like web browsers would be better off adopting a third category of "connections considered secure by the global certificate authority cabal | 21:58 |
| *** yaume has quit IRC | 21:58 | |
| jeblair | cody-somerville: that looks pretty cool :) | 21:59 |
| jhesketh | Morning | 21:59 |
| fungi | cody-somerville: oh, neat! | 21:59 |
| anteaya | jhesketh: morning | 21:59 |
| anteaya | cody-somerville: that looks useful | 21:59 |
| fungi | kevinbenton: though take my opinions with a grain of salt. i'm pretty antagonistic to the world wide web and shake my old man cane a lot at those hip youngster protocols | 22:00 |
| *** annegentle has quit IRC | 22:00 | |
| kevinbenton | fungi: you can change your own cert store :) | 22:00 |
| fungi | kevinbenton: and indeed i do | 22:00 |
| *** annegentle has joined #openstack-infra | 22:00 | |
| *** thorst_ has joined #openstack-infra | 22:01 | |
| fungi | i also sometimes change it by accepting specific self-signed certs for specific sites in fact | 22:01 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add subunit-worker02.o.o to the heira data https://review.openstack.org/288152 | 22:01 |
| kevinbenton | fungi: then that category of "connections considered secure by the cabal" no longer makes sense | 22:01 |
| jeblair | fungi: i think tcp shows promise. it might catch on. | 22:01 |
| mtreinish | nibalizer: ^^^ | 22:01 |
| fungi | jeblair: udp 4evar | 22:01 |
| fungi | cody-somerville: wow, neat! | 22:02 |
| *** sridhar_ram has quit IRC | 22:02 | |
| jeblair | (^ the bumper sticker on the back of afs's vw) | 22:02 |
| fungi | i especially like the use of unicode triangle codepoints | 22:02 |
| *** asettle has joined #openstack-infra | 22:02 | |
| *** ihrachys has joined #openstack-infra | 22:02 | |
| *** asettle has quit IRC | 22:03 | |
| *** alivigni has joined #openstack-infra | 22:03 | |
| *** asettle has joined #openstack-infra | 22:03 | |
| cody-somerville | :) | 22:03 |
| jeblair | is that a unicode clock? | 22:03 |
| openstackgerrit | Ben Swartzlander proposed openstack-infra/project-config: Fix manila-image-elements publish job https://review.openstack.org/288149 | 22:03 |
| cody-somerville | Aye. That is a unicode clock. | 22:04 |
| * jeblair swoons | 22:04 | |
| bswartz | I hate to beg for reviews, but this change is sort of high priority for manila because we're trying to wrap up our install guide and it needs to reference images published by the job in this patch: https://review.openstack.org/288149 | 22:04 |
| fungi | kevinbenton: true. just taking a dig at the ca/bforum, who factor into many of my conspiracy theories | 22:04 |
| mtreinish | ugh, I guess I really need to fix unicode on my laptop now... | 22:04 |
| jeblair | cody-somerville: hah, you are too graphical for mtreinish! | 22:05 |
| *** ihrachys_ has quit IRC | 22:05 | |
| *** thorst_ has quit IRC | 22:05 | |
| *** sridhar_ram has joined #openstack-infra | 22:05 | |
| fungi | mtreinish is even more of a luddite than i am, it seems | 22:05 |
| *** sarob has quit IRC | 22:05 | |
| *** amotoki has joined #openstack-infra | 22:05 | |
| fungi | mtreinish: are you nostalgic for iso-8859-1, or msdos cp-437? | 22:06 |
| *** thorst_ has joined #openstack-infra | 22:06 | |
| cody-somerville | lol | 22:06 |
| mtreinish | fungi: I have unicode working fine on all my other machines, but for whatever reason I've never gotten it to work on my laptop at all | 22:06 |
| *** kzaitsev_mb has quit IRC | 22:07 | |
| *** ihrachys_ has joined #openstack-infra | 22:07 | |
| jeblair | i use a cp437->unicode translator for making ansi art for presentty | 22:07 |
| anteaya | bswartz: I'm confused, where are you publishing qcow images to and why are you doing so? | 22:07 |
| mtreinish | I want to blame lenovo or archlinux, but it works fine on my old arch thinkpad | 22:07 |
| fungi | gordc: do you have a pending governance change to correct ceilometer-specs to telemetry-specs in governance now that it's renamed? | 22:07 |
| anteaya | am I being old fashioned believing that the artifacts we offered were tarballs? | 22:08 |
| mtreinish | I'm probably just missing 1 super obvious config somewhere, but it's eluded me for almost 2 years :) | 22:08 |
| *** asettle has quit IRC | 22:09 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:09 | |
| gordc | fungi: hm.. i believe one exists. let me verify | 22:09 |
| gordc | fungi: https://review.openstack.org/#/c/285167/ | 22:09 |
| gordc | not under my name :) | 22:09 |
| fungi | gordc: just making sure. my summit invite script alerts me to repos listed in governance which don't exist | 22:09 |
| fungi | thanks | 22:09 |
| gordc | ah i see. | 22:09 |
| *** shardy has quit IRC | 22:10 | |
| *** ihrachys has quit IRC | 22:10 | |
| *** esikachev has quit IRC | 22:10 | |
| fungi | looks like it missed the window for the last tc meeting. that explains it | 22:10 |
| fungi | will probably be corrected next week | 22:10 |
| *** thorst_ has quit IRC | 22:11 | |
| *** jtomasek_ has quit IRC | 22:11 | |
| nibalizer | hrm evidently launch_node.py requires roo | 22:11 |
| nibalizer | root now | 22:11 |
| nibalizer | OSError: [Errno 13] Permission denied: '/var/cache/ansible-inventory/ansible-inventory.cache' | 22:11 |
| *** esikachev has joined #openstack-infra | 22:12 | |
| fungi | mordred: the patch to import ansible-build-image into gerrit still hasn't merged, huh? | 22:12 |
| *** nwkarste_ has quit IRC | 22:12 | |
| fungi | is that still planned to get used? | 22:12 |
| *** zeih has joined #openstack-infra | 22:12 | |
| fungi | we're coming up on a year of it being listed in governance | 22:13 |
| bswartz | anteaya: to the tarballs site, and because there's not a better alternative to my knowledge -- the ironic team uses the same approach to upload their images | 22:13 |
| *** ociuhandu has joined #openstack-infra | 22:14 | |
| fungi | anteaya: yeah, at least ironic-python-agent does this, but i want to say the trove agent has something similar as well | 22:14 |
| bswartz | anteaya: the docs team wanted to make sure that any images referred to in the manila install guide came from an "official" openstack place and not a 3rd party site, which I completely agree with | 22:15 |
| fungi | EmilienM: is there a pending patch to create/import puppet-octavia into gerrit? | 22:15 |
| fungi | EmilienM: looks like it's been listed in governance for a while now | 22:15 |
| EmilienM | fungi: no I was about doing that soon | 22:15 |
| *** sarob has joined #openstack-infra | 22:16 | |
| fungi | EmilienM: okay, cool. just making sure it was still planned | 22:16 |
| *** esikachev has quit IRC | 22:16 | |
| *** jamesmcarthur has joined #openstack-infra | 22:16 | |
| EmilienM | fungi: it's my fault, I created the entry in governance first and then got back by urgent stuffs.. I'll initiate the module soon i think | 22:16 |
| *** zeih has quit IRC | 22:17 | |
| fungi | bswartz: is manila something that might make sense in the app catalog? i wonder if we should start looking to that for an "official" place that application-layer projects would distribute their released install artifacts | 22:17 |
| anteaya | bswartz fungi okay thanks, guess I am behind the times | 22:17 |
| anteaya | thanks for the explaination | 22:17 |
| fungi | not for now necessarily, but longer term | 22:17 |
| *** moravec has joined #openstack-infra | 22:17 | |
| anteaya | and I agree with the docs folks too | 22:17 |
| nibalizer | mtreinish: okay node is online | 22:18 |
| nibalizer | and ansible its pansibling | 22:18 |
| nibalizer | so it should hit it on this run | 22:18 |
| nibalizer | also dns is setup | 22:18 |
| fungi | bswartz: i believe bootable vm images are one of the sorts of artifacts they cater to | 22:19 |
| anteaya | yay! | 22:19 |
| *** alivigni has quit IRC | 22:19 | |
| *** jtomasek has quit IRC | 22:19 | |
| nibalizer | or more accurately 'dns is comming soon to a resolver near you!' | 22:19 |
| *** regXboi has quit IRC | 22:19 | |
| bswartz | fungi: I don't know anything about the app catalog | 22:19 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump upper constraints of openstackclient https://review.openstack.org/288157 | 22:19 |
| bswartz | where can I learn more | 22:19 |
| fungi | bswartz: oh! http://apps.openstack.org/ but also docaedo could probably talk your ear off about it | 22:19 |
| mtreinish | nibalizer: ok, very cool | 22:19 |
| bswartz | okay I have to run right now but I will check it out | 22:19 |
| fungi | cool deal. it's some awesome stuff | 22:20 |
| openstackgerrit | Merged openstack-infra/project-config: Add auto-discovery test job to ironic-inspector https://review.openstack.org/277843 | 22:20 |
| * docaedo reads scrollback and prepares to talk some ears off | 22:20 | |
| bswartz | in the short term I'd appreciate you considering my publish-to-tarballs fix because the install guide team has relatively soon deadlines | 22:20 |
| mtreinish | nibalizer: once it's all finished installing everything I'm pretty sure you'll have to start the subunit worker daemon | 22:20 |
| bswartz | docaedo: I'll be back in a couple or hours -- or you can talk my ear off tomorrow | 22:21 |
| mtreinish | nibalizer: also: https://review.openstack.org/#/c/288152/1 should be good to go, so we can get cacti | 22:21 |
| mtreinish | and I can obsess over the graphs | 22:21 |
| *** claudiub|2 has quit IRC | 22:21 | |
| docaedo | bswartz: sounds good, ping me later on #openstack-app-catalog and we'll catch up | 22:21 |
| nibalizer | mtreinish: cool | 22:21 |
| nibalizer | i havee +2 and rechecked that | 22:21 |
| nibalizer | hopefully they don't cancel each other out | 22:22 |
| anteaya | bswartz: +2 | 22:22 |
| *** baoli_ has quit IRC | 22:22 | |
| *** thorst_ has joined #openstack-infra | 22:23 | |
| mtreinish | nibalizer: oh, I just double checked the puppet, we can't start the new worker until that hiera patch lands | 22:24 |
| mtreinish | the elasticsearch_clients variable is used to set the firewall rules to allow gearman traffic | 22:24 |
| *** sarob has quit IRC | 22:24 | |
| nibalizer | yep | 22:24 |
| nibalizer | fungi: can you look at 288152 when you get a chance | 22:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add roles for CI users https://review.openstack.org/288160 | 22:25 |
| *** amitgandhinz has quit IRC | 22:25 | |
| jeblair | mtreinish, nibalizer: theoretically, you should be able to start the worker and it will just keep trying to connect until 288152 lands and then it will succeed | 22:25 |
| openstackgerrit | Akihiro Motoki proposed openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0 https://review.openstack.org/286587 | 22:25 |
| *** _amrith_ is now known as amrith | 22:27 | |
| jeblair | 2016-03-03 22:12:43.701 | Error: Could not get latest version: Timeout while contacting pypi.python.org: Timeout::Error | 22:27 |
| *** thorst_ has quit IRC | 22:27 | |
| jeblair | why are the beaker tests not using our mirrors? | 22:27 |
| *** edmondsw has quit IRC | 22:28 | |
| *** piet has joined #openstack-infra | 22:28 | |
| jeblair | let me ask a different question. who understands the beaker tests? | 22:28 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Set the bridge MTU dynamically https://review.openstack.org/287958 | 22:28 |
| anteaya | jeblair: I would start with EmilienM | 22:28 |
| *** aysyd has quit IRC | 22:29 | |
| *** bhunter71 has joined #openstack-infra | 22:29 | |
| anteaya | fungi: you want to approve https://review.openstack.org/#/c/288080/ | 22:29 |
| fungi | anteaya: you're quick! | 22:30 |
| fungi | thanks for watching for that | 22:30 |
| anteaya | fungi: welcome | 22:30 |
| *** kzaitsev_mb has quit IRC | 22:30 | |
| anteaya | figured you would want that in asap | 22:30 |
| jeblair | anteaya: EmilienM has never committed a change to puppet-openstackci | 22:30 |
| anteaya | jeblair: oh sorry | 22:30 |
| anteaya | he is who I would go to for beaker things personally | 22:30 |
| *** _sarob has joined #openstack-infra | 22:30 | |
| *** arxcruz has quit IRC | 22:30 | |
| EmilienM | I would ping nibalizer :-P | 22:30 |
| anteaya | at least to start | 22:30 |
| *** _sarob has quit IRC | 22:31 | |
| *** sarob has joined #openstack-infra | 22:31 | |
| *** sarob has quit IRC | 22:31 | |
| *** dimtruck is now known as zz_dimtruck | 22:31 | |
| *** sarob has joined #openstack-infra | 22:31 | |
| crinkle | jeblair: what needs to be set to tell a node to use a mirror? if it's an environment variable, beaker probably throws it out when it ssh's to localhost | 22:33 |
| jeblair | crinkle: nothing needs to be done | 22:34 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: IPv4 network isolation testing for Ceph https://review.openstack.org/288163 | 22:34 |
| jeblair | crinkle: so i'm wondering if something is undoing it | 22:34 |
| jeblair | perhaps our own pip module? | 22:34 |
| *** matrohon has quit IRC | 22:35 | |
| *** piet has quit IRC | 22:35 | |
| *** e0ne has quit IRC | 22:35 | |
| fungi | the mirror determination is based on evaluating the pip.conf we install on the servers, so presumably something in the job is replacing/removing it | 22:35 |
| *** gnuoy_ has joined #openstack-infra | 22:35 | |
| *** andymaier_ has joined #openstack-infra | 22:35 | |
| *** Sukhdev has joined #openstack-infra | 22:35 | |
| *** gnuoy_ has quit IRC | 22:35 | |
| *** gnuoy_ has joined #openstack-infra | 22:35 | |
| jeblair | however, pip.conf does not show up in the log at http://logs.openstack.org/52/288152/1/check/gate-openstackci-beaker-trusty-dsvm/8402689/console.html | 22:36 |
| *** ayoung has joined #openstack-infra | 22:36 | |
| *** dprince has quit IRC | 22:36 | |
| fungi | wow, final count for purposes of free summit admission. 29% of the people getting free passes to the summit for contributing have only one change merged during the qualifying period | 22:36 |
| *** thorst_ has joined #openstack-infra | 22:36 | |
| *** tpsilva has quit IRC | 22:37 | |
| EmilienM | that's weird | 22:37 |
| jeblair | fungi: one good change is worth 200 spelling corrections. | 22:37 |
| fungi | i'm all for convincing people to contribute and learn the our development toolchain, but wish they would stick around after they got their free admission | 22:38 |
| docaedo | wow that's a high percentage | 22:38 |
| *** andymaier has quit IRC | 22:38 | |
| bkero | Huh, I figured that to get free pass would need like 5 changes | 22:39 |
| fungi | bkero: nah, then the people who are just in it for the free pass would find 5 tiny things to patch instead of 1 | 22:39 |
| anteaya | fungi: woooo | 22:39 |
| fungi | and we'd still see basically the same pattern i think | 22:40 |
| bkero | fungi: What's that? 4 free patches to all projects? | 22:40 |
| jeblair | again, getting one substantial change in nova can be a signficant undertaking. and i'm certain nova would not like people to push 5 times as many changes as they are now. | 22:40 |
| bkero | That's true. Small changes like that can be a burden on cores. | 22:40 |
| fungi | yep | 22:40 |
| anteaya | 29% is an all time high is it not? | 22:40 |
| loquacities | it's why docs geta an uptick before every summit | 22:41 |
| *** camunoz has joined #openstack-infra | 22:41 | |
| anteaya | or is asking that question too much work for tonight? | 22:41 |
| loquacities | docs gets* | 22:41 |
| *** rhallisey has quit IRC | 22:41 | |
| anteaya | loquacities: yeah, poor docs | 22:41 |
| anteaya | :( | 22:41 |
| loquacities | 29% is awful high | 22:41 |
| jeblair | loquacities: drat, i was going to correct you for my free pass | 22:41 |
| anteaya | ha ha ha | 22:41 |
| *** mriedem has quit IRC | 22:41 | |
| loquacities | anteaya: well, we're kind of used to it these days ;) | 22:41 |
| loquacities | lol | 22:41 |
| anteaya | jeblair: you need a few co-authors to drag along with you | 22:41 |
| loquacities | jeblair: patches welcome :P | 22:41 |
| *** gnuoy_ has quit IRC | 22:41 | |
| fungi | ultimately, the point is to find a way to make it as easy as possible for active contributors to attend, and to have a system which can identify as many active contributors as possible. if a bunch of freeloaders come along for the ride, i guess that's the tradeoff for being open and inclusive (and maybe slightly gullible) | 22:42 |
| nibalizer | mtreinish: http://puppetboard.openstack.org/node/subunit-worker02.openstack.org | 22:42 |
| ianw | i dunno, devstack gets quite a few and i just kick them in. a fix is a fix, we don't try to spell things wrong. if, however, it makes financial sense from a conference organising perspective, i don't know | 22:42 |
| loquacities | yeah, i don't know that the system could (or even should) be ungameable | 22:42 |
| loquacities | it comes down to cores being sane, i think | 22:42 |
| mtreinish | nibalizer: very cool | 22:43 |
| anteaya | ianw: it is less financial and more full seats that drain energy yet offer nothing in return | 22:43 |
| nibalizer | jeblair: in theory i understand our beaker tests | 22:43 |
| loquacities | even docs won't accept a patch for a patch's sake | 22:43 |
| docaedo | loquacities: is there such a thing as a sane core? | 22:43 |
| *** baoli has joined #openstack-infra | 22:43 | |
| loquacities | docaedo: that's probably an oxymoron | 22:43 |
| fungi | docaedo: only when they first start | 22:43 |
| docaedo | :P | 22:43 |
| loquacities | we're all at least slightly insane, right ;) | 22:43 |
| jeblair | nibalizer: i'm wondering what's happening to pip.conf; working theory: our own pip module is overwriting it in the openstackci beaker tests. but i don't see evidence of that in the jenkins log | 22:43 |
| nibalizer | ya | 22:44 |
| loquacities | anyway, the point is we need to continue to apply the "is it better than we already have" test | 22:44 |
| nibalizer | i think i've put my head into that at least once and never got anywhere | 22:44 |
| anteaya | loquacities: agreed | 22:44 |
| nibalizer | honestly im a bit too fragmented right now to look at it | 22:44 |
| loquacities | so, if it's a legit typo that someone has fixed, it goes in, no matter who proposes it | 22:44 |
| *** apevec has left #openstack-infra | 22:44 | |
| loquacities | but if you're rearranging deck chairs, let's not bother | 22:44 |
| * anteaya offers nibalizer another stripe for his shirt sleeve | 22:44 | |
| openstackgerrit | Merged openstack-infra/system-config: Add subunit-worker02.o.o to the heira data https://review.openstack.org/288152 | 22:44 |
| crinkle | jeblair: I see http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/server.pp#n18 which leads to http://git.openstack.org/cgit/openstack-infra/puppet-pip/tree/templates/pip.conf.erb#n2 is that what controls the mirror setting? | 22:45 |
| *** sc68cal has quit IRC | 22:45 | |
| crinkle | I don't see puppet-openstackci setting that anywhere | 22:45 |
| mtreinish | nibalizer: ^^^ once that takes effect on logstash.o.o we should be good to go on the subunit-worker02.o.o daemon | 22:45 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Temporarily stop using 2-node workers in vexxhost" https://review.openstack.org/288080 | 22:45 |
| fungi | crinkle: nodepool ready scripts in project-config | 22:46 |
| jeblair | crinkle: yeah, that's the only thing i can think of off the top of my head that could be erroneously resetting it | 22:46 |
| jeblair | fungi: right, that's what *correctly* sets it before the job runs | 22:47 |
| jeblair | crinkle: ^ | 22:47 |
| fungi | crinkle: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh is run by nodepool when the instance is first booted | 22:47 |
| nibalizer | infra-root i am cleaning uuid nodes from puppetdb | 22:48 |
| fungi | crinkle: that creates the /etc/pip.conf which is in place by the time the job starts | 22:48 |
| crinkle | I see | 22:48 |
| mtreinish | clarkb: on the temporarily stop multinode vexxhost patch, was it failing before devstack?: o-h shows all green on ymq1: http://status.openstack.org/openstack-health/#/g/node_provider/ymq-1?groupKey=node_provider&resolutionKey=hour&searchJob=multinode | 22:48 |
| mtreinish | granted it was just reverted, I'm just curious | 22:48 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Fixes for latest cinder and neutron clients https://review.openstack.org/288110 | 22:48 |
| jeblair | so to try to clarify at the risk of being repetitive: the nodepool ready scripts set the mirror in pip.conf before a job runs. apparently, during the job run, pip.conf is updated to remove the mirror and replace it with pypi.python.org. at least, it seems that way because we get timeout errors from pypi.python.org. i don't know what could cause the file to be rewritten other than the code crinkle pointed at, but i see no evidence ... | 22:49 |
| jeblair | ... it actually is responsible. | 22:49 |
| pleia2 | nibalizer: thanks for looking into into that for mtreinish, headache today, ruining my day :( | 22:49 |
| nibalizer | pleia2: no worries | 22:49 |
| nibalizer | rest up and feel better! | 22:49 |
| fungi | mtreinish: i believe the symptom was tempest test ssh timeouts to instances booted on the subnode | 22:49 |
| pleia2 | nibalizer: thanks <3 | 22:49 |
| fungi | mtreinish: mtu blachhole | 22:50 |
| fungi | blackhole | 22:50 |
| fungi | mtreinish: the fix was to make mtu determination in d-g dynamically calculated from the interface mtu | 22:50 |
| mtreinish | fungi: heh, but yeah that should have be shown in openstack-health (unless it was all check) | 22:50 |
| crinkle | jeblair: got it...I also don't see where puppet would be overwriting pip.conf yet | 22:51 |
| fungi | mtreinish: rather than blindly assuming all providers give us at least a 1500 byte mtu | 22:51 |
| mtreinish | ah, that's probably a good call :) | 22:51 |
| fungi | mtreinish: did you have any show up for vexxhost in openstack-health at all? the quota was a relatively small percentage of our overall global quota | 22:52 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Allow passing tenant_id to create_network https://review.openstack.org/288166 | 22:52 |
| mtreinish | fungi: http://status.openstack.org/openstack-health/#/g/node_provider/ymq-1?groupKey=node_provider&resolutionKey=hour | 22:52 |
| fungi | and didn't get ramped up until last night and then got pulled out before we got too far into today | 22:52 |
| mtreinish | it's not many jobs but it's definitely there (w/ 4 multinodes) | 22:53 |
| fungi | mtreinish: i know in the past when we had cross-node communication impact it was only when tempest and the instance it was hitting were on different nodes, so maybe the ones in the gate just got lucky? | 22:54 |
| fungi | mtreinish: i think sdague linked to some failure examples this morning, but i don't have time to dig them out of the irc log right now | 22:55 |
| *** ayoung has quit IRC | 22:55 | |
| *** moravec has quit IRC | 22:55 | |
| fungi | he said something about hitting it twice this morning on patches of his, which is how he spotted it | 22:56 |
| *** ihrachys has joined #openstack-infra | 22:56 | |
| mtreinish | fungi: yeah, so it could have been on check, o-h is still only for gate | 22:57 |
| *** sdake has quit IRC | 22:57 | |
| *** ihrachys has quit IRC | 22:58 | |
| anteaya | I keeping thinking o-h is for ovh but somehow that is too sacrosanct | 22:59 |
| *** piet has joined #openstack-infra | 22:59 | |
| *** ihrachys_ has quit IRC | 22:59 | |
| mtreinish | anteaya: heh, well you were the one who came up with openstack-health as the name in ft collins :) | 23:00 |
| *** sdake has joined #openstack-infra | 23:00 | |
| kevinbenton | hello all. Can we shutoff the Cisco 3rd party CI accounts for Neutron? They are just spitting out 'failure to merge' errors on each patchset. (e.g. https://review.openstack.org/#/c/287611/) | 23:01 |
| anteaya | mtreinish: I only have myself to blame | 23:02 |
| anteaya | kevinbenton: thanks I'll take a look | 23:02 |
| *** yaume_ has quit IRC | 23:02 | |
| *** jsavak has quit IRC | 23:02 | |
| kevinbenton | thanks, it's not critical because it's not -1. it's just a little harder to filter through the other CI responses with them | 23:03 |
| anteaya | cisco ucsm isn't listed in the third party systems wikipage | 23:04 |
| anteaya | kevinbenton: nope I understand, thanks for speaking up | 23:04 |
| anteaya | going to start by seeing if I can find any of the contacts in irc | 23:04 |
| kevinbenton | anteaya: thanks for doing a thankless job! :) | 23:04 |
| anteaya | ha ha ha | 23:04 |
| anteaya | I'm sending you your invoice :) | 23:05 |
| anteaya | and thanks for putting up with me | 23:05 |
| anteaya | same two people on the three accounts that are listed | 23:05 |
| *** xyang1 has quit IRC | 23:06 | |
| *** xyang1 has joined #openstack-infra | 23:07 | |
| openstackgerrit | Merged openstack/requirements: Bump upper-constraint for heatclient to 1.0.0 https://review.openstack.org/287351 | 23:07 |
| *** sridhar_ram has quit IRC | 23:07 | |
| *** marcusvrn_ has quit IRC | 23:07 | |
| *** baoli has quit IRC | 23:08 | |
| *** ybathia has quit IRC | 23:08 | |
| *** baoli has joined #openstack-infra | 23:09 | |
| *** gildub has joined #openstack-infra | 23:09 | |
| *** xyang1 has quit IRC | 23:09 | |
| *** geekinutah has joined #openstack-infra | 23:10 | |
| *** sdake has quit IRC | 23:11 | |
| *** pradk has quit IRC | 23:11 | |
| mtreinish | nibalizer: so we should be good to start the worker (if you didn't already): http://puppetboard.openstack.org/report/logstash.openstack.org/9284c73ad1ffd3606ab8bcc59ba77aa7a9ff5d55 | 23:11 |
| nibalizer | mtreinish: so just service something | 23:12 |
| mtreinish | yeah, it's service subunit-gearman(-worker)(-A) start (or something like that) | 23:13 |
| *** zeih has joined #openstack-infra | 23:13 | |
| pleia2 | s/gearman/jenkins | 23:13 |
| mtreinish | pleia2: heh, it's jenkins? | 23:13 |
| pleia2 | mtreinish: yeah, not confusing at all! | 23:13 |
| mtreinish | oh I guess I just used whatever naming scheme clarkb used for the log processor worker | 23:14 |
| pleia2 | probably so | 23:14 |
| nibalizer | http://puppetboard.openstack.org/report/subunit-worker02.openstack.org/a8dd05fac84d031b72c1bf095adda2ae34665d80 | 23:14 |
| nibalizer | mtreinish: we still have errors | 23:14 |
| nibalizer | oh you know what | 23:14 |
| fungi | les cultes des cargo? | 23:14 |
| nibalizer | ansible cached | 23:15 |
| nibalizer | so it didn't pick it up | 23:15 |
| nibalizer | it wouldn't have cached except that I ran launch-node as nonroot | 23:15 |
| *** amotoki has quit IRC | 23:16 | |
| *** sridhar_ram has joined #openstack-infra | 23:16 | |
| anteaya | fungi: is it better if it is French? | 23:16 |
| *** sdake has joined #openstack-infra | 23:17 | |
| fungi | anteaya: maybe. that was probably also not really french | 23:17 |
| *** markvan has quit IRC | 23:17 | |
| anteaya | probably | 23:17 |
| *** zeih has quit IRC | 23:17 | |
| *** rkukura_ has joined #openstack-infra | 23:17 | |
| *** ayoung has joined #openstack-infra | 23:18 | |
| *** fawadkhaliq has quit IRC | 23:18 | |
| *** fawadkhaliq has joined #openstack-infra | 23:18 | |
| anteaya | yeah according to google translate cargo cult to french isn't really exciting | 23:19 |
| anteaya | however | 23:19 |
| anteaya | english to zulu is really interesting | 23:19 |
| anteaya | lwezinhlangano ezingathandwa ezibhekwa cargo | 23:19 |
| *** rkukura has quit IRC | 23:19 | |
| *** rkukura_ is now known as rkukura | 23:19 | |
| anteaya | look at that | 23:19 |
| *** tobiash_ has quit IRC | 23:19 | |
| anteaya | and cargo didn't even move | 23:19 |
| fungi | wow | 23:19 |
| anteaya | that was just cult | 23:19 |
| fungi | rather a verbose language | 23:20 |
| *** sarob has quit IRC | 23:20 | |
| anteaya | so it appears | 23:20 |
| *** sarob has joined #openstack-infra | 23:20 | |
| *** kzaitsev_mb has joined #openstack-infra | 23:20 | |
| fungi | i can only imagine how time-consuming incidencal conversation must be | 23:20 |
| anteaya | ha ha ha | 23:20 |
| fungi | er, incidental | 23:20 |
| anteaya | both | 23:20 |
| *** tobiash has joined #openstack-infra | 23:20 | |
| *** annegentle has quit IRC | 23:21 | |
| *** fawadkhaliq has quit IRC | 23:21 | |
| *** weshay has quit IRC | 23:21 | |
| *** fawadkhaliq has joined #openstack-infra | 23:21 | |
| *** annegentle has joined #openstack-infra | 23:21 | |
| anteaya | kevinbenton: trying to find the switch on the cisco side now, going to wait a bit hoping they are successful | 23:22 |
| anteaya | kevinbenton: thanks for saying something | 23:22 |
| kevinbenton | anteaya: cool. glad they are helping | 23:22 |
| *** sdake has quit IRC | 23:22 | |
| anteaya | yeah me too | 23:22 |
| *** Jeffrey4l has joined #openstack-infra | 23:23 | |
| *** markvan has joined #openstack-infra | 23:24 | |
| *** sarob has quit IRC | 23:24 | |
| *** baoli has quit IRC | 23:26 | |
| *** baoli has joined #openstack-infra | 23:27 | |
| *** Sukhdev has quit IRC | 23:27 | |
| *** ianw has quit IRC | 23:28 | |
| *** gordc has quit IRC | 23:29 | |
| *** tqtran has joined #openstack-infra | 23:30 | |
| *** dangers_` is now known as dangers_away | 23:32 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: test all jobs w/ NETISO_V4 https://review.openstack.org/288176 | 23:32 |
| *** annegentle has quit IRC | 23:32 | |
| *** annegentle has joined #openstack-infra | 23:33 | |
| *** ianw has joined #openstack-infra | 23:35 | |
| *** ccrouch1 has quit IRC | 23:35 | |
| *** yamahata has joined #openstack-infra | 23:35 | |
| *** ccrouch has joined #openstack-infra | 23:38 | |
| *** annegentle has quit IRC | 23:38 | |
| *** sdake has joined #openstack-infra | 23:41 | |
| *** zz_dimtruck is now known as dimtruck | 23:41 | |
| ianw | does anyone use weechat / glowing bear? | 23:42 |
| *** rhallisey has joined #openstack-infra | 23:42 | |
| ianw | i'm wondering if it can be a real irc bouncer (like znc), or if you can only use glowing bear to it | 23:42 |
| fungi | ianw: i do use weechat, but i just run it under tmux on a very stable shell server in the cloud and mosh to that from everywhere | 23:45 |
| fungi | familiar for me, since i was doing the same thing via telnet to a gnu screen session running ircii (if i rewind this pattern far enough) | 23:46 |
| *** Jeffrey4l has quit IRC | 23:47 | |
| Clint | with your arsenal of ircii warscripts | 23:47 |
| *** ybathia has joined #openstack-infra | 23:47 | |
| *** sflanigan has joined #openstack-infra | 23:47 | |
| *** sflanigan has joined #openstack-infra | 23:47 | |
| *** jamesmcarthur has quit IRC | 23:47 | |
| *** baoli has quit IRC | 23:48 | |
| ianw | fungi: i was looking for something with a half-decent phone client, but on my desktop, i also want a separate irc client connected over the vpn for internal irc | 23:48 |
| *** jamesmcarthur has joined #openstack-infra | 23:48 | |
| ianw | i guess i could have weechat for freenode externally hosted (and phone available), then my inside-vpn irc bouncer, and use weechat client on my desktop to talk to both ... hmm, that might work | 23:50 |
| *** grue_pm has joined #openstack-infra | 23:51 | |
| fungi | yeah, when i've needed to connect to internal work irc servers in the past, i just run a separate instance of the client somewhere with access to the vpn or internal network | 23:52 |
| *** salv-orl_ has joined #openstack-infra | 23:52 | |
| fungi | which usually meant i treated my workstation on my desk at hq as a shell server, and ssh'd into that over my work vpn from home/the road | 23:52 |
| fungi | ran irssi or whatever was du jour at the time under screen and detached/reattached as needed | 23:53 |
| anteaya | the eyes in the glowing bear icon | 23:53 |
| *** jamesmcarthur has quit IRC | 23:54 | |
| fungi | it meant personal and work irc were always in separate xterms, but in some ways i preferred that | 23:54 |
| anteaya | it is good to have work in a separate xterm | 23:55 |
| *** salv-orlando has quit IRC | 23:55 | |
| *** abregman|nb has quit IRC | 23:56 | |
| crinkle | jeblair: fungi I can't reproduce the mirror issue locally, it doesn't seem to me like anything in beaker or puppet is changing pip.conf, at least for puppet-openstackci | 23:57 |
| *** ybathia has quit IRC | 23:58 | |
| fungi | an infra-root might need to catch a node running that job, hold it and perform some local inspection to validate assumptions | 23:58 |
| fungi | or write a throwaway change to echo various files to the console before and after the job | 23:58 |
| fungi | or maybe enhance the job to collect configs we're unsure about | 23:59 |
| *** ianw has quit IRC | 23:59 | |
| *** ddieterly has quit IRC | 23:59 | |
| *** sarob has joined #openstack-infra | 23:59 | |
| *** ianw has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!