Tuesday, 2016-01-26

*** amotoki has joined #openstack-infra00:00
*** ashleighfarnham has quit IRC00:01
*** Jeffrey4l has joined #openstack-infra00:02
dougwigproxy errors from gerrit, fyi.00:03
*** Sukhdev has joined #openstack-infra00:03
dougwigand it's back. :)00:03
*** yamamoto_ has quit IRC00:06
*** maishsk has quit IRC00:07
*** maishsk has joined #openstack-infra00:08
*** r-daneel has quit IRC00:08
*** sabeen2 has quit IRC00:09
openstackgerritKevin Carter proposed openstack-infra/project-config: Added new project for the OSA role break out  https://review.openstack.org/27233100:11
*** ybathia has joined #openstack-infra00:12
jeblairdougwig: errors or delays at 00:00 UTC are database backups :(00:14
*** jtomasek has quit IRC00:14
*** amotoki has quit IRC00:15
*** Qiming has quit IRC00:15
jeblairmordred, krotscheck: the dkms patch automatically fixed gra1: http://mirror.gra1.ovh.openstack.org/pypi/00:15
jeblairso we're up in rax and ovh00:15
dougwigjeblair: oh didn't notice it was 00:00.  wow, las vegas and boise are in different timezones. thanks.00:16
*** diana_clarke has joined #openstack-infra00:16
jeblairdougwig: the rule is that no matter where in the world you are, when gerrit does db backups, it's time for beer00:16
*** salv-orl_ has quit IRC00:17
dougwigjeblair: i love that rule.00:17
*** chlong has quit IRC00:20
*** chlong has joined #openstack-infra00:21
*** rfolco has joined #openstack-infra00:21
*** bodepd has joined #openstack-infra00:21
*** bodepd has quit IRC00:22
*** bodepd has joined #openstack-infra00:22
*** rbrndt has quit IRC00:24
*** dhp_ has quit IRC00:24
openstackgerritJohn L. Villalovos proposed openstack/requirements: Add openstack/bifrost to projects.txt  https://review.openstack.org/27233400:25
*** rfolco has quit IRC00:26
jeblairmordred: i ran shade-launch-node on hpcloud and got 'Server reached ACTIVE state without being allocated an IP address.'00:26
*** hichihara has joined #openstack-infra00:27
*** jtomasek has joined #openstack-infra00:27
*** zz_dimtruck is now known as dimtruck00:29
jeblair(an aside -- wrapping lines at at 78 characters makes it difficult to grep for error messages)00:30
*** gildub has quit IRC00:31
nibalizeris shade-launch-node something in infra or something in shade?00:32
jeblairnibalizer: something in infra; patch has not landed: https://review.openstack.org/23497500:32
jeblairnibalizer: regardless, it's the only way to launch mirror nodes :|00:32
jeblairnibalizer: since (regular) launch-node really only works on rax and hpcloud00:33
jeblairnibalizer: so i'll probably switch back to that just to get this node up00:33
jeblairoh, except somehow novaclient got upgraded on the system and now that doesn't work anymore :(00:36
*** Sukhdev has quit IRC00:39
nibalizerjeblair: ugh00:39
*** chenli has joined #openstack-infra00:40
*** oomichi has joined #openstack-infra00:40
*** chlong has quit IRC00:41
dstufftjeblair: has this mirror not benes ynced in some time?00:41
jeblairdstufft: correct, it was months00:41
dstufftwhen we migrated to the new database server there was some blips where some pacakges got recorded in the DB but didn't get actually uploaded that I deleted00:41
*** chlong has joined #openstack-infra00:42
dstufftprobably related ot that00:42
jeblairdstufft: ok, so my solution of removing them from todo was probably okay then00:42
*** david-lyle has joined #openstack-infra00:42
jeblaircool, seems to have been okay after that00:42
dstufftI can't wait till warehouse gets some saner transaction handling up on PyPI :/00:43
dstufftpart of the reason I'm going to be migrating to /packages/<sha256 hash>/<filename> is so we can write files to storage *before* committing the transaction without possibly overwriting a file00:44
dstufft(or at least, since the package name includes a hash of the file, if we do overwrite, it'll be with a file that is completely equal to what was already there)00:45
jeblairdstufft: ++; do you care about leaking those files in case the transaction doesn't complete?00:46
jeblairdstufft: (maybe clean them up periodically based on a check of the db?)00:46
*** rbrndt_ has joined #openstack-infra00:47
dstufftjeblair: not specifically, but I might just make a garbage collector job that looks for any files > some age that doesn't exist in the DB yea00:47
jeblairnibalizer: i downgraded novaclient and launched (regular) launch-node00:48
*** chlong has quit IRC00:48
*** chlong has joined #openstack-infra00:49
*** ociuhandu has quit IRC00:49
*** zhurong has joined #openstack-infra00:51
*** EricGonczer_ has joined #openstack-infra00:52
*** chlong has quit IRC00:53
*** piet has quit IRC00:53
*** chlong has joined #openstack-infra00:54
*** FallenPegasus has joined #openstack-infra00:56
*** pvaneck_ has joined #openstack-infra00:57
nibalizerjeblair: cool00:57
*** ajmiller has quit IRC00:58
*** pvaneck_ has quit IRC01:01
*** pvaneck has quit IRC01:01
*** tphummel has quit IRC01:01
*** Daisy has joined #openstack-infra01:02
*** Qiming has joined #openstack-infra01:02
jeblairthe kernel module failed to build on internap01:03
jeblairthere's something funny going on with the kernel package there i think01:03
*** Daisy has quit IRC01:03
*** amotoki has joined #openstack-infra01:04
jeblairmgagne: where did linux version 3.16.0-59.79~14.04.1 come from?01:05
jeblairmgagne: it looks like a backported utopic kernel01:05
*** zhurong has quit IRC01:06
*** ybathia has quit IRC01:08
mgagnejeblair dpkg --list | grep kernel01:08
*** sridhar_ram1 is now known as sridhar_ram01:08
prometheanfiremordred: so... if I were to switch to default to be syslinux, that wouldn't mess anything up would it?01:08
prometheanfirejust curious, probably not going to01:08
prometheanfiresyslinux works just fine01:08
jeblairmgagne: http://paste.openstack.org/show/484941/01:09
prometheanfiremordred: also, now doing minimal stage-4s https://gist.github.com/0ccef20b5dc82cbf5df501:09
mgagnelts-utopic =)01:09
prometheanfirebuilding one now :D (from stage1, should take a while...)01:09
mgagnecould be pull by cloud-initramfs-growroot01:09
mgagneor could also be the base image having lts installed by default01:09
mgagnejeblair ^01:10
*** yuanying has joined #openstack-infra01:10
mgagnejeblair now it should be vivid, maybe it used to be utopic01:10
*** peterlisak has quit IRC01:10
jeblairmgagne: i kind of just want it to be trusty :)01:10
*** kzaitsev_mb has quit IRC01:11
mgagnejeblair I'm not sure how we can do that01:11
mgagneas the kernel is part of the base image AFAIK01:11
jeblairmgagne: i'm running trusty01:11
*** yuanying_ has quit IRC01:11
*** onovy has quit IRC01:12
mgagnejeblair yes, I know, I'm telling that Trusty 14.04.1 might be coming with lts-utopic01:12
mgagneand it's not something we do on purpose01:12
mgagneit's part of the Ubuntu release01:12
nibalizerjeblair: perhaps we could upload the ubuntu trusty image ourselves? either via dib or just the cannonical provided cloud images?01:12
*** peterlisak has joined #openstack-infra01:12
jeblairmgagne: hrm, that page makes it look like this is all add-on stuff; and it even recommends not doing it for cloud images...01:12
mgagnenibalizer I thought it was already the case, unless we are talking about a non-nodepool image01:12
*** onovy has joined #openstack-infra01:12
jeblairmgagne: maybe SpamapS can shed some light on it01:12
jeblairmgagne: we are talking about a non-nodepool image01:13
mgagne"If you have installed with older media you can use the following to install the newer kernel"01:13
*** pc_m has quit IRC01:13
jeblairmgagne: i'm spinning up a replacement mirror server, so using the internap trusty image01:13
nibalizerI'm assuming some of jeblair's behaviour, but I think he's looking at the otput of 'nova image-list' and using the most trusty-looking ubuntu image in that list01:13
jeblairmgagne: (we don't have dib-built images for our long-lived servers yet, though we plan to, and it will solve this sort of problem)01:13
jeblairnibalizer, mgagne: correct, specifically  "Ubuntu 14.04 LTS (Trusty Tahr)"01:14
mgagnewe just can't ship a non-updated image, that would be negligence, like shipping windows without updates01:14
jeblairmgagne: i'm not asking you too01:14
nibalizerjeblair: perhaps http://apps.openstack.org/#tab=glance-images&asset=Ubuntu%20Trusty%2014 could be used in the interm between now and when we get dib-for-servers up?01:14
jeblairmgagne: i'm saying that somewhere, something has gone wrong and the trusty image is not a simple trusty image; it's trusty + stuff backported from new releases01:15
*** pc_m has joined #openstack-infra01:15
*** Liuqing has joined #openstack-infra01:15
mgagnelet me reword: we won't ship an older version so you can get the "trusty" kernel as it's no longer part of what Ubuntu ships on their mirror. behavior is "expected" based on wiki page01:15
jeblairmgagne: i think you are misreading that01:15
jeblairmgagne: that page says "nd not recommended for cloud or virtual images"01:15
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue  https://review.openstack.org/26732301:16
mgagnejeblair well, I consider the "something went wrong" a misunderstanding or misinformation about what Ubuntu ships IRL01:16
mgagneand expectations are wrong01:16
mgagnejeblair I missed that part then =)01:16
*** Daisy has joined #openstack-infra01:16
mgagnejeblair we used this issue: http://releases.ubuntu.com/14.04/ubuntu-14.04.2-server-amd64.iso01:17
*** alivigni has joined #openstack-infra01:17
*** salv-orlando has joined #openstack-infra01:17
jeblairmgagne, nibalizer: so perhaps i should try manually uploading the cloud image nibalizer linked to at http://apps.openstack.org/#tab=glance-images&asset=Ubuntu%20Trusty%2014 and see if that has it01:19
mgagneproposed alternative solutions aren't great tbh: 1) install older versions (wth) 2) upgrade to newest LTS or downgrade to previous LTS (more WTH) 3) use netboot images instead of <release>-netboot images. (not sure what it means)01:19
*** Swami has quit IRC01:19
nibalizermgagne: it is possible (and mostly supported) to plop a new kernel into ubuntu LTS, but ubuntu does maintain a trusty kernel for the entire lifetime of trusty01:19
mgagnejeblair last time we tested "vanilla" images, it had *very* outdated cloudinit and images wouldn't boot and configure itself.01:20
jeblairmgagne: yeesh.01:20
jeblairmgagne: if that fails, i'll try the 14.04.3 server image.01:20
mgagnejeblair you will have the vivid kernel01:20
*** salv-orlando has quit IRC01:20
jeblairmgagne: :(01:20
alivigniQuestion what is the stance on using Jenkins Job Builder vs. the Workflow plugin?01:20
mgagnejeblair unless you have a ubuntu cloud image01:20
mgagnejeblair but I lost all hopes in having an updated cloudinit01:21
mgagnejeblair which module are you trying to compile? AFS?01:21
jeblairmgagne: maybe i'll just perform surgery on this instance to unblock what i'm working on -- let's pick this up tomorrow with mordred and SpamapS01:21
jeblairmgagne: yep01:22
mgagnejeblair I guess you could forcefully downgrade the kernel01:22
mgagnebut then I see it as a problem with the AFS kernel module as would it be a metal instance, you would have the same problem and "but it's cloud" wouldn't help you01:22
*** gildub has joined #openstack-infra01:23
mgagnejeblair what are the specific problem you have? is there a way I can test it on my side and see if upgrading to latest lts-vivid kernel fix it?01:23
jeblairmgagne: yes, there's a bug in the utopic kernel package for sure; but if i were running on metal, i would also remove the lts enablement stack unless needed :)01:23
mgagnejeblair as said, it's part of the base iso. little we can do about it, there is no cloud .iso01:24
openstackgerritMaksim Malchuk proposed openstack-infra/project-config: Launch unit tests for only changed modules  https://review.openstack.org/27234601:24
mgagneso I don't know what I'm supposed to do. DIB might be the solution where we could download the ubuntu cloud image, crack it open, update cloud-init and ship it.01:25
jeblairmgagne: we'll get a lot more answers tomorrow with other folks around :)01:25
mgagneit's 8:30pm already :O01:26
*** erlon has quit IRC01:26
mgagnelet me know if you can find an option I can feed to preseed to not enable this kernel update01:27
mgagnejeblair does vivid kernel fix the problem?01:28
* krotscheck reads scrollback01:28
mgagnejeblair we could rebuild against 14.04.3 and see what happens01:28
jeblairmgagne: the _only_ kernel i want to run is trusty :)01:28
mgagne"it used to work before you touched the computer, put it back!" "yes mom..." =)01:29
jeblairmgagne: heh :)  i have too many machines to have them be different01:29
*** jpr has joined #openstack-infra01:29
krotscheckjeblair:, mordred: Brief summary response to all the things, not necessarily in any order: Awesome! Wat. That's ironic. Well, it could be that it's adding a little extra lag because it's fetching things.01:30
*** mtanino__ has quit IRC01:30
mgagnejeblair https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64.manifest01:30
mgagnedoesn't have lts kernel01:30
mgagnebut our challenge is what our metal instances, we use the same image ^^'01:31
*** apoorvad has quit IRC01:35
*** piet has joined #openstack-infra01:36
*** jpr has quit IRC01:36
*** ildikov has quit IRC01:36
*** angdraug has quit IRC01:37
*** sridhar_ram has quit IRC01:37
*** bpokorny has quit IRC01:38
*** r-daneel has joined #openstack-infra01:39
*** peterlisak has quit IRC01:39
*** r-daneel has quit IRC01:39
*** peterlisak has joined #openstack-infra01:39
jeblairmgagne, nibalizer: utopectomy complete and http://mirror.nyj01.internap.openstack.org/ is up :)01:42
mgagnewhat happened? =)01:43
*** sdake has quit IRC01:43
jeblairmgagne: ?01:43
mgagneto the kernel01:43
jeblairmgagne: apt-get remove --purge01:43
jeblairmgagne: then reinstall of linux-image-generic01:43
jeblairmgagne: we rewrite sources.list, so we should't upgrade to it anymore01:44
mgagneyou shouldn't be able to upgrade to lts-utopic afaik, it's a new package (not an upgrade)01:44
jeblairthat too01:44
jeblairmgagne: and yeah, i make sure to uninstall the meta-packages too01:45
*** FallenPegasus is now known as MarkAtwood_01:45
*** flepied has quit IRC01:46
*** piet has quit IRC01:50
*** gildub has quit IRC01:50
jeblairthe bluebox mirror is up01:51
*** jpr has joined #openstack-infra01:52
jeblair(fwiw, if you set a 50G afs cache, the first time afs starts, it does take a bit as it pre-populates the cache dir)01:52
*** salv-orlando has joined #openstack-infra01:54
*** piet has joined #openstack-infra01:54
*** sdake has joined #openstack-infra01:55
*** chlong has quit IRC01:55
*** chlong has joined #openstack-infra01:56
*** salv-orlando has quit IRC01:59
*** yuanying has quit IRC02:00
*** flepied has joined #openstack-infra02:00
*** amitgandhinz has joined #openstack-infra02:02
*** tiswanso has joined #openstack-infra02:02
*** jsavak has joined #openstack-infra02:03
*** yuanying has joined #openstack-infra02:03
jeblairhttp://mirror.region-b.geo-1.hpcloud.openstack.org/ is up02:04
jeblairinfra-root, krotscheck: the new afs-backed pypi mirrors in all of the nodepool cloud regions are up and running now02:05
*** david-lyle has quit IRC02:05
jeblairi'm rechecking https://review.openstack.org/272215 to spot any errors02:06
jeblairnext step is https://review.openstack.org/23875602:06
*** tiswanso has quit IRC02:06
*** tiswanso has joined #openstack-infra02:07
*** dimtruck is now known as zz_dimtruck02:09
*** zhurong has joined #openstack-infra02:10
nibalizerjeblair: great02:11
*** davidlenwell has quit IRC02:14
*** yamamoto_ has joined #openstack-infra02:16
*** davidlenwell has joined #openstack-infra02:16
*** salv-orlando has joined #openstack-infra02:19
*** keedya has joined #openstack-infra02:19
*** fawadkhaliq has joined #openstack-infra02:21
*** baoli has joined #openstack-infra02:21
*** yuanying has quit IRC02:25
*** gongysh has joined #openstack-infra02:25
*** salv-orlando has quit IRC02:27
*** yuanying has joined #openstack-infra02:27
*** amitgandhinz has quit IRC02:28
*** rossella_s has quit IRC02:31
*** rossella_s has joined #openstack-infra02:32
jeblairinfra-root, krotscheck: it looks like we still have some _very_ out of date images, so we can't use the new mirrors until they are updated with the trusted hosts list02:32
*** Daisy has quit IRC02:32
jpri'm seeing some problems during image create via nodepool on my openstackci deploy02:36
jprI can get a simple image to build under a normal user account using `disk-image-create ubuntu vm`02:37
*** fawadkhaliq has quit IRC02:38
jprif I register this definition under nodepool.yaml and i can get the build to start with nodepool only if I start the nodepoold.  It then triggers a build with the command line "/bin/bash /usr/local/bin/disk-image-create -x -t --no-tmpfs -o /opt/nodepool_dib/5 ubuntu vm"  (I'm concerned about the bare -t btw)02:39
greghaynesjpr: -t is for the output format02:39
jpryes, is a bare -t ok?02:40
greghaynesjpr: it is not02:40
greghaynesjpr: you need to set02:40
greghayneser, sorry02:40
jprgreghaynes:  ok, so I'm assuming the only way to spec that is with the image-type in the provider section02:40
greghaynesjpr: yes02:41
greghaynesjpr: nodepool should handle that better and either fail or default to no -t02:41
*** salv-orlando has joined #openstack-infra02:41
jprgreghaynes: yeah.  I thought that might be the case.  i'll look at the code for that logic to see if there's something simple02:42
*** yamamoto_ has quit IRC02:42
greghaynesjpr: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/builder.py#n352 is likely useful ;)02:43
jprin any case.  the builds all fail with a bork of the ci node (ie. the main node in the openstackci config) because the /dev/ tree gets messed up.  all the devices are deleted.   don't know if this is from a bad bind mount or something intermediate that fails02:43
greghaynesjpr: oh crap02:43
greghaynesjpr: what elements?02:43
greghaynesdiskimage-builder had a bug years ago that deleted all of /dev02:43
*** Liuqing has quit IRC02:43
greghaynesI hope we didnt repeat02:43
jprgreghaynes: definitely the ones for the dpc image but also the simplified ones I'm testing "ubuntu vm"02:44
*** Liuqing has joined #openstack-infra02:44
greghaynesI feel like I would have noticed that for ubuntu vm02:44
jprgregheynes: if i run disk image build manually it works fine02:44
jprit's through nodepool that its a problem02:45
greghaynesthat really should have no effect, so I dont have any good ideas02:45
*** zhurong has quit IRC02:46
*** zhurong has joined #openstack-infra02:47
*** salv-orlando has quit IRC02:48
*** chlong has quit IRC02:49
jprgreghaynes: btw, i can only initiate the nodepool image build via the cli if nodepoold itself is running.  based on comments from asselin over the weekend, that may be expected behavior now02:49
*** chlong has joined #openstack-infra02:50
greghaynesjpr: it is. You could also run a nodepool-builder instead of nodepoold02:50
*** ansiwen_ has joined #openstack-infra02:52
*** zhurong has quit IRC02:53
*** zhurong has joined #openstack-infra02:54
*** fawadkhaliq has joined #openstack-infra02:54
jprgreghayes: I'll try setting image-type to see if that makes a difference in behavior02:55
*** daemontool has quit IRC02:56
*** ansiwen has quit IRC02:59
jprgreghaynes: here's the tail end of the image.log file during the nodepool image build.  it looks like there's a mv command the moves all the entries from the bind-mounted dev dir into the new image..  That's likely the cause of the issue.  things go south from there. http://paste.openstack.org/show/484948/03:01
*** pradk has quit IRC03:02
*** ansiwen_ has quit IRC03:03
*** ansiwen has joined #openstack-infra03:03
greghaynesjpr: hrm, so can you file a bug and mention what distro youre building on, what elements you are using, and the command you ran?03:04
greghaynesjpr: its late here so I'll have to look at it tomorrow03:05
greghaynesjpr: a bug against diskimage-builder on launchpad, that is03:05
jprgreghaynes: yes. thanks for the target.   also gonna try and debug a little further.03:05
*** unicell has quit IRC03:06
*** tlian has quit IRC03:06
*** jsavak has quit IRC03:06
*** baoli has quit IRC03:08
*** gildub has joined #openstack-infra03:09
*** achanda has joined #openstack-infra03:09
*** alivigni has quit IRC03:10
*** zz_dimtruck is now known as dimtruck03:13
*** pradk has joined #openstack-infra03:16
*** chlong has quit IRC03:17
*** chlong has joined #openstack-infra03:18
*** chlong has quit IRC03:19
*** yuanying_ has joined #openstack-infra03:19
*** chlong has joined #openstack-infra03:20
*** yuanyin__ has joined #openstack-infra03:20
*** yuanying has quit IRC03:22
*** yuanying_ has quit IRC03:23
*** david-lyle has joined #openstack-infra03:25
*** baoli has joined #openstack-infra03:25
*** woodster_ has quit IRC03:26
*** yamamoto_ has joined #openstack-infra03:28
*** yuanyin__ has quit IRC03:29
openstackgerritAndre keedy proposed openstack-infra/project-config: Configure GerritBot to Announce shovel changes  https://review.openstack.org/27236703:29
*** yuanying has joined #openstack-infra03:29
*** Daisy has joined #openstack-infra03:33
*** unicell has joined #openstack-infra03:33
*** bnemec has quit IRC03:34
*** MarkAtwood_ has quit IRC03:34
*** rguillebert has quit IRC03:37
*** bnemec has joined #openstack-infra03:38
*** Daisy has quit IRC03:38
*** kzaitsev_mb has joined #openstack-infra03:39
*** binbincong has quit IRC03:39
*** amotoki_ has joined #openstack-infra03:42
*** yamahata has quit IRC03:43
*** MarkAtwo_ has joined #openstack-infra03:44
*** sdake has quit IRC03:46
*** dims has joined #openstack-infra03:49
*** dims has quit IRC03:51
*** mtanino has quit IRC03:51
*** mtanino_ has quit IRC03:51
*** binbincong has joined #openstack-infra03:53
*** harlowja has quit IRC03:54
*** EricGonczer_ has quit IRC03:54
*** dims has joined #openstack-infra03:56
*** gongysh has quit IRC03:56
*** yuanying has quit IRC03:57
*** yuanying has joined #openstack-infra03:58
*** piet has quit IRC03:59
*** yuanying has quit IRC04:02
*** zhurong has quit IRC04:03
*** zhurong has joined #openstack-infra04:04
*** flwang1 has quit IRC04:04
*** chenli has quit IRC04:05
*** dims has quit IRC04:05
*** rguillebert has joined #openstack-infra04:05
*** wei-wang has joined #openstack-infra04:05
*** yuanying has joined #openstack-infra04:08
*** amotoki has quit IRC04:08
*** fawadkhaliq has quit IRC04:12
*** amotoki_ has quit IRC04:20
*** chenli has joined #openstack-infra04:20
*** achanda has quit IRC04:23
*** achanda has joined #openstack-infra04:23
*** diana_clarke has quit IRC04:26
*** amotoki has joined #openstack-infra04:28
*** vivekd has joined #openstack-infra04:28
*** nelsnelson has joined #openstack-infra04:29
*** rlandy has quit IRC04:29
*** nelsnelson has quit IRC04:31
*** nelsnelson has joined #openstack-infra04:31
*** fawadkhaliq has joined #openstack-infra04:32
*** wei-wang has quit IRC04:33
*** zhurong has quit IRC04:33
*** achanda has quit IRC04:36
*** piet has joined #openstack-infra04:38
*** dimtruck is now known as zz_dimtruck04:38
*** diana_clarke has joined #openstack-infra04:39
*** tiswanso has quit IRC04:44
*** stevemar has quit IRC04:47
*** vivekd has quit IRC04:48
*** jpr has quit IRC04:49
*** pradk has quit IRC04:56
*** stevemar_znc has joined #openstack-infra04:57
*** yamahata has joined #openstack-infra04:57
*** baoli has quit IRC04:58
*** piet has quit IRC04:59
*** cody-somerville has quit IRC05:00
*** ayoung is now known as ayoung_ZZZzzzz05:01
*** rbrndt_ has quit IRC05:02
*** piet has joined #openstack-infra05:03
*** zhurong has joined #openstack-infra05:07
*** pradk has joined #openstack-infra05:08
*** wei-wang has joined #openstack-infra05:13
*** changzhi has joined #openstack-infra05:13
*** ansiwen has quit IRC05:16
*** unicell1 has joined #openstack-infra05:21
*** Sukhdev has joined #openstack-infra05:21
*** unicell has quit IRC05:22
*** vivekd has joined #openstack-infra05:24
openstackgerritgreghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support  https://review.openstack.org/27238405:24
*** yamamot__ has joined #openstack-infra05:25
*** yamamoto_ has quit IRC05:26
openstackgerritMasayuki Igawa proposed openstack/os-testr: Updated from global requirements  https://review.openstack.org/27238605:27
*** abregman has joined #openstack-infra05:28
*** piet has quit IRC05:30
*** kzaitsev_mb has quit IRC05:31
*** jpr has joined #openstack-infra05:31
*** maishsk has quit IRC05:39
*** zhurong has quit IRC05:41
*** zhurong has joined #openstack-infra05:42
*** Liuqing has quit IRC05:44
*** Liuqing has joined #openstack-infra05:45
openstackgerritWang Bo proposed openstack-dev/hacking: Add hacking check rule for not using xrange()  https://review.openstack.org/27220705:45
*** hdd has quit IRC05:46
openstackgerritDavid Pursehouse proposed openstack-infra/jenkins-job-builder: Allow to omit list of projects in Gerrit trigger config  https://review.openstack.org/27239205:46
*** baoli has joined #openstack-infra05:49
*** baoli has quit IRC05:54
*** ildikov has joined #openstack-infra06:02
*** mrmartin has joined #openstack-infra06:02
*** diana_clarke has quit IRC06:11
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/27239906:11
*** fawadkhaliq has quit IRC06:13
*** hichihara has quit IRC06:17
*** amotoki has quit IRC06:19
openstackgerritDavid Pursehouse proposed openstack-infra/jenkins-job-builder: Allow to omit list of projects in Gerrit trigger config  https://review.openstack.org/27239206:21
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27190206:23
*** redrobot has left #openstack-infra06:26
*** redrobot has joined #openstack-infra06:27
*** kzaitsev_mb has joined #openstack-infra06:27
*** achanda has joined #openstack-infra06:28
*** rossella_s has quit IRC06:31
*** rossella_s has joined #openstack-infra06:31
*** maishsk has joined #openstack-infra06:32
*** kzaitsev_mb has quit IRC06:32
*** pcrews has quit IRC06:35
openstackgerritSergey Kraynev proposed openstack/requirements: Bump up heatclient version to 0.9.0  https://review.openstack.org/27240506:36
*** gyee has quit IRC06:37
*** rguillebert has quit IRC06:37
*** salv-orlando has joined #openstack-infra06:38
*** MarkAtwo_ has quit IRC06:38
*** mrmartin has quit IRC06:42
*** salv-orlando has quit IRC06:42
*** maishsk has quit IRC06:45
*** _nadya_ has joined #openstack-infra06:48
prometheanfirecircular dep?  https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/openstack-repos/element-deps#L2  https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/puppet/element-deps#L206:49
*** Sukhdev has quit IRC06:49
*** zhurong has quit IRC06:50
*** rcernin has joined #openstack-infra06:53
*** zhurong has joined #openstack-infra06:54
*** salv-orlando has joined #openstack-infra06:56
*** pcrews has joined #openstack-infra06:57
*** _nadya_ has quit IRC06:58
*** vivekd has quit IRC06:58
prometheanfirealso, that dep makes me think I'll need to support the puppt element06:58
prometheanfiremordred: ^06:59
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/27239907:04
prometheanfirealso, looks like I'll need to add gentoo to bindep or somehow make a package mapping for gentoo07:05
*** Daisy has joined #openstack-infra07:06
*** Daisy has quit IRC07:07
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Remove redundant tempest job for heatclient  https://review.openstack.org/27241107:07
*** Daisy has joined #openstack-infra07:08
*** Daisy has quit IRC07:12
openstackgerritMerged openstack-infra/project-config: Revert "Reduce bluebox max_servers from 39 to 0"  https://review.openstack.org/27229307:13
*** binbincong has quit IRC07:14
AJaegerianw, yolanda: Could you review https://review.openstack.org/#/c/266565/ , please? It's a larger translation change that we need to move forward...07:14
*** ybathia has joined #openstack-infra07:16
*** aopincar has joined #openstack-infra07:17
*** vgridnev has joined #openstack-infra07:17
openstackgerritMerged openstack-infra/project-config: Add job to test neutron against neutron-lib master  https://review.openstack.org/26824007:18
*** wei-wang has quit IRC07:18
*** flepied has quit IRC07:19
*** cody-somerville has joined #openstack-infra07:19
*** fawadkhaliq has joined #openstack-infra07:20
*** Daisy has joined #openstack-infra07:22
*** [1]Thelo has joined #openstack-infra07:23
*** Thelo has quit IRC07:25
*** [1]Thelo is now known as Thelo07:25
*** binbincong has joined #openstack-infra07:26
dougwigare the unit test log formats changing? i've never seen this kind of failure before: https://review.openstack.org/#/c/272413/07:26
*** nijaba has quit IRC07:27
AJaegerdougwig: which failures?07:27
*** amotoki has joined #openstack-infra07:27
*** nijaba has joined #openstack-infra07:27
*** scheuran has joined #openstack-infra07:27
AJaegerdougwig: please wait 30 mins before a merge to project-config for zuul and jenkins to get updated...07:28
dougwigAJaeger: look at the unstable jobs in that link, and the scp failure at the end.07:28
dougwigAJaeger: oh right, that check experimental won't do anything.  good call.07:28
AJaegerdougwig: no idea what's going on there ;(07:28
*** simondodsley has quit IRC07:28
dougwigAJaeger: that's straight up python-jobs template action there.07:29
AJaegerdougwig: we have quite a few networking changes in project-config, will you find some time to go through them and give a +1 in the next days? Should I add you or one of the other liasons to them?07:29
AJaegerdougwig: yeah, failure in scp ;(07:30
dougwigAJaeger: let me look now. and you're always welcome to add me. :)07:30
*** sshnaidm has quit IRC07:30
AJaegerdougwig: while it'S not ready, this one is also for you: https://review.openstack.org/#/c/271128/07:32
*** _nadya_ has joined #openstack-infra07:32
AJaegerdougwig: thanks for reviewing!07:37
dougwigAJaeger: you beat me to correcting myself.  :)07:43
*** flepied has joined #openstack-infra07:44
*** ybathia has quit IRC07:45
AJaeger271128 needs some iterations and discussion...07:46
*** salv-orlando has quit IRC07:46
*** esikachev has joined #openstack-infra07:47
dougwigAJaeger: being up since 3am makes the -1 stick a bit heavy tonight.  :)07:48
AJaegerdougwig: Oops, so take https://review.openstack.org/259536 https://review.openstack.org/270595 another time ;)07:49
AJaegerdougwig: I let you sleep now!07:49
*** fawadkhaliq has quit IRC07:49
*** fawadkhaliq has joined #openstack-infra07:50
dougwigAJaeger: oh wow, i think not a single +1 so far.07:56
AJaegerdougwig: https://review.openstack.org/#/c/259536/ has a depends on a *merged* https://review.openstack.org/#/c/244736/07:57
AJaegerplease check again...07:57
dougwigAJaeger: whoa, they merged a neutron stadium change when the PTL removed his +1?  that doesn't look right.07:58
AJaegerdougwig: "I wish I didn't have to deal with these types of patches, and from now on, I am not going to. I let Kyle and/or Russell (or whoever is fond of the stadium concept and wants to to act as liaison) take over."07:59
AJaegerand kyle gave a +107:59
AJaegerdougwig: I'm fine waiting with 259536 if you want to discuss with armax.08:00
dougwigAJaeger: wow, ok, i don't think that's his current thinking, but that's what is in writing, so I won't hold that up while I clarify.08:00
AJaegerdougwig: let'S wait a few more days while you discuss...08:00
*** jlanoux has joined #openstack-infra08:01
*** maishsk has joined #openstack-infra08:01
AJaegerinfra-root, we have lots of UNSTABLE jobs - errors out while uploading. Anybody around that could check this, please?08:02
AJaegerMost recent: https://jenkins07.openstack.org/job/gate-bifrost-python27/137/console08:02
dougwigAJaeger: pinged armax and ttx, and noted your correct observations.08:03
yolandahi AJaeger, i can look08:03
AJaegerthanks, yolanda08:04
AJaegerthanks, dougwig !08:04
yolandamm, problems with cert..08:04
AJaegerstrange - https://jenkins03.openstack.org/job/gate-bifrost-docs/136/console succeeds and also uploaded...08:06
*** MarkAtwo_ has joined #openstack-infra08:07
yolandamm, cert is not the problem, just a red herring, i looked at upper logs and there is a problem copying08:08
yolandagoing to check static08:08
AJaegeryolanda: those run on rax - I was wondering whether my approval of https://review.openstack.org/272293 was premature and whether anything needed to be done manually there (not that I know of) but that's bluebox08:08
*** arxcruz has joined #openstack-infra08:08
yolandaAJaeger, that shouldn't cause problems, just force nodepool to stop providing nodes in bluebox08:09
AJaegeryolanda: bluebox is up again...08:10
*** pahuang has quit IRC08:10
*** lezbar__ has quit IRC08:11
AJaegerso, do we need to start nodepool there again?08:11
dougwigAJaeger: i had to go do a few non-neutron p-c reviews, to see if my +1 button still worked.08:11
AJaegerthanks, dougwig !08:11
yolandaAJaeger, i cannot find anything strange on static, disk space is fine, and perms on that directories are ok08:12
yolandado you have more samples of that failure?08:12
*** cody-somerville has quit IRC08:14
dougwigyolanda: https://review.openstack.org/#/c/272413/08:14
*** Daisy has quit IRC08:15
*** Daisy has joined #openstack-infra08:16
AJaegeryolanda: Just check the red bullets on http://status.openstack.org/zuul/08:16
*** Hal has joined #openstack-infra08:16
yolandaweird that it complains when copying, and about not finding the files, static has space and /srv/static/logs is present here08:16
*** armax has joined #openstack-infra08:16
*** Hal is now known as Guest4474208:17
AJaegeryolanda: yeah, and if you look at the bifrost jobs: 136 created the directory but 137 failed.08:17
AJaegerSo, was there an update of software and suddenly the *second* mkdir fails of the same dir fails - while it worked before?08:17
AJaegeryolanda: a different problem - no puppet updates if puppetboard is correct: http://puppetboard.openstack.org/node/jenkins01.openstack.org08:19
AJaegeroverview shows no results for 2+ hours08:20
yolandaoh dear, failures day  :(08:20
*** Daisy has quit IRC08:20
yolandai'm checking scp on jenkins first, trying to see if there is some problem, in terms of perms or keys08:20
AJaegeryolanda: sure, one issue at a time. I just wanted to mention it08:21
*** dizquierdo has joined #openstack-infra08:21
*** flepied has quit IRC08:22
*** pahuang has joined #openstack-infra08:22
*** Daisy has joined #openstack-infra08:23
*** kzaitsev_mb has joined #openstack-infra08:23
yolandaweird, i test scp connection in jenkins, and looks fine08:23
*** achanda has quit IRC08:23
*** _nadya__ has joined #openstack-infra08:24
*** Daisy has quit IRC08:24
*** shardy has joined #openstack-infra08:24
*** _nadya_ has quit IRC08:24
*** nmagnezi_ has joined #openstack-infra08:26
*** jcoufal has joined #openstack-infra08:26
yolandamkdir 27242008:27
yolandamkdir: cannot create directory `272420': Read-only file system08:27
yolandalooks as disk problems on static08:27
*** mikelk has joined #openstack-infra08:28
*** amotoki has quit IRC08:28
yolandadmesg also shows a ton of errors on disk08:28
AJaegeryolanda: so, it fails for some jobs but works for others???08:28
AJaegerOoops ;(08:29
AJaegertime to send a service notice?08:29
*** Daisy has joined #openstack-infra08:29
*** amotoki has joined #openstack-infra08:29
yolandaAJaeger, please do it yes. I'll start a disk check08:30
*** ifarkas has joined #openstack-infra08:30
AJaegeryolanda: I have no permissions ;( You have to do it08:31
yolandaAJaeger sorry, i remember that i should have perms now yes, let me check manual and i'll do it08:31
yolanda#status alert Jenkins jobs are unstable and failing to upload artifacts, due to a disk problem on static server08:32
yolandamm, that should have worked?08:33
openstackgerritGhanshyam Mann proposed openstack/requirements: Add yasfb (Sphinx extension) in global requirements  https://review.openstack.org/27072008:37
AJaegeryolanda: are you authenticated?08:37
AJaegeryolanda: I'll send an email to openstack-dev...08:39
*** jaosorior has joined #openstack-infra08:40
yolandaAJaeger, i am yes08:41
*** Liuqing has quit IRC08:41
*** dtardivel has joined #openstack-infra08:41
AJaegerstrange - but the last of our worries08:41
*** haiwei has joined #openstack-infra08:42
yolandamaybe bot is down as well08:42
*** Liuqing has joined #openstack-infra08:42
yolandai am running a check on the logs volume to try to repair08:42
*** kzaitsev_mb has quit IRC08:43
*** sshnaidm has joined #openstack-infra08:46
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Retire icehouse and juno for networking-midonet  https://review.openstack.org/27200508:47
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental rally job  https://review.openstack.org/26885908:47
*** salv-orlando has joined #openstack-infra08:47
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add experimental rally job  https://review.openstack.org/26885908:49
yolandathanks for the email, volume is unmounted, and i'm running a filesystem check, it may take a while08:49
*** armax has quit IRC08:49
*** esikachev has quit IRC08:51
electricallong live hardware :-)08:52
*** armax has joined #openstack-infra08:55
*** dizquierdo has quit IRC08:56
*** derekh has joined #openstack-infra08:56
*** fawadkhaliq has quit IRC08:57
*** fawadkhaliq has joined #openstack-infra08:57
*** BobBall_AWOL is now known as BobBall08:58
*** sshnaidm has quit IRC09:00
yolanda(it's taking ages)09:00
*** salv-orlando has quit IRC09:01
*** mriedem has joined #openstack-infra09:01
*** salv-orlando has joined #openstack-infra09:03
*** flepied has joined #openstack-infra09:05
*** hashar has joined #openstack-infra09:06
*** acabot has quit IRC09:06
*** esikachev has joined #openstack-infra09:07
*** hashar has quit IRC09:08
*** hashar has joined #openstack-infra09:09
*** lucas-dinner is now known as lucasagomes09:12
*** lezbar has joined #openstack-infra09:12
*** sshnaidm has joined #openstack-infra09:13
*** bexelbie has quit IRC09:15
*** bexelbie has joined #openstack-infra09:15
*** dtantsur|afk is now known as dtantsur09:16
*** _nadya__ has quit IRC09:18
*** ihrachys has joined #openstack-infra09:18
*** thorst has joined #openstack-infra09:18
*** jistr has joined #openstack-infra09:19
*** matrohon has joined #openstack-infra09:21
*** toabctl_ has quit IRC09:21
*** yaume has joined #openstack-infra09:22
*** Daisy has quit IRC09:22
anteaya<- nova mid-cycle09:22
*** toabctl_ has joined #openstack-infra09:22
*** markvoelker has quit IRC09:23
*** Daisy has joined #openstack-infra09:23
*** achanda has joined #openstack-infra09:24
yolandafsck in static arrived to step 2, checking directory structure...09:25
*** jaypipes has joined #openstack-infra09:25
*** keedya has quit IRC09:26
*** ihrachys has quit IRC09:26
*** arxcruz has quit IRC09:27
*** electrofelix has joined #openstack-infra09:28
*** arxcruz has joined #openstack-infra09:28
mtreinishyolanda, fungi: I'm trying to look at the logs for: https://review.openstack.org/#/c/270077/2 but the link for the layer4 tempest job is returning a not found09:29
*** chenli has quit IRC09:29
yolandamtreinish, we are having problems with disk on static09:29
mtreinishyolanda: ah, ok. Thanks09:29
yolandai had to unmount the volume and run fsck09:29
*** kaisers has quit IRC09:29
yolandai tried to send an status update but not working for me, Andreas sent a post to openstack-dev mailing list09:29
*** Liuqing has quit IRC09:29
*** sfinucan has joined #openstack-infra09:30
*** achanda has quit IRC09:30
*** Liuqing has joined #openstack-infra09:30
AJaegeryolanda: if you have time for a longer review later on, please check 26656509:32
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215609:34
*** nihilifer has quit IRC09:35
*** kaisers has joined #openstack-infra09:36
openstackgerritMerged openstack-infra/project-config: Fix zuul cloner macro for requirements repo  https://review.openstack.org/27125709:37
*** Yanyanhu has joined #openstack-infra09:37
*** nihilifer has joined #openstack-infra09:38
*** kzaitsev_mb has joined #openstack-infra09:38
esikachevhi all! i need requirements cores for merge https://review.openstack.org/#/c/270676/09:38
AJaegeresikachev: I haven't seen any of these around today besides the infra fallback09:39
*** fawadkhaliq has quit IRC09:39
openstackgerritMerged openstack-infra/project-config: puppet: make spec-helper-beaker job voting  https://review.openstack.org/27226609:39
*** fawadkhaliq has joined #openstack-infra09:39
esikachevAJaeger: okay :(09:40
openstackgerritMatthew Treinish proposed openstack/os-testr: Enable testr run passthrough arguments  https://review.openstack.org/26782409:40
*** ociuhandu has joined #openstack-infra09:41
*** jordanP has joined #openstack-infra09:42
*** thorst has quit IRC09:44
*** unicell1 has quit IRC09:44
*** kzaitsev_mb has quit IRC09:44
*** thorst has joined #openstack-infra09:44
*** ociuhandu has quit IRC09:45
*** dizquierdo has joined #openstack-infra09:46
openstackgerritMerged openstack-infra/project-config: Add new django extraction  https://review.openstack.org/26656509:47
*** dtantsur is now known as dtantsur|brb09:48
*** thorst has quit IRC09:49
openstackgerritMerged openstack-infra/project-config: magnum-ui use generic django translation processing  https://review.openstack.org/27059609:49
*** kzaitsev_mb has joined #openstack-infra09:50
*** Liuqing has quit IRC09:51
AJaegeryolanda: if you took the volume off, why are jobs succeeding? "uploading file: '/srv/static/logs/73/270373/2/gate/gate-mistral-python27/1cc659b/testr_results.html.gz'" succeeded just a few minutes ago09:53
yolandaweird, volume is not mounted09:54
*** hashar has quit IRC09:54
*** vivekd has joined #openstack-infra09:54
AJaegerAh, so it uploads content to the root disk...09:55
yolandaAJaeger, and that's uploading to static.openstack.org?09:55
AJaegeryes, see http://logs.openstack.org/73/270373/2/gate/gate-mistral-python27/1cc659b/console.html09:55
AJaegerso, you need to move content from root disk to safe place before mounting volume and then moving to volume...09:55
yolandalooks like... thanks for that warning. I see it's using root disk, yes09:56
yolandafsck is on step 5 now09:56
AJaegeryolanda: do you want to enable nodepool for bluebox again?09:57
yolandaAJaeger, i am not aware of the causes to turn it off, do you know more about it?09:58
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version  https://review.openstack.org/22978909:58
AJaegeryolanda: there was a maintenance window and they asked us to turn it  off - and the window is open again...09:58
yolandaAJaeger, then i'm ok enabling it yes09:58
*** Daisy has quit IRC10:04
*** Qiming has quit IRC10:05
*** salv-orl_ has joined #openstack-infra10:06
*** Daisy has joined #openstack-infra10:06
yolandaAJaeger, is there some change we can revert?10:06
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Move Manila LVM Tempest job run in check queue  https://review.openstack.org/27244910:07
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: Move Manila LVM Tempest job to check queue  https://review.openstack.org/27244910:08
openstackgerritDavid Pursehouse proposed openstack-infra/lodgeit: Fix minor typo in README  https://review.openstack.org/27245010:08
yolandaAJaeger, fsck finished, so i'm moving logs and will mount after it's done10:08
*** salv-orlando has quit IRC10:08
*** vivekd has quit IRC10:10
*** Daisy has quit IRC10:10
*** Daisy has joined #openstack-infra10:11
*** zhurong has quit IRC10:12
*** baoli has joined #openstack-infra10:13
*** Yanyanhu has quit IRC10:15
*** jamielennox is now known as jamielennox|away10:15
AJaegeryolanda: don't know from reading backscroll. I thought initially merging the change would be fine...10:17
AJaegerglad to hear that fsck is finished10:17
yolandai'm moving the old content now10:17
mtreinishyolanda: if you get a sec it looks like the health.o.o lost it's db connection: http://health.openstack.org/status if you could check the logs to see if there's an error10:17
mtreinishdefinitely a lower prio right now though10:17
*** baoli has quit IRC10:18
yolandayep, let's see if volume is fine now and let's fix more things :)10:18
yolandatoday is one of these days...10:18
yolandaAJaeger, move finished, and i can create directories now on that volume without errors10:19
yolandalet's see if jobs succeed10:19
*** vivekd has joined #openstack-infra10:22
yolanda[Tue Jan 26 10:16:03.280792 2016] [:error] [pid 7181:tid 140191161534208] [remote]   File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect10:23
yolanda[Tue Jan 26 10:16:03.280809 2016] [:error] [pid 7181:tid 140191161534208] [remote]     return _ConnectionFairy._checkout(self)10:23
yolanda[Tue Jan 26 10:16:03.280818 2016] [:error] [pid 7181:tid 140191161534208] [remote]   File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout10:23
yolanda[Tue Jan 26 10:16:03.280833 2016] [:error] [pid 7181:tid 140191161534208] [remote]     fairy = _ConnectionRecord.checkout(pool)10:23
yolanda[Tue Jan 26 10:16:03.280844 2016] [:error] [pid 7181:tid 140191161534208] [remote]   File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout10:23
yolanda[Tue Jan 26 10:16:03.280859 2016] [:error] [pid 7181:tid 140191161534208] [remote]     rec = pool._do_get()10:23
yolanda[Tue Jan 26 10:16:03.280868 2016] [:error] [pid 7181:tid 140191161534208] [remote]   File "/opt/openstack-health/.venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get10:23
yolanda[Tue Jan 26 10:16:03.280884 2016] [:error] [pid 7181:tid 140191161534208] [remote]     (self.size(), self.overflow(), self._timeout))10:23
yolanda[Tue Jan 26 10:16:03.280912 2016] [:error] [pid 7181:tid 140191161534208] [remote] TimeoutError: QueuePool limit of size 20 overflow 10 reached, connection timed out, timeout 3010:23
*** jaosorior has quit IRC10:23
*** markvoelker has joined #openstack-infra10:23
*** jaosorior has joined #openstack-infra10:24
*** MarkAtwo_ has quit IRC10:24
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out  https://review.openstack.org/27233110:24
yolandamtreinish, i restarted apache there, is that better?10:25
*** jamielennox|away is now known as jamielennox10:25
mtreinishyolanda: yeah that fixed it10:25
mtreinishbut, I wonder what caused the queue pool to overflow10:26
mtreinishI can probably up that limit if it happens again10:26
*** achanda has joined #openstack-infra10:27
*** yamamot__ has quit IRC10:27
*** markvoelker has quit IRC10:28
* AJaeger send an all clear by email..10:29
*** Zara is now known as Zara__10:31
*** rossella_s has quit IRC10:31
yolandagoing to check statusbot, it's weird that command didn't work for me, i should be admin10:31
*** rossella_s has joined #openstack-infra10:31
*** Daisy has quit IRC10:32
dansmithgerrit seems super slow again today.. is it time for a restart yet?10:33
*** achanda has quit IRC10:33
AJaegerLet's try #success10:33
*** ldnunes has joined #openstack-infra10:33
AJaeger#success Yolanda repaired static harddisk, so no more UNSTABLE jobs.10:34
openstackstatusAJaeger: Added success to Success page10:34
yolanda#success Volume on static is repaired10:34
openstackstatusyolanda: Added success to Success page10:34
AJaegeryolanda: so, that worked - it's the same bot but in this case it does not need permissions10:34
yolandai may miss my creds somewhere else10:35
AJaegerask jeblair or others later on for help10:36
AJaegeraccessbot/channels.yaml looks fine, you're listed10:36
AJaegeryolanda: no new unstable jobs spotted so far10:36
*** sshnaidm has quit IRC10:37
*** Zara has joined #openstack-infra10:37
yolandaAJaeger, so what about bluebox? change alreay landed,so we should have bluebox with 39 nodes now?10:37
AJaegerwe should - but we have not.10:38
AJaegerSo, is nodepool stopped there?10:39
yolandamm, looks as puppet not running10:39
yolandabecause i still see max-servers to 0 on nodepool10:39
openstackgerritAkihiro Motoki proposed openstack/requirements: Cap cliff-tablib <2.0 in stable/kilo  https://review.openstack.org/27245710:40
*** fhubik has joined #openstack-infra10:40
AJaegeryolanda: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-01-25.log.html#t2016-01-25T19:56:4910:40
yolandagoing to check puppetmaster10:40
AJaegeryolanda: otherwise let mordred and jeblair handle it later today...10:41
yolandai need to run in a while, but i'm taking a look10:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out  https://review.openstack.org/27233110:43
*** yamahata has quit IRC10:44
AJaegerodyssey4me: do you have a governance change as well for 272331?10:44
yolandalast update i see for puppet-nodepool is 22 jan10:45
*** salv-orl_ has quit IRC10:45
*** claudiub has joined #openstack-infra10:46
AJaegeryolanda: see the link from eavesdrop and scroll up a few minutes - they mentioned somethign about puppet and nodepool there...10:46
*** fhubik is now known as fhubik_brb10:47
AJaegerso, might be disabled on purpose...10:47
*** markmcd has quit IRC10:47
*** fhubik_brb is now known as fhubik10:48
*** ihrachys has joined #openstack-infra10:49
*** SotK is now known as SotK_10:49
*** markmcd has joined #openstack-infra10:50
*** bmwiedemann has left #openstack-infra10:50
yolandaAJaeger yes, it's on the list of disabled nodes10:51
yolandai can apply the change manually10:51
*** SotK has joined #openstack-infra10:51
*** sshnaidm has joined #openstack-infra10:51
*** changzhi has quit IRC10:52
*** _nadya_ has joined #openstack-infra10:54
yolandaok i updated max-servers manually on nodepool, it shall reload the config and start building on them10:55
*** eliqiao has quit IRC10:55
*** dizquierdo has quit IRC10:56
*** Qiming has joined #openstack-infra10:56
*** rfolco has joined #openstack-infra10:56
*** thorst has joined #openstack-infra10:57
*** eliqiao has joined #openstack-infra10:57
yolandai need to go now, i'll be back in the afternoon10:59
*** ansiwen has joined #openstack-infra10:59
*** dtantsur|brb is now known as dtantsur11:00
*** vivekd has quit IRC11:01
*** salv-orlando has joined #openstack-infra11:01
openstackgerritAles Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs  https://review.openstack.org/27246911:02
*** fhubik has quit IRC11:03
*** chlong has quit IRC11:05
*** kzaitsev_mb has quit IRC11:06
*** Qiming has quit IRC11:09
*** Qiming has joined #openstack-infra11:09
*** chlong has joined #openstack-infra11:10
*** kzaitsev_mb has joined #openstack-infra11:12
*** thorst has quit IRC11:13
*** zeih has joined #openstack-infra11:15
*** thorst has joined #openstack-infra11:18
*** amotoki has quit IRC11:20
*** ihrachys has quit IRC11:21
*** MCoLo has quit IRC11:24
*** ihrachys has joined #openstack-infra11:24
*** baoli has joined #openstack-infra11:25
*** xwizard_ has joined #openstack-infra11:25
*** zhurong has joined #openstack-infra11:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new project for the OSA role break out  https://review.openstack.org/27233111:26
*** hashar has joined #openstack-infra11:26
*** zeih has quit IRC11:26
odyssey4meAJaeger I've updated the review to include the link to the governance change, and added my vote. Thanks for the reminder. :)11:27
*** claudiub has quit IRC11:27
*** erlon has joined #openstack-infra11:28
*** zeih has joined #openstack-infra11:28
*** kzaitsev_mb has quit IRC11:29
*** achanda has joined #openstack-infra11:30
AJaegerthanks, yolanda ! Bluebox has nodes now..11:31
AJaegerodyssey4me: thanks11:31
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector job timeouts  https://review.openstack.org/27247811:33
*** zeih has quit IRC11:33
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version  https://review.openstack.org/22978911:33
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client  https://review.openstack.org/27247911:34
*** fawadkhaliq has quit IRC11:34
*** achanda has quit IRC11:35
*** rguillebert has joined #openstack-infra11:35
*** dizquierdo has joined #openstack-infra11:35
*** aysyd has joined #openstack-infra11:36
*** thorst has quit IRC11:37
*** zhurong has quit IRC11:40
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove timeout of ironic jobs  https://review.openstack.org/27248111:41
*** zhurong has joined #openstack-infra11:41
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector job timeouts  https://review.openstack.org/27247811:43
AJaegerdtantsur: could you remove your -1, please? It survived the rebase...11:44
*** vgridnev has quit IRC11:45
*** gildub has quit IRC11:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce ironic-inspector and ironic-functional timeouts  https://review.openstack.org/27247811:56
*** amotoki has joined #openstack-infra11:56
*** sdake has joined #openstack-infra11:58
dtantsurAJaeger, +1 thanks for rebasing11:58
AJaegerthanks for reviewing, dtantsur11:59
dtantsuryolanda, hi! if you're by chance around, could you review/approve https://review.openstack.org/#/c/264168/ please?11:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Reduce timeout of ironic jobs  https://review.openstack.org/27248111:59
dtantsurAJaeger, btw, what's the reasoning behind reducing these timeouts? the ironic patch worries me, our gate is unstable enough already, and often suffers from tempest timeouts12:01
*** mkoderer_cloud has joined #openstack-infra12:05
*** sdake_ has joined #openstack-infra12:09
openstackgerritIhar Hrachyshka proposed openstack-infra/irc-meetings: Revert to bi-weekly schedule for the neutron team meeting  https://review.openstack.org/27249412:09
*** sdake has quit IRC12:11
*** salv-orlando has quit IRC12:11
*** jaosorior has quit IRC12:12
*** salv-orlando has joined #openstack-infra12:13
*** jaosorior has joined #openstack-infra12:13
*** sshnaidm has quit IRC12:13
*** daemontool has joined #openstack-infra12:14
*** sshnaidm has joined #openstack-infra12:14
*** yamamoto has joined #openstack-infra12:15
*** baoli has quit IRC12:17
AJaegerdtantsur: we use 60 mins as normal timeout, everything above is excpetion. Seems there'S a cargo-cult to copy the highest number and keep it without analysis ;(12:19
AJaegerdtantsur: if you really need longer, let's update again because the runtime is longer...12:19
*** _amrith_ is now known as amrith12:20
AJaegerdtantsur: if you have better numbers for the timeout, I'm happy to change but my look at quite a few recent ironic ones shows that it seldom is 50 mins and never more than 60 - so 70 mins should be fine.12:21
*** doug-fish has joined #openstack-infra12:21
dtantsurAJaeger, ok fine, let's wait for jroll to show up and review it12:21
AJaegerdtantsur: sure - that's why I added him ;)12:21
*** vgridnev has joined #openstack-infra12:22
*** doug-fish has quit IRC12:23
*** yamamoto has quit IRC12:23
*** thorst has joined #openstack-infra12:24
*** doug-fish has joined #openstack-infra12:24
*** yamamoto has joined #openstack-infra12:24
*** markvoelker has joined #openstack-infra12:25
*** thorst has quit IRC12:28
*** thorst has joined #openstack-infra12:28
openstackgerritAles Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs  https://review.openstack.org/27246912:28
*** dims has joined #openstack-infra12:28
*** yamamoto has quit IRC12:28
*** doug-fish has quit IRC12:28
*** markvoelker has quit IRC12:29
*** kushal has quit IRC12:30
*** doug-fish has joined #openstack-infra12:31
*** kushal has joined #openstack-infra12:32
*** kushal has quit IRC12:34
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101112:35
*** weshay has quit IRC12:35
*** doug-fish has quit IRC12:35
*** vgridnev has quit IRC12:41
*** vgridnev has joined #openstack-infra12:42
*** Daisy has joined #openstack-infra12:43
*** dims has quit IRC12:45
*** Daisy has quit IRC12:48
*** rhallisey has joined #openstack-infra12:51
*** thorst_ has joined #openstack-infra12:52
*** pradk has quit IRC12:52
*** daemontool has quit IRC12:55
*** thorst has quit IRC12:55
*** salv-orlando has quit IRC12:57
*** baoli has joined #openstack-infra12:58
*** dprince has joined #openstack-infra12:59
*** zhurong has quit IRC13:01
*** baoli has quit IRC13:01
*** baoli has joined #openstack-infra13:01
*** zhurong has joined #openstack-infra13:01
*** fawadkhaliq has joined #openstack-infra13:02
*** bryan_att has joined #openstack-infra13:03
*** thorst_ has quit IRC13:03
lucasagomeshi all, I've just started a new project for out-of-tree ironic drivers called ironic-staging-drivers the project has been approved already and I've put the initial commit up too (https://review.openstack.org/#/c/272499/)13:03
openstackgerritAles Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs  https://review.openstack.org/27246913:03
lucasagomesbut I need core access to the project, someone can add me to the ironic-staging-drivers-{core, release} gerrit groups ?13:04
*** trown|outttypeww is now known as trown13:04
lucasagomesthank you13:04
*** baoli has quit IRC13:05
lucasagomesAJaeger, ^ is it something you can do?13:05
*** thorst has joined #openstack-infra13:06
*** gongysh has joined #openstack-infra13:06
*** betherly has joined #openstack-infra13:07
* lucasagomes brb13:07
openstackgerritMerged openstack-infra/irc-meetings: Move the Neutron drivers meeting  https://review.openstack.org/26978013:07
*** lucasagomes is now known as lucas-hungry13:07
betherlyAJaeger: hi there!13:07
betherlyAJaeger: thanks again for your patch. having a look at it now13:07
*** thorst has quit IRC13:07
betherlyso the project is not django although it does contain python. it is a plugin for horizon containing an angular based panel13:08
*** thorst has joined #openstack-infra13:11
openstackgerritEmilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: fix $LOG_DIR permissions  https://review.openstack.org/27251813:12
*** adrienverge has joined #openstack-infra13:13
*** baoli has joined #openstack-infra13:14
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add reno jobs for puppet-keystone  https://review.openstack.org/27226113:14
*** amrith is now known as _amrith_13:16
*** baoli has quit IRC13:17
*** jsavak has joined #openstack-infra13:18
*** thorst_ has joined #openstack-infra13:18
*** zhurong has quit IRC13:18
*** gordc has joined #openstack-infra13:18
*** edmondsw has joined #openstack-infra13:18
*** weshay_xchat has joined #openstack-infra13:19
*** kgiusti has joined #openstack-infra13:19
*** gongysh has quit IRC13:21
*** thorst has quit IRC13:21
*** bcourt has quit IRC13:22
*** zz_dimtruck is now known as dimtruck13:22
*** baoli has joined #openstack-infra13:22
*** smarcet has joined #openstack-infra13:23
*** fawadkhaliq has quit IRC13:25
*** markvoelker has joined #openstack-infra13:25
*** weshay_xchat is now known as weshay13:27
*** markvoelker has quit IRC13:28
*** markvoelker has joined #openstack-infra13:28
*** ociuhandu has joined #openstack-infra13:29
*** dizquierdo has quit IRC13:29
AJaegerlucas-hungry: enjoy lunch! I can't, you need one of the infra admins for that.13:29
openstackgerritMerged openstack-infra/elastic-recheck: Add query for Ironic bug 1537076  https://review.openstack.org/27149613:29
openstackbug 1537076 in Ironic "Timed out waiting for Nova hypervisor-stats count >= 1 due to Nova Unable to establish connection to" [High,Confirmed] https://launchpad.net/bugs/153707613:29
*** doug-fish has joined #openstack-infra13:30
AJaegerbetherly: I need to see how to support this properly. If you follow the spec, we aim to give a framework that all repositories can follow for translation - special casing each and every one is complex and error prone. I'm not sure how angular behaves...13:31
*** dimtruck is now known as zz_dimtruck13:32
*** achanda has joined #openstack-infra13:32
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting  https://review.openstack.org/27253813:37
openstackgerritAles Komarek proposed openstack-infra/project-config: Add openstack-salt project and jobs  https://review.openstack.org/27246913:37
*** doug-fish has quit IRC13:37
*** achanda has quit IRC13:38
*** doug-fish has joined #openstack-infra13:38
*** piet has joined #openstack-infra13:38
*** burgerk has joined #openstack-infra13:38
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting  https://review.openstack.org/27253813:41
openstackgerritEmilien Macchi proposed openstack-infra/project-config: copy_puppet_logs: use SUDO to rename config/log files  https://review.openstack.org/27251813:42
EmilienMAJaeger: sorry, I updated it to use sudo ^13:42
*** doug-fish has quit IRC13:43
*** jaypipes has quit IRC13:44
AJaegerEmilienM: ok13:44
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Set gate-designate-python34 voting  https://review.openstack.org/27253813:44
Kiallakscram: dooh, missed your comment before PS#2 .. PS#3 uses python3-jobs13:44
KiallAJaeger: *13:44
*** ldnunes_ has joined #openstack-infra13:45
betherlyAJaeger: which spec do i need to take a look at with regard the translation setup? i set the project up as per creators documentation but is there another doc im missing?13:46
*** krtaylor has quit IRC13:46
betherlyhaving taken a look at the patch i think it will work correctly for the plugins needs but i will confirm13:46
AJaegerbetherly: the one linked from the patch, it's http://specs.openstack.org/openstack-infra/infra-specs/specs/translation_setup.html13:46
betherlyAJaeger: oh my bad! awesome thanks13:47
*** terriyu has joined #openstack-infra13:47
AJaegerbetherly: the creators documentation did not fully cover translations, read the spec.13:47
* AJaeger does not consider the spec cast in stone but the base for implementation, we'll update once we learn more - so comments are welcome13:47
AJaegerbetherly: for django you *need* django.pot as translation files - not sure whether angular is different...13:48
*** ldnunes has quit IRC13:48
*** julim has joined #openstack-infra13:50
*** tlian has joined #openstack-infra13:52
esikachevi need requirements cores for merge https://review.openstack.org/#/c/270676/13:53
openstackgerritDevananda van der Veen proposed openstack-infra/elastic-recheck: Add query for Ironic bug 1537754  https://review.openstack.org/27254113:53
openstackbug 1537754 in Ironic "Unable to associate floating IP to fixed IP for instance ..." [High,Triaged] https://launchpad.net/bugs/153775413:53
*** asselin_ has joined #openstack-infra13:54
*** terriyu has quit IRC13:54
*** acabot has joined #openstack-infra13:54
*** jordanP has quit IRC13:55
devanandamtreinish: another e-r query ^13:55
*** asselin__ has quit IRC13:56
*** piet has quit IRC13:57
*** krtaylor has joined #openstack-infra13:58
*** asselin__ has joined #openstack-infra13:58
*** salv-orlando has joined #openstack-infra13:58
*** piet has joined #openstack-infra14:01
*** dkranz has joined #openstack-infra14:01
*** asselin_ has quit IRC14:01
*** asselin_ has joined #openstack-infra14:03
anteayamorning pabelanger14:03
*** salv-orlando has quit IRC14:03
*** dane-fichter_ has joined #openstack-infra14:04
rossella_shi Andreas I have a question for you. For Neutron I created a script to generate dynamically a file for gerrit-dash-creator to get a dashboard with critical bugs and patch related to approved blueprint. As said  it's dynamic so the url for gerrit changes every time.14:04
*** sriram has joined #openstack-infra14:04
rossella_sAJaeger forgot to mention your nick in the msg before14:04
dane-fichter_anteaya: hello14:05
anteayadane-fichter_: hi14:05
anteayaI can comment on patches in gerrit: https://review.openstack.org/#/c/272292/214:05
anteayathis is the sandbox repo I just commented on this patch14:05
rossella_sAJaeger, going back to the question...On my server I update the link every day but I wanted to put the link in a public place. The Neutron doc is not an option because it would require the submission of a patch and to get it approved every time the link is updated. Any ideas? Thanks!14:05
anteayahave you been able to comment on patches in gerrit in the past, dane-fichter_ ?14:05
dane-fichter_anteaya: yes.14:06
*** lucas-hungry is now known as lucasagomes14:06
lucasagomesAJaeger, thank you14:06
anteayacan you share the url of a patch you commented on dane-fichter_?14:06
lucasagomesI will look at someone that can add me14:06
rossella_santeaya, hi! you might also have some ideas14:06
*** asselin__ has quit IRC14:06
dane-fichter_anteaya: https://review.openstack.org/#/c/272071/14:06
*** asselin__ has joined #openstack-infra14:07
anteayarossella_s: actually I'm at nova mid-cycle and trying to concentrate on the discussion, sorry, just trying to help a nova dev troubleshoot their gerrit status14:07
betherlyAJaeger: thats awesome. i will look into it further and let you know on it. are you happy for me to amend the patch?14:07
rossella_santeaya, no worries, thanks for letting me know!14:07
*** asselin_ has quit IRC14:08
dane-fichter_anteaya: I've been reviewing things for about a year now. For some reason today I am not being presented with the button that lets you review things14:08
lucasagomeshi all, I've just started a new project for out-of-tree ironic drivers called ironic-staging-drivers the project has been approved already and I've put the initial commit up too (https://review.openstack.org/#/c/272499/)14:08
lucasagomesbut I need core access to the project, someone can add me to the ironic-staging-drivers-{core, release} gerrit groups ?14:08
anteayadane-fichter_: you have two gerrit accounts14:08
krotscheckjeblair: Just got your note. Sounds like we just have to wait for the rest of the slaves to recycle?14:08
*** amitgandhinz has joined #openstack-infra14:09
dane-fichter_anteaya: Ugh right. There was a weird capitalization bug.14:09
*** rajinir has joined #openstack-infra14:09
*** amotoki has quit IRC14:09
anteayadane-fichter_: you do go to the gerrit text search box and enter: "reviewer:dane" you have two accounts, they ahve different email addresses14:09
anteayadane-fichter_: what capitalization bug?14:09
dane-fichter_anteaya: It had to do with stacklytics not tracking my reviews correctly with any caps in the e-mail address I think. It was over a year ago.14:10
anteayadane-fichter_: my sense of things is that you have two gerrit accounts each with their own gerrit id14:11
*** ansiwen has quit IRC14:11
*** jordanP has joined #openstack-infra14:11
dane-fichter_anteaya: Yeah I can see why that'd be a problem. Can we delete the unused one?14:11
anteayadane-fichter_: no, you can't delete gerrit accounts, what we would have to do is figure out which one you want to keep and set the other one to disabled14:12
*** ansiwen has joined #openstack-infra14:12
*** tiswanso has joined #openstack-infra14:13
anteayaas well as acertain what your launchpad id is since if we don't access that gerrit will just create a new gerrit account if it things it is a new launchpad id14:13
anteayadane-fichter_: fungi usually takes care of this issue of duplicate gerrit accounts but is away this week, perhaps once jeblair is online you can explain the situation and gently ask for some of his time to assist you14:14
anteayathis is low priority and we are on reduced infra-roots this week14:14
anteayadane-fichter_: I counsel patience14:14
*** rlandy has joined #openstack-infra14:14
anteayadane-fichter_: go into your gerrit settings, what is the gerrit id for the account you are signed into right now?14:15
*** ansiwen has quit IRC14:16
anteayarossella_s: I hope you get some help for your thing, ping me in a few days if you still need assistance14:16
dane-fichter_anteaya: dane-fichter14:16
*** ansiwen has joined #openstack-infra14:16
anteayadane-fichter_: that is your username, gerrit id is a number14:16
dane-fichter_anteaya: the account ID is 1552414:16
rossella_santeaya, thanks a lot! you are always very helpful. Enjoy the mid-cycle!14:16
anteayadane-fichter_: thank you, stay in this channel, if jeblair has time he will ping you if he has more questions14:16
anteayarossella_s: thanks for understanding :)14:17
*** pblaho has quit IRC14:17
dane-fichter_anteaya: thanks!14:17
anteayadane-fichter_: welcome14:17
dane-fichter_anteaya: Seriously y'all are super helpful. Thanks.14:17
anteayaglad to help, thanks for the appreciation14:18
anteayavery grateful14:18
*** pblaho has joined #openstack-infra14:18
anteayajeblair: priority, graphite seems to have stopped reporting14:19
anteayainfra-root ^^14:19
*** sabeen1 has joined #openstack-infra14:20
*** bhunter71 has left #openstack-infra14:21
anteayadid graphite perhaps get adversly affected when yolanda dealt with the logging issue of a few hours ago?14:23
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Add possibility to use SSH instead of HTTP to get ref list  https://review.openstack.org/24643314:23
AJaegerbetherly: sure, take over my change - and tell me about anything you find. Thanks!14:24
*** daemontool has joined #openstack-infra14:24
AJaegerrossella_s: I'M here now..14:24
betherlyAJaeger: thanks and will do14:25
*** doug-fish has joined #openstack-infra14:26
AJaegerrossella_s: what'S the content of the script? How do you update it? Manually add bugs etc?14:26
*** doug-fish has quit IRC14:26
*** dims has joined #openstack-infra14:27
rossella_sAJaeger, this is the script https://github.com/openstack/neutron/blob/master/tools/milestone-review-dash.py14:27
*** doug-fish has joined #openstack-infra14:27
rossella_sAJaeger, you just run it passing the milestone you want to track, then it writes a .dash file that can be used by gerrit-dash-creator to produce the url14:28
*** sabeen3 has joined #openstack-infra14:28
AJaegerso, this searches bugs in launchpad and generates the review from it?14:28
*** peristeri has joined #openstack-infra14:29
rossella_sAJaeger, yes14:29
*** dims has quit IRC14:29
*** sabeen1 has quit IRC14:30
rossella_sAJaeger, I have a cron job right now on my server that generates the url daily and publish it on my blog. I want to publish it in a public place...without having to create a gerrit review every day :)14:30
AJaegerseveral options in general - but I'm not sure which one is best, this would be for the broader team to discus:14:30
AJaegera periodic job (that's like cron) that publishes a document that includes the URL14:30
rossella_sAJaeger, but then somebody should approve it right?14:31
*** rossella_s has quit IRC14:31
*** abregman is now known as abregman|mtg14:31
*** tiswanso has quit IRC14:31
*** rossella_s has joined #openstack-infra14:31
AJaegerwe normally publish docs from the post job whenever something changes - so that would mean a change14:32
*** tiswanso has joined #openstack-infra14:32
AJaegerBut a periodic queue could do it as well - but we have no such job, so that's why it needs broader discussion.14:32
*** dims has joined #openstack-infra14:32
AJaegerSince most infra folks are travelling this week, what about sending an email to openstack-infra?14:33
rossella_sAJaeger, ok, thanks for your help!14:33
*** mkoderer_cloud has quit IRC14:34
AJaegeryou're welcome - and if you need help further on, please ask14:34
*** pradk has joined #openstack-infra14:34
*** rajinir has quit IRC14:34
*** boris-42 has quit IRC14:34
*** roaet_ has quit IRC14:35
*** iremizov has quit IRC14:35
*** pblaho has quit IRC14:35
*** boris-42 has joined #openstack-infra14:35
*** davidlenwell has quit IRC14:35
*** rajinir has joined #openstack-infra14:36
*** iremizov has joined #openstack-infra14:36
*** mkoderer_cloud has joined #openstack-infra14:36
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27190214:37
AJaegerdims: no sense in rechecking the argparse removal yet - still too many repos have it ;(14:37
*** ansiwen has quit IRC14:37
dimsAJaeger : ack :( just hoping against hope14:38
*** ansiwen has joined #openstack-infra14:38
AJaegerdims: check https://review.openstack.org/#/q/status:open++branch:master+topic:rm-argparse - it still contains several python clients and dib which is blocked by ironic... - once that list does not contain entries from projects.txt, you should be fine AFAIK14:38
*** ansiwen has quit IRC14:39
*** dmsimard is now known as dmsimard|afk14:39
*** pblaho has joined #openstack-infra14:39
*** roaet_ has joined #openstack-infra14:41
*** zz_dimtruck is now known as dimtruck14:42
*** _amrith_ is now known as amrith14:43
*** daemontool has quit IRC14:43
*** coreycb` is now known as coreycb14:44
*** daemontool has joined #openstack-infra14:47
*** asselin_ has joined #openstack-infra14:47
AJaegerdims, feel free to ping folks ;)14:47
*** asselin__ has quit IRC14:48
openstackgerritRussell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl.  https://review.openstack.org/26917414:49
*** fawadkhaliq has joined #openstack-infra14:49
*** jlanoux has quit IRC14:49
*** davidlenwell has joined #openstack-infra14:49
*** jlanoux has joined #openstack-infra14:51
jlkjeblair: mordred: how has the Blue Box nodepool been performing today?14:51
anteayajlk: can't tell, graphite is down14:52
jlkso I noticed14:52
anteayathere seemed to be a traffic spike for graphite and it isn't reporting now14:52
*** Qiming has quit IRC14:53
*** jpr has quit IRC14:53
anteayajlk: prior to that it looks like bluebox had a spike (which had errors) but other than that looked error free14:53
jlkokay. Mostly I was curious if build times and run times got "faster" perhaps. The change out for underlying instance storage was kind of a big one14:54
anteayajlk: I understand14:54
*** regXboi has joined #openstack-infra14:54
*** salv-orlando has joined #openstack-infra14:54
anteayaand would like to get you some data14:54
*** jaypipes has joined #openstack-infra14:55
anteayaunfortunately I have no data to share14:55
anteayaI'm sorry14:55
jlkheh, no worries14:55
jlkit's not urgent14:55
anteayajlk: hopefully when jeblair is online and after he gets graphite up again14:56
anteayaperhaps he might have some numbers14:56
anteayajlk: thanks for asking14:56
anteayaand thanks for doing the work, woooo!14:56
*** salv-orlando has quit IRC14:57
*** jlanoux has quit IRC14:57
*** jlanoux has joined #openstack-infra14:58
*** ansiwen has joined #openstack-infra14:58
*** piet has quit IRC14:59
*** angdraug has joined #openstack-infra15:00
*** ilyashakhat has joined #openstack-infra15:00
*** placko_ has joined #openstack-infra15:01
*** dizquierdo has joined #openstack-infra15:01
openstackgerritFlavio Percoco proposed openstack-infra/project-config: Dismiss glance-drivers team  https://review.openstack.org/27257315:01
openstackgerritMerged openstack-infra/elastic-recheck: Add query for Ironic bug 1537754  https://review.openstack.org/27254115:01
openstackbug 1537754 in Ironic "Unable to associate floating IP to fixed IP for instance ..." [High,Triaged] https://launchpad.net/bugs/153775415:01
*** rubasov has joined #openstack-infra15:01
*** alivigni has joined #openstack-infra15:02
*** pblaho has quit IRC15:03
*** EricGonczer_ has joined #openstack-infra15:04
*** EricGonc_ has joined #openstack-infra15:06
*** jlanoux has quit IRC15:06
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version  https://review.openstack.org/22978915:07
*** amotoki has joined #openstack-infra15:08
*** EricGonczer_ has quit IRC15:09
*** jlanoux has joined #openstack-infra15:09
*** salv-orlando has joined #openstack-infra15:10
rubasovhello there15:10
rubasovdid I come to the proper channel if I have a gerrit nuisance?15:11
*** salv-orlando has quit IRC15:11
anteayawhat is your situation?15:11
*** sigmavirus24_awa is now known as sigmavirus2415:11
anteayagerrit questions are answered in this channel, yes15:11
rubasovmy user can't be added to reviews15:11
*** salv-orlando has joined #openstack-infra15:11
rubasovlike this15:11
rubasovBence Romsics <bence.romsics@ericsson.com> does not identify a registered user or group15:11
rubasovI have found this bug15:11
openstackLaunchpad bug 1394191 in OpenStack Core Infrastructure "can't be added as a gerrit reviewer " [Medium,Fix released] - Assigned to Jeremy Stanley (fungi)15:11
rubasovsounds like just my problem15:12
anteayarubasov: you have two accounts15:12
*** esker has quit IRC15:12
rubasovyes, I don't use one of them15:12
rubasovcan I delete one?15:12
anteayaAs discussed in #openstack-infra on IRC, this was because you tried authenticating with a second Launchpad OpenID account, causing Gerrit to generate conflicting accounts for the same E-mail address. At your request I have disabled the second account in Gerrit.15:12
*** esker has joined #openstack-infra15:13
anteayano, an infra-root with gerrit permissions has to do that15:13
*** piet has joined #openstack-infra15:13
anteayarubasov: what is the gerrit id from your settings for the account you are currently in?15:13
rubasova moment15:13
*** esker has quit IRC15:14
rubasovthis is the one I'm actually using:15:14
*** esker has joined #openstack-infra15:14
rubasovAccount ID: 1555415:14
anteayathank you15:14
anteayaso this is the account you wish to keep, the one you are in now?15:14
*** tiswanso has quit IRC15:15
*** jlanoux has quit IRC15:15
anteayathank you, when a gerrit admin is online hopefully they will read this information in backscroll and hopefully have time to help15:15
anteayathis issue is low priority, so stay in channel and have patience15:15
*** tiswanso has joined #openstack-infra15:15
*** tiswanso has quit IRC15:15
rubasovanteaya: thanks a lot for your help15:15
rubasovwill stay and wait15:15
anteayathanks in advance for your patience15:16
anteayathank you15:16
rubasovthank you, have a nice day15:16
anteayahighest priorty right now is getting graphite reporting again15:16
anteayarubasov: and you15:16
*** jlanoux has joined #openstack-infra15:16
pabelangerany project-config-core mind helping 264994 land? Adds templating support to one of our dashboards15:18
yolandahi, back... anteaya, is there still the problem with graphite?15:19
krotscheckmordred: You around? Do you know what the command is for "Execute things with these afs credentials?"15:19
anteayayolanda: thanks, can you take a look at graphite? it isn't reporting15:19
* krotscheck is patch digging15:19
anteayayolanda: it looks like it had a traffic spike and then the service stopped shortly thereafter15:20
anteayayolanda: I'm not sure if it is related to the issue you had with static.o.o or not15:20
krotscheckmordred: Ah, nevermind, found it15:20
yolandaanteaya, i don't think it could be related, as the problem with static was a disk corruption on a volume, but i'll look at graphite logs15:21
anteayayolanda: thanks15:21
lucasagomesyolanda, mordred  you are infra-admin right? Do you mind adding myself to the ironic-staging-drivers-core gerrit group? The ironic-staging-project got approved already but the core gerrit group is empty15:21
lucasagomesso I can't approve anything there :-)15:21
*** mtanino has joined #openstack-infra15:21
*** nmagnezi_ has quit IRC15:22
*** mtanino_ has joined #openstack-infra15:22
*** zhurong has joined #openstack-infra15:22
*** esker has quit IRC15:22
*** mtanino__ has joined #openstack-infra15:23
*** ajmiller has joined #openstack-infra15:24
*** pblaho has joined #openstack-infra15:24
*** zhurong has quit IRC15:25
*** rbrndt_ has joined #openstack-infra15:26
*** david-lyle_ has joined #openstack-infra15:26
*** simondodsley has joined #openstack-infra15:26
openstackgerritAlexander Evseev proposed openstack-infra/zuul: Allow using numeric change number in addition to ID in Depends-On tag  https://review.openstack.org/21555315:27
*** tonytan4ever has joined #openstack-infra15:28
*** burgerk has quit IRC15:28
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Allow builder.py to start without gearman servers  https://review.openstack.org/27259015:29
pabelangergreghaynes: clarkb: ^ minor change to nodepool-builder15:29
anteayapabelanger: clarkb is away for the week15:29
pabelangeranteaya: thanks15:29
*** yamahata has joined #openstack-infra15:30
*** tiswanso has joined #openstack-infra15:30
*** MarkAtwo_ has joined #openstack-infra15:30
*** jlanoux has quit IRC15:30
yolandaanteaya, i restarted carbon-cache and statsd, and restarted apache as well. Let's see if that helps15:31
yolandaoh dear, disk corruption also there!15:32
anteayayolanda: did the logs show any useful hints about why it quit?15:32
yolandaexceptions.IOError: [Errno 30] Read-only file system: '/var/lib/graphite/storage/whisper/stats/haproxy/balance_git_daemon/git06/openstack/org/eresp.wsp'15:32
anteayaoh no :(15:32
yolandamaybe rax had some power outage?15:32
*** jlanoux has joined #openstack-infra15:32
anteayayolanda: can you pm the server uuids to mikal  please?15:33
mordredyolanda: I do not know enough about recovering from that to be useful15:33
yolandagoing to unmount the volume and proceed with an fsck as i did on static15:33
anteayamordred: morning15:33
mordredyolanda: did we have some disk corruption?15:33
openstackgerritMerged openstack-infra/project-config: Switch to IPA as a primary ramdisk for ironic-inspector jobs  https://review.openstack.org/26416815:33
yolandamordred, we had on static this morning, on /srv/static/logs15:33
yolandaand now i'm seeing the same on graphite15:33
mordredthat's fantastic15:34
anteayayolanda: mikal is with rax and sitting here beside me at the nova mid-cycle, he will try to get some information on our servers if he has the uuids15:34
*** isaacb has joined #openstack-infra15:34
mordredyolanda: I'm mostly cheering from the side - but I can get uuids to mikal from here15:34
*** salv-orlando has quit IRC15:34
*** nmagnezi_ has joined #openstack-infra15:34
yolandathanks mordred15:34
*** placko_ has quit IRC15:34
anteayaI'm not sure if that should be private or not so defaulting to private15:34
anteayaapologies if I'm wrong15:34
yolandai'll work on the volume fix15:35
anteayamordred: awesome, thank you15:35
*** salv-orlando has joined #openstack-infra15:35
mikalIs the disk corruption on volumes or ephemeral disk?15:35
mordredI might be lying15:35
mikalOk, so volume ids as well would be good then15:35
mordredbut I'm pretty sure volumes15:35
*** sridhar_ram has joined #openstack-infra15:35
*** abregman|mtg is now known as abregman15:35
*** achanda has joined #openstack-infra15:36
*** ajmiller has quit IRC15:37
*** sdake_ is now known as sdake15:38
jrollmordred: is this dfw15:38
*** ajmiller has joined #openstack-infra15:38
yolandaok, fsck running on graphite15:38
jrolland volumes15:38
yolandamordred, can you send some status notice? i tried this morning but for some reason i still don't have the powers15:38
*** jpr has joined #openstack-infra15:38
jrollmordred: if so, https://status.rackspace.com/index/viewincidents?group=21&start=1453784400 could have cut iscsi connections15:38
jrollmikal: ^15:39
anteayamordred: please send an email to mikal and johnthetubaguy including a description of the issue on the static.o.o server and the graphite server including server uuids and cinder volume ids15:39
mordredanteaya: I have private messaged mikal uuids15:39
anteayamordred: they will find someone to forward that email to in order to escalate15:39
mordredbut I will also send an email15:39
jrollanteaya: ^ see that link if this is with cinder volumes in dfw15:39
jrollcould be related15:39
mordredjroll: it is15:40
anteayamordred: thank you, that is the latest request15:40
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here  https://review.openstack.org/11101115:40
mordredanteaya: I think we're likely fine once yolanda fscks things - although we might need to do the other servers with volumes too, depending on how they are15:40
anteayajroll: thank you15:40
anteayamordred: very good15:40
anteayamikal is eating a piece of cake in support of the work to fix things15:41
*** achanda has quit IRC15:41
anteayago yolanda15:41
* anteaya also will wander to the dessert table15:41
mordredelasticsearch, afs01.dfw, mirror.dfw, and nodpool all also have volumes15:41
yolandayep, fsck running but it will be slow15:41
mordredin dfw15:41
anteayayolanda: thank you15:41
*** armax has quit IRC15:42
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_libvirt job  https://review.openstack.org/26959215:42
anteayaI think nodepool is the priority15:42
mordredyolanda: do you have bandwidth to see if the filesystems on the other servers are in good or bad state?15:42
anteaya<- cake15:42
mtreinishmordred: if you get a sec can you take a look at: https://review.openstack.org/23444715:43
yolandamordred, the ones you listed?15:43
*** HeOS has quit IRC15:43
jeblairyolanda, mordred, anteaya: o/15:43
mordredmtreinish: WOW - devstack is emitting subunit?15:44
jeblairi get invalid issuer on https://status.rackspace.com/index/viewincidents?group=21&start=145378440015:44
mordredjeblair: yay! more help for yolanda that is not on an aeroplane15:44
mtreinishmordred: yep, I added that not too long ago15:44
mordredmtreinish: neat15:44
*** jlanoux_ has joined #openstack-infra15:44
*** jlanoux has quit IRC15:44
mtreinishI was thinking we might want d-g to emit subunit in the future too, but I'm not sure15:44
jeblairmordred, yolanda: cinder vl on afs01 looks ok15:47
*** zeih has joined #openstack-infra15:47
yolandai'm checking elasticsearch ones, they look fine15:48
jeblairmordred, yolanda: mirror.dfw is fine15:49
jeblairnodepool is not file15:49
yolandai remounted volume on graphite after the fsck15:49
mordredyolanda: ossum15:49
jeblairyolanda: have you started any repair work on nodepool?15:49
yolandagraphite is showing things now :)15:49
yolandajeblair, no15:50
*** placko_ has joined #openstack-infra15:50
jeblairyolanda, mordred: i will start work on nodepool.o.o15:50
*** gokrokve has joined #openstack-infra15:50
* mordred will watch yolanda and jeblair perform useful work15:50
sc68calmordred: if you can afford a context switch, could you take a look at https://review.openstack.org/#/c/267847/4/devstack-vm-gate.sh ?15:51
*** dimtruck is now known as zz_dimtruck15:52
ansiwenmordred, lifeless: I'm thinking of keeping the behaviour if the domain config is set. That is, if domain=foo is set, only that domain is compiled. If you want to enable automatic discovery, you have to leave it unset. what do you think? Aren't we on the safer side then?15:52
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711715:53
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492715:53
krotscheckmordred: You can be useful over here ^^ :)15:53
jeblairi'm moving to #openstack-infra-incident15:53
*** shardy_ has joined #openstack-infra15:53
*** Daisy has joined #openstack-infra15:53
*** shardy has quit IRC15:54
openstackgerritDoug Hellmann proposed openstack/requirements: update to oslo.config 3.4.0  https://review.openstack.org/27260715:54
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Add wheel mirror to configure_mirror.sh  https://review.openstack.org/26711715:56
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492715:56
*** piet has quit IRC15:58
*** ihrachys has quit IRC15:59
*** jlanoux has joined #openstack-infra16:00
*** ihrachys has joined #openstack-infra16:00
*** maishsk has quit IRC16:00
adrienvergemordred: Hi! About 271844 and 271845: the YAML linter is configurable, I added the option so it gives the same as `pyyaml.dump()`. It can even be made less strict.16:01
adrienvergeWould you have a new fresh look? https://review.openstack.org/#/c/271844/16:01
adrienvergeI'm convinced there's a real value to check our yaml like our python. It would prevent both CI-breaking syntax errors and useless (but real) commits like b005206 Fix trailing whitespace, 3e78b33 Check indent etc.16:01
*** jsavak has quit IRC16:01
*** jlanoux_ has quit IRC16:01
*** armax has joined #openstack-infra16:01
*** isaacb has quit IRC16:01
openstackgerritTim Birkett proposed openstack-infra/jenkins-job-builder: Add RabbitMQ properties, trigger adn publisher  https://review.openstack.org/27261116:02
*** placko_ has quit IRC16:04
amotokiAJaeger: hi16:04
*** ilyashakhat has quit IRC16:05
*** jsavak has joined #openstack-infra16:05
*** salv-orl_ has joined #openstack-infra16:06
*** ilyashakhat has joined #openstack-infra16:07
*** vgridnev has quit IRC16:08
*** mgould has joined #openstack-infra16:08
dhellmannrequirements cores, can I get a review on https://review.openstack.org/#/c/272607/ to update our oslo.config, please?16:08
*** salv-orlando has quit IRC16:09
*** vgridnev has joined #openstack-infra16:09
*** burgerk has joined #openstack-infra16:10
*** r-daneel has joined #openstack-infra16:10
openstackgerritAlex Schultz proposed openstack-infra/project-config: Add fuel-plugin-etckeeper repository for Fuel  https://review.openstack.org/26020816:11
*** jcoufal_ has joined #openstack-infra16:11
*** jcoufal has quit IRC16:12
*** amotoki has quit IRC16:16
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215616:16
*** Daisy has quit IRC16:16
mgouldhi everyone!16:16
prometheanfirecan I get a release cut for https://github.com/openstack-infra/glean/releases16:16
prometheanfireso that the simple-init element works on gentoo :P16:17
*** baoli has quit IRC16:17
mgouldmy patch is failing on check-osc-plugins because it's trying to build python-tuskarclient, which is now empty16:17
mgouldthe patch to check-osc-plugins to stop it doing that was merged at 1900UTC last night16:17
mgouldis there any further step required to deploy that code to Jenkins workers, or do I just have to wait?16:18
*** baoli has joined #openstack-infra16:18
*** Daisy has joined #openstack-infra16:18
*** david-lyle_ has quit IRC16:18
anteayadhellmann: I think you are looking at dims, lifeless and markmcclain16:19
*** fawadkhaliq has quit IRC16:20
*** unicell has joined #openstack-infra16:21
prometheanfirewho handles the releases for glean?16:21
anteayain gerrit, when you are signed in, people > list groups > use the search field16:22
anteaya<project>-release is the group that you are looking for16:22
dimsdhellmann : +2 for now, please feel free to +A when jobs finish16:22
dhellmanndims : ty16:22
*** tongli has joined #openstack-infra16:22
*** Daisy has quit IRC16:23
openstackgerritMerged openstack-infra/project-config: copy_puppet_logs: use SUDO to rename config/log files  https://review.openstack.org/27251816:23
dhellmannanteaya : thanks, I think some of the rest of infra have +2 there as well and I thought dims was unavailable today :-)16:23
prometheanfireanteaya: thanks16:23
anteayadhellmann: ah yeah, infra-core is basically down to jeblair mordred and yolanda right now16:24
*** dims has quit IRC16:24
anteayaprometheanfire: welcome16:24
dhellmannanteaya : what did you do with the rest of them? ;-)16:24
openstackgerritMerged openstack-infra/project-config: Set gate-designate-python34 voting  https://review.openstack.org/27253816:24
jeblairmordred is on a plane16:25
anteayadhellmann: as other folks are doing other things as far as I know16:25
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281416:25
jeblairand i think yolanda is about to go to sleep16:25
anteayajeblair: ah yes16:25
openstackgerritMerged openstack-infra/project-config: Added new project for the OSA role break out  https://review.openstack.org/27233116:25
*** aopincar has quit IRC16:25
anteayaso jeblair is doing all the things16:25
anteayathanks jeblair16:25
dhellmannjeblair , anteaya: that's much less alarming than I originally interpreted "down to"16:25
yolandajeblair, not yet :)16:25
yolanda5pm is not a good time to sleep16:25
openstackgerritMerged openstack-infra/project-config: Move Manila LVM Tempest job to check queue  https://review.openstack.org/27244916:25
anteayadhellmann: sorry, I chose words poorly16:26
dhellmannanteaya : I'm just teasing you16:26
anteayadhellmann:  :)16:26
prometheanfireok, guess I need to wait for one of the infra-core people16:26
anteayayolanda: feel like addressing the two folks with double gerrit accounts?16:26
anteayayolanda: in backscroll?16:26
anteayaunless jeblair wants them16:26
jeblairthere are some nodepool images that have been stuck in building for 2 days, i'm doing to delete their db records16:26
*** jlanoux has quit IRC16:27
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281416:27
anteayajeblair: ack16:27
jeblairgreghaynes, clarkb: ^16:27
yolandaanteaya, i am with some other non-infra things, but i can take a look in a while16:27
anteayayolanda: thanks16:27
anteayayolanda: thanks for your help with the file system issues16:27
*** abregman is now known as abregman|brb16:28
*** jlanoux has joined #openstack-infra16:28
anteayanice work on static.o.o16:28
yolandanp... fun today :)16:28
*** chlong is now known as chlong_zzz16:28
openstackgerritMerged openstack-infra/project-config: acl cleanup for openstack-chef project  https://review.openstack.org/26684316:28
prometheanfireyolanda: if you have time can you cut a release for glean?16:29
yolandaprometheanfire, i'd better wait for mordred or greghaynes for that16:29
yolandai'm not much related with glean, apart from contributing16:29
*** isaacb has joined #openstack-infra16:30
anteayajeblair: zuul memory usage is 10G of a 30G capacity, looking stable atm16:30
jeblairanteaya: thanks; if it climbs again, i will restart it without the connections patchs in order to collect more data16:30
markmcclaindhellmann: +2/A16:30
dhellmannmarkmcclain : thanks!16:31
anteayajeblair: very good16:31
anteayajeblair: jlk was asking about bluebox numbers if you have any space on your list16:31
anteayajeblair: he was wondering if we see performance changes16:31
jeblairanteaya: we'll have to wait until the graphs at the bottom of http://grafana.openstack.org/dashboard/db/nodepool-blue-box populate again16:32
* anteaya nods16:32
anteayajeblair: also wondering if you feel it is worthwhile to take a peek at java melody16:32
anteayabut not if you have something more important16:32
jeblairanteaya: i'm going to focus on nodepool16:33
* anteaya nods16:33
anteayasorry to interrupt16:33
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215616:33
*** abregman|brb is now known as abregman16:34
*** abregman has quit IRC16:34
*** jflynn has joined #openstack-infra16:34
*** zhurong has joined #openstack-infra16:35
greghaynesjeblair: someone else reported seeing fs corruption yesterday but I couldn't pin it down to being something unspecific to them16:35
greghaynesjeblair: so, thats scary16:35
*** woodster_ has joined #openstack-infra16:36
greghaynesthey were seeing dib delete /dev nodes on the build host16:36
jeblairgreghaynes: if it was in a job, those should be all local access, not iscsi16:36
jeblairbut who knows :)16:36
*** e0ne has joined #openstack-infra16:37
*** scheuran has quit IRC16:37
*** stevemar_znc is now known as stevemar16:38
*** dtantsur is now known as dtantsur|afk16:39
greghaynesjeblair: It was someone running nodepool as a third party ci16:39
lucasagomeshi all, sorry for the noob question... I've started a new project called ironic-staging-drivers, I got added to the gerrit ironic-staging-drivers-core group but I still have no -2...+2 powers16:39
*** ilyashakhat has quit IRC16:39
lucasagomeslooked at the acl file in project-config it seems good16:39
lucasagomesthere's anything I'm missing here?16:39
greghaynesjpr: ^16:40
greghaynesjpr: You were seeing the /dev device deletion16:40
openstackgerritTim Birkett proposed openstack-infra/jenkins-job-builder: Remove unused variable in rabbitmq trigger  https://review.openstack.org/27263116:40
prometheanfiregreghaynes: hi :D16:40
greghaynesprometheanfire: hello16:40
anteayalucasagomes: you are in the core group, and the release team is also in the core group and you are in the release team16:41
jeblairgreghaynes: there are 20 images that have not been uploaded to hpcloud more recently than 11 days ago16:41
lucasagomesanteaya, that's correct16:41
jprgreghaynes: yes.  I think I just found something that could lead to different behavior of disk-image-create under nodepoold vs direct from the command line16:41
lucasagomesanteaya, https://review.openstack.org/#/admin/groups/1258,members16:41
anteayaI'm wondering if part of the issue is you are in both groups, so essentially listed twice in -core16:41
prometheanfiregreghaynes: release glean please :D16:41
lucasagomesanteaya, https://review.openstack.org/#/admin/groups/1259,members16:41
anteayalucasagomes: I haven't seen that before16:41
prometheanfiregreghaynes: can't use it if it's not tagged I think16:42
lucasagomesanteaya, oh... I can try to remove the release team from the core16:42
prometheanfirethe simple-init element16:42
anteayaor remove yourself the individual from -core16:42
anteayaleaveing yourself as the release team as a member of -core16:42
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281416:43
*** zhurong has quit IRC16:43
jeblairgreghaynes: so i will upload all 20 of those manually16:43
greghaynesjeblair: :( I bet that is the same disconnext issue16:43
*** zhurong has joined #openstack-infra16:43
greghaynesjeblair: I am coding the disconnect handling stuff right now so hopefully itll be better soon16:43
lucasagomesanteaya, no dice... lemme try to log out and in again16:43
greghaynesprometheanfire: I don't have release perms on glean16:44
lucasagomesanteaya, :-/ no16:44
*** diazjf has joined #openstack-infra16:44
greghaynesprometheanfire: itll have to be mordred16:44
jprgreghaynes:  the disk-image-create script sets up it's env based on the path to the script.  nodepoold calls it with /usr/local/bin/disk-image-create which causes it to set using supporting scripts out of /usr/local/share/diskimage-builder .  When I've called it manually under an ordinary user account I've not specified the path, so it defaults to my cwd16:44
lucasagomesanteaya, the ACL looks fine right? https://github.com/openstack-infra/project-config/blob/master/gerrit/acls/openstack/ironic-staging-drivers.config16:44
prometheanfiregreghaynes: k, thanks :D16:44
*** apoorvad has joined #openstack-infra16:45
jeblairgreghaynes: ok.  it will be good to at least log it, because we also can't actually function over the long run if we have disconnects16:45
greghaynesjpr: Yep, I really dont know how that would cause dib to break out of the dib chroot though16:45
anteayalucasagomes: acl file looks like I would expect16:45
jeblairgreghaynes: i mean, we should handle it if it happens, but then, after that, we need to make it stop happening, because we'll never get any images built16:45
anteayalucasagomes: okay so only person with gerrit admin permisisons online right now is jeblair and he is kind of busy16:46
jprgreghaynes: haven't dug into the scripts in that dir but it seems to be the difference between success and failure, even for something as simple as the elements "ubuntu vm"16:46
lucasagomesanteaya, yeah I've no idea then :-/16:46
greghaynesjeblair: yep. The beset theory on why it is happening is that we now blast all the image-upload jobs at once at gearman16:46
anteayalucasagomes: stand by, grab a tea, perhaps work on something else for a bit16:46
lucasagomesanteaya, yeah doing that, no worries heh16:46
greghaynesjeblair: and that seems to just about coincide with when the disconnect occurs16:46
anteayalucasagomes: neither do I, I think I need a gerrit admin to look and noone is availble who has time atm16:47
lucasagomesjeblair, when you have time (today or tomorrow) mind taking a look at this problem please? Thank you16:47
anteayalucasagomes: thanks for your patience16:47
*** nmagnezi_ has quit IRC16:47
dtroyerquick LP/Gerrit question:  a dev who used their employer's email for their account is changing jobs, and has a number of in-flight reviews.  Is there something that should be done before giving up the email access when they change employers?  (no, this is not me).16:47
lucasagomesanteaya, it's all good... thanks for the help16:47
greghaynesjpr: ok, so in theory if I just pip install diskimage-builder and then run from /usr/local/bin/disk-image-create I should see the same issue...16:47
*** baoli has quit IRC16:47
anteayadtroyer: they need to add the email address to gerrit16:47
anteayadtroyer: the non-employeer email16:47
ansiwenmordred: can I mock setup.cfg in the pbr tests? I just see examples that call setup.py as a normal command line tool.16:48
jeblairgreghaynes: interesting.  the number of jobs should not be at issue (other systems send far more), but it's good to have that as something suspicious for us to focus on16:48
dtroyeranteaya: to their existing account?16:48
greghaynesjeblair: so my suspicion there was it is something related to send/recv buffers filling on that socket, but my glancing over the gear code didnt reveal anything obvious16:48
anteayadtroyer: yes16:48
jprgreghaynes:  yes.16:48
anteayato their existing gerrit account16:48
*** mgould has quit IRC16:48
dtroyerawesome.  Thanks!16:48
anteayadtroyer: keep the same launchpad creds16:48
anteayadtroyer: don't add the new email to launchpad, just to gerrit16:48
*** achanda has joined #openstack-infra16:49
jprgreghaynes: it may not be breaking out as much as assuming to much....  the problematic command is http://git.openstack.org/cgit/openstack/diskimage-builder/tree/bin/disk-image-create#n28216:49
anteayadtroyer: they just need to remember their launchpad password and not need a reset going forward16:49
*** sshnaidm has quit IRC16:49
greghaynesjpr: ya, I wonder if we have a case where the image build fails, we clean up, then we do the mv anyways16:50
jprgreghaynes:  ${TMP_BUILD_DIR}/built/* sweeps up the ./dev and ./proc bind mounted dirs in that build space and that's when the system /dev files disappear.16:50
*** kzaitsev_mb has joined #openstack-infra16:50
*** dims has joined #openstack-infra16:50
greghaynesjpr: ok, I need to refresh my memory on how that is supposed to work16:50
anteayadtroyer: they also need to set their new email in gerrit to their prefered email address, as that is where summit passes and election emails get sent to, the gerrit prefered email16:50
dtroyeranteaya: ah, right.  changing their LP account would be making a new one then…  and I'm guessing changing Gerrit to match is also then making a new account?16:50
jprgreghaynes:  it doesn't look like it's even gotten to  a clean up phase.  this looks more like the build work16:50
anteayadtroyer: you can add as many emails to gerrit as you like, it doesn't create new gerrit accounts16:51
greghaynesjpr: we use trap exit to do some cleanup16:51
greghaynesjpr: so itll happen at any point16:51
*** vgridnev has quit IRC16:51
anteayadtroyer: and yes, adding a new email to launchpad makes launchpad openid think it is a new account and creates a new gerrit account16:51
*** adrienverge has quit IRC16:52
*** gokrokve has quit IRC16:52
anteayadtroyer: and the scripts for summit passes and election voting sends emails to the gerrit prefered email16:52
dtroyerright, thanks16:52
anteayadtroyer: thanks for asking16:52
*** baoli has joined #openstack-infra16:53
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215616:53
jprgreghaynes: ah ok.   btw, i'm doing a command line build of the dpc image as well16:53
*** pblaho has quit IRC16:53
greghaynesjpr: what is dpc?16:53
*** gokrokve has joined #openstack-infra16:54
jprgreghaynes: sorry that's the default image name in the project-config-example nodepool.yaml  which i'm using as part of openstackci16:54
greghaynesjpr: ah, ok16:54
jprgreghaynes: it builds with these elements "ubuntu vm openstack-repos puppet nodepool-base node-devstack"16:55
greghaynesYou saw it with just 'ubuntu vm' though, correct?16:56
openstackgerritRichard Theis proposed openstack/requirements: Bump openstacksdk to 0.7.4  https://review.openstack.org/27264516:56
*** dims has quit IRC16:56
anteayagreghaynes: http://git.openstack.org/cgit/openstack-infra/project-config-example/tree/nodepool/nodepool.yaml#n3316:56
*** fawadkhaliq has joined #openstack-infra16:57
*** mikelk has quit IRC16:57
greghaynesanteaya: ty16:57
*** kushal has joined #openstack-infra16:58
*** hashar has quit IRC17:00
*** Swami has joined #openstack-infra17:01
*** jlanoux has quit IRC17:01
*** mriedem has quit IRC17:01
greghaynesjpr: hrm, ubuntu vm with sytem installed diskimage-builder works fine for me still.17:01
greghaynesI wonder if this is a case where loopback devices get in a state which confuses dib17:01
greghaynesand dib does not handle the case well17:01
*** Jeffrey4l has quit IRC17:02
*** yaume_ has joined #openstack-infra17:02
*** zeih_ has joined #openstack-infra17:02
*** jlanoux has joined #openstack-infra17:02
*** julim_ has joined #openstack-infra17:02
asselindpc is the short for devstack precise check...should be renamed....17:02
openstackgerritIsaac Beckman proposed openstack-infra/puppet-openstackci: Pass Java arguments to Jenkins master  https://review.openstack.org/26302117:03
*** fawadk has joined #openstack-infra17:03
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215617:03
*** mgould has joined #openstack-infra17:03
jprgreghaynes:  yeah. i'm wondering if this has something to do with the nodepool environment based of the openstackci work.  I looked back at my notes and see that I tested an unadorned disk-image-create command from the nodepool account home dir.  the same command from the ubuntu account home dir succeeds. "TMPDIR=/opt/dib_tmp DIB_IMAGE_CACHE=/opt/dib_cache disk-image-create -x -t qcow2 --no-tmpfs -o /opt/nodepool_dib/6 ubuntu vm"17:03
*** diazjf has quit IRC17:03
*** diazjf1 has joined #openstack-infra17:03
*** gokrokve has quit IRC17:03
*** markmcd1 has joined #openstack-infra17:04
anteayaasselin: or keep the name and dream up a new meaning17:04
*** kaisers1 has joined #openstack-infra17:04
*** hdd has joined #openstack-infra17:05
*** ashleighfarnham has joined #openstack-infra17:05
*** binbincong_ has joined #openstack-infra17:05
*** bpokorny has joined #openstack-infra17:06
*** zeih_ has quit IRC17:06
*** zeih_ has joined #openstack-infra17:06
*** isaacb has quit IRC17:07
*** kaisers has quit IRC17:07
*** zeih has quit IRC17:07
*** yaume has quit IRC17:07
*** stevemar has quit IRC17:07
*** fawadkhaliq has quit IRC17:07
*** peristeri has quit IRC17:07
*** julim has quit IRC17:07
*** markmcd has quit IRC17:07
*** nijaba has quit IRC17:07
*** binbincong has quit IRC17:07
*** jflynn has quit IRC17:07
*** jtomasek has quit IRC17:07
*** peristeri has joined #openstack-infra17:07
*** EricGonczer_ has joined #openstack-infra17:08
*** nijaba has joined #openstack-infra17:08
*** nijaba has joined #openstack-infra17:08
*** pblaho has joined #openstack-infra17:08
*** matrohon has quit IRC17:08
*** tphummel has joined #openstack-infra17:08
*** jtomasek has joined #openstack-infra17:08
*** stevemar_znc has joined #openstack-infra17:08
*** EricGonc_ has quit IRC17:08
anteayagetting close to the end of our day here, doubt I will be present at infra meeting17:08
*** jflynn has joined #openstack-infra17:09
*** signed8b_ has joined #openstack-infra17:10
*** jlanoux has quit IRC17:10
*** esikachev has quit IRC17:11
jklare@yolanda hi, i just saw that you merged https://review.openstack.org/#/c/266843/ , do you know when the new groups mentioned in the patch will be created and usable?17:11
*** ashleighfarnham has quit IRC17:12
*** jlanoux has joined #openstack-infra17:12
anteayajklare: so it sounds like this group needs to be disabled then: https://review.openstack.org/#/admin/groups/161,members17:13
anteayajklare: the groups haven't been created yet: https://review.openstack.org/#/admin/groups/?filter=openstack-chef17:14
jklare@anteaya yeah, if its possible, i would add all active core reviewers to the newly created groups17:14
jklare@anteaya so if you add me first to the core and the release one, i should be able to take care of the rest17:15
*** jordanP has quit IRC17:15
anteayawell ideally we would just rename the group rather than create a new one and disable the old one17:15
*** thorst_ has quit IRC17:15
*** EricGonc_ has joined #openstack-infra17:15
anteayahowever I don't have the ability to do that since I'm not a gerrit admin17:15
jklareanteaya sure, whatever is the best way to do that17:16
*** jaypipes has quit IRC17:16
anteayaand I'm feeling like our jeepyb manage-projects script will create the new group before anyone else can get to ti17:16
jklareworks for me :)17:16
jklareas long as someone adds me to these groups and i can add the other cores so we can still merge stuff17:17
anteayajklare: so check that link I offered and you can ask to be added to the new groups, have patience17:17
*** openstackgerrit has quit IRC17:17
anteayasomeone can add you but it make take a while before someone has time to add you17:17
*** openstackgerrit has joined #openstack-infra17:17
anteayapatience is such a wonderful quality17:18
anteayaroom is breaking for the evening, going offline, have a good day everyone17:18
*** gokrokve has joined #openstack-infra17:18
*** jaime has joined #openstack-infra17:18
*** e0ne_ has joined #openstack-infra17:18
pleia2good morning17:19
jklaremmh... usually i am very patient, but in this case i intentionally commented on this patch, that the groups should be created first (or in the case of automatic creation soon after merging), since afterwards we will not be able to merge patches until its done :)17:19
*** notnownikki has joined #openstack-infra17:20
*** sfinucan has quit IRC17:20
*** jflynn has quit IRC17:20
*** EricGonczer_ has quit IRC17:20
*** e0ne has quit IRC17:20
mordredgreghaynes: we need a new glean release?17:21
*** maishsk has joined #openstack-infra17:21
greghaynesmordred: yes, if we can get https://review.openstack.org/#/c/272253/ in first that would be awesome17:21
mordredgreghaynes: looks great17:22
yolandagreghaynes ++17:23
*** piet has joined #openstack-infra17:23
yolandaif we can also get https://review.openstack.org/267672 it will be even better17:23
yolandamordred, SpamapS17:23
*** [1]Thelo has joined #openstack-infra17:23
*** thorst_ has joined #openstack-infra17:23
openstackgerritMerged openstack-infra/glean: Fix bug in double configuring vlans on glean  https://review.openstack.org/27225317:23
*** jsavak has quit IRC17:23
*** vgridnev has joined #openstack-infra17:24
mordredyolanda: for that one, I feel like we had a localhost issue a while ago17:24
mordredgreghaynes: do you remember this?17:24
jeblaircan we maybe only release glean if there are people around to fix it if it breaks?17:24
mordredjeblair: I agree17:24
jeblairbecause we're going on 2 weeks without image updates in some situations17:24
greghaynesI am around, I am currently swimming in fixing other breaks though17:25
jeblairand i'd like to get further ahead on that before we get behind17:25
greghaynesya, I am totally fine with slowing things down while we catch up17:25
SpamapS+1 on releasing at the early part of human support windows. Sounds like we need better functional testing of glean too.17:25
greghaynesbecuase I have a stack of about 3 bad breaks that need fixing ATM17:25
*** Thelo has quit IRC17:25
*** [1]Thelo is now known as Thelo17:26
greghaynesmordred: the localhost thing I dont remember...17:26
greghaynesyolanda: I left a comment on that second patch17:26
yolandagreghaynes, your comment makes a lot of sense, i'll fix it17:26
jeblairapropos of nothing, it seems to take a really long time to upload a centos7 image to hpcloud17:26
yolandaor we are going to get recursion problems again, this time with localhost17:26
*** ashleighfarnham has joined #openstack-infra17:26
*** jsavak has joined #openstack-infra17:27
*** whoops has quit IRC17:27
*** jsavak has quit IRC17:27
*** whoops has joined #openstack-infra17:28
*** baoli has quit IRC17:28
*** thorst_ has quit IRC17:28
mordredyolanda: something like that is my concern - as is the conern that we might fight with puppet on that one17:29
jeblairgreghaynes: weird; the first image i uploaded to hpcloud took 10 mins; the second has been running for 40 minutes and is not done.  they are the same size.17:29
greghaynesSpamapS: ohai17:29
mordredyolanda: but I do not have a specific issue in mind - just an uneasy feeling17:29
greghaynesSpamapS: jpr is the one who hit the dib issue first, and then infra hit it on the nodepool nodes17:29
jeblairmordred: see my comment to greghaynes ^17:29
*** baoli has joined #openstack-infra17:29
mordredjeblair: I see this comment17:30
*** stevemar_znc has quit IRC17:30
*** mika has quit IRC17:30
*** rvba has quit IRC17:30
jeblairmordred, greghaynes: strace says data are still being moved17:30
*** dmsimard|afk has quit IRC17:30
SpamapSgreghaynes: do we have artifacts?17:30
*** EmilienM has quit IRC17:30
*** mika has joined #openstack-infra17:30
*** acruz has joined #openstack-infra17:30
greghaynesSpamapS: http://paste.openstack.org/show/484948/ is a lot jpr posted last night17:31
*** e0ne_ has quit IRC17:31
yolandamordred, puppet is working fine with that, not doubling entries for that. But we may found another problems. However i found that not having localhost entry created, prevented services like rabbitmq to run. We can add that on glean or if you think is risky, add the entry in our infra cloud in puppet17:31
greghaynesSpamapS: but not a lot more info to go on17:31
SpamapSgreghaynes: AH, that's in the mv step17:32
SpamapSHaven't looked down that hole yet17:32
*** maishsk has quit IRC17:32
*** jlanoux has quit IRC17:32
greghaynesjeblair: :/17:32
greghaynesjeblair: throttling?17:32
jeblairgreghaynes: you think hpcloud might do that?17:32
greghaynesjeblair: or rax17:32
*** jsavak has joined #openstack-infra17:33
SpamapSI think somebody finally told the cloud about its impending doom.17:33
*** arxcruz has quit IRC17:33
jeblairgreghaynes: rax has bandwidth limits, but they are consistent17:33
openstackgerritZara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View  https://review.openstack.org/27266717:34
*** dgonzalez has quit IRC17:34
jeblairit looks like we got 100mbit for my first upload, and now we're stuck at 3mbit17:34
mordredjeblair: maybe threshold-induced throttling?17:34
*** dmsimard has joined #openstack-infra17:34
jeblairmordred: from whom?17:34
greghaynesjeblair: and it looks i/o bound?17:34
mordredjeblair: good question17:34
*** stevemar_znc has joined #openstack-infra17:34
*** EmilienM has joined #openstack-infra17:34
*** dgonzalez has joined #openstack-infra17:34
*** rvba has joined #openstack-infra17:34
*** rvba has quit IRC17:34
*** rvba has joined #openstack-infra17:34
*** signed8b_ has quit IRC17:34
SpamapSjeblair: expect HP Cloud to start stalling as sunset day gets nearer. ;)17:35
mmedvedeyolanda: do you mind revisiting puppet-ciwatch review https://review.openstack.org/#/c/238606/? It is ready for approval17:35
*** ayoung_ZZZzzzz is now known as ayoung17:35
jeblairshould we just shut down hpcloud?17:35
* SpamapS imagines hpcloud asking over and over "Any word from the governor?"17:36
*** ayoung has quit IRC17:36
SpamapSjeblair: well it's certainly not worth investing much time.17:36
greghaynesjeblair: I thought the plan was to do that any day now anyhow17:36
mordredjeblair: I'm tempted to agree with that idea - I don't think debugging it is valuable at this point17:36
*** amaretskiy has quit IRC17:36
greghaynesjeblair: I want to say the 28th was the plan?17:36
greghayneswhich is two days17:36
jeblairi'm doing math to see if the image uploads would finish before then17:36
*** jsavak has quit IRC17:37
*** tonytan4ever has quit IRC17:37
SpamapSSunset is in 5 days. Though IIRC, it will still keep sort of working for a while beyond that. They're just not making uptime guarantees anymore... because.. before... they were.17:37
jeblairit will take 3 days to upload these images at 3mbit.17:38
greghaynesit is funny that if this is a WAN throttling issue we have the solution just in time to shut down the cloud needingit17:38
SpamapSgreghaynes: Ok, so what I see in jpr's report is somehow dev got mounted in 'built', but unmount_image didn't catch it17:38
mordredjeblair: then I say we sunset early17:39
greghaynesSpamapS: aha17:39
mordredjeblair: if our original plan was to turn off in 2 days17:39
greghaynesmordred: jeblair agreed on sunset now, even if it was 1 day it wouldn't be very useful17:39
greghaynessince we would get just a few hours of testing in then shutdown17:40
*** achanda has quit IRC17:40
SpamapSgreghaynes: recent changes on 12/24 and 12/31 in unmount_dir17:40
SpamapS6b7cf266 and 966f12dc17:40
*** diana_clarke has joined #openstack-infra17:40
SpamapSgreghaynes: 966f12dc being your pipefail fix17:40
SpamapSgreghaynes: Guessing that's it.17:41
*** ayoung has joined #openstack-infra17:41
greghaynesSpamapS: awesome :(17:41
greghaynesSpamapS: seems very plausible17:41
openstackgerritLukas Vacek proposed openstack/python-jenkins: Add optional kerberos support  https://review.openstack.org/23853517:41
mordredjeblair: hpcloud is the only provider providing devstack-centos7 fyi17:41
jeblairi guess no one ever got around to fixing that17:42
prometheanfiremordred: tag new glean?17:42
greghaynesprometheanfire: were fighting too many fires ATM to release that I think17:42
mordredprometheanfire: I'm in the air right now and not in a good positoin to debug things if it breaks things17:42
*** vgridnev has quit IRC17:43
pleia2pabelanger: did you want to add the summit proposals to the agenda? (busy agenda, and it's time sensitive)17:43
pabelangerif we move default_data.json from stackalytics into project-config, do you think it will be feasible to preserve the commit history of that file?17:43
pabelangerpleia2: Oh, ya. Thanks. I should do that now17:43
*** nmagnezi_ has joined #openstack-infra17:44
jeblairpabelanger, ianw, greghaynes, fungi: anyone know the status of getting centos7 working in our other clouds?17:44
SpamapSgreghaynes: I'm guessing && awk | grep on the conditional somehow failed17:44
greghaynesjeblair: I do not17:44
greghaynesjeblair: I didn't know there was an issue17:45
pabelangerjeblair: I believe ianw is focusing on fedora. I could see what is needed for centos7 dibs17:45
*** cody-somerville has joined #openstack-infra17:45
prometheanfireguess I'm stalled then17:45
greghaynesSpamapS: ok, if we want to back out the pipefail change for now I would be fine with it17:45
*** jsavak has joined #openstack-infra17:45
greghaynesSpamapS: since it should be totally safe to revert17:45
*** unicell has quit IRC17:45
jeblairgreghaynes: i have this really vague idea that its dib images weren't working on rax or something17:46
*** vgridnev has joined #openstack-infra17:46
prometheanfirenot doing frequent releases will make it harder to debug, last release was a while ago :|17:47
greghaynesprometheanfire: yep, things are very abnormably bad right now though17:47
prometheanfirebeen a bad last two weeks17:47
reedhow does one access documentation of previous releases? links from http://docs.openstack.org/kilo/ all  go to master17:49
openstackgerritMonty Taylor proposed openstack-infra/project-config: Turn off HPCloud  https://review.openstack.org/27267417:49
*** sshnaidm has joined #openstack-infra17:49
mordredjeblair: ^^17:49
*** jsavak has quit IRC17:50
jeblairi'm doing some terribly bad things to the builder to clear out the bunch of upload jobs i enqueued17:50
jeblairmordred, greghaynes: i stopped and started the builder17:50
jlkHeh, would it be inappropriate for me to +1 the removal of HP Cloud?  :D17:51
jeblairthat aborted the slow job17:51
greghaynesSpamapS: Ya, the pipefail is extremely suspicious given that it there are a lot of cases related to loopback detachment with pipes and infra was seeing /dev/loop oddness17:51
jeblairjlk: certainly not if you also added 500 more instances to bluebox ;)17:51
jlkall it takes is blue dollars17:51
nibalizeroh is today the day17:51
*** hockeynut is now known as hockeynut_otr17:51
mordrednibalizer: not originally17:51
jeblairmordred, greghaynes: i restarted the builder, and the next upload is going much faster17:52
greghaynesjeblair: is it !hpcloud?17:52
mordrednibalizer: but we can't upload images to hp anymore without it taking actually multiple days17:52
jeblairgreghaynes: all hpcloud17:52
greghaynesjeblair: :(17:52
*** diazjf1 has quit IRC17:52
*** hockeynut_otr is now known as hockeynut17:52
mordrednibalizer: o - maybe spoke too soon17:52
*** tonytan4ever has joined #openstack-infra17:52
greghaynesjeblair: and the builder wasn't chewing up cpu before or anything, correct?17:52
jeblairgreghaynes: correct17:52
openstackgerritMerged openstack/requirements: Cap PyKMIP version  https://review.openstack.org/26059917:52
openstackgerritZara proposed openstack-infra/storyboard-webclient: Show Each Task ID in Story Detail View  https://review.openstack.org/27266717:52
*** electrofelix has quit IRC17:53
*** jsavak has joined #openstack-infra17:53
*** cody-somerville has quit IRC17:53
greghaynesjeblair: I don't have any great ideas, then. All that code should be the same and is just blocking on openstack clients17:53
*** dgonzalez has quit IRC17:53
jeblairgreghaynes: yeah, so maybe hpcloud had a sad and then got less sad17:53
greghaynesvery possible17:53
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a update in trunk repository version  https://review.openstack.org/22978917:54
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Switch the overcloud pingtest to use the new heat client  https://review.openstack.org/27247917:54
*** ifarkas has quit IRC17:54
greghayneswe could have some smarts to timeout uploads...17:54
*** zz_zz_ja is now known as zz_zz_zz_ja17:54
*** jaosorior has quit IRC17:55
greghaynesjpr: Do you have the ability to reproduce the diskimage-builder failure locally?17:55
greghaynesjpr: if so I have a change i'd like you to test17:55
*** dgonzalez has joined #openstack-infra17:55
jprgreghaynes: yes. on a call back in a few17:55
SpamapSI have to step away too17:55
greghaynesSpamapS: ok, ty for the debug help17:56
SpamapSjpr: try reverting 966f12dc4cd6f48f57a6d170d7fbccc5a28892a9 and running again, if you can.17:56
SpamapSI don't actually see the logic error17:56
*** abregman has joined #openstack-infra17:56
SpamapSbut I think things may get a bit.. confusing.. with pipefails and the like.17:56
SpamapSwe may want to make this very inelegant and brute-force... checking $? explicitly and maybe operating on temp files, just to be safe.17:57
*** jsavak has quit IRC17:57
greghaynesYep, I think this is going to require a while of paging in how all our unmounting stuff works17:58
*** abregman has quit IRC17:58
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements  https://review.openstack.org/24611417:58
*** abregman has joined #openstack-infra17:58
greghaynesso hopefully someone can do that when we arent potentially harming people's build hosts17:58
*** sigmavirus24 is now known as sigmavirus24_awa17:59
*** hdd has quit IRC17:59
*** zhurong has quit IRC17:59
*** hockeynut_afk has joined #openstack-infra18:00
*** julim has joined #openstack-infra18:00
jeblairgreghaynes, mordred: on to the next upload; seems faster too18:00
openstackgerritSpencer Krum proposed openstack-infra/system-config: PL released 1.8.0 of the apache module  https://review.openstack.org/27267618:00
*** derekh has quit IRC18:01
greghaynesjeblair: awesome18:01
nibalizerpabelanger: ^^18:01
* greghaynes assumes hpcloud sad, then18:01
jeblairgreghaynes, mordred: so maybe we can leave hpcloud in for a bit and give folks some time to fix centos7?18:01
greghaynesjeblair: SGTM18:01
jeblairi'll keep babysitting it18:01
*** flepied has quit IRC18:03
*** sigmavirus24_awa is now known as sigmavirus2418:03
*** julim_ has quit IRC18:03
mordredjeblair: ++18:03
*** sigmavirus24 is now known as sigmavirus24_awa18:04
pabelangernibalizer: done18:04
*** lucasagomes is now known as lucas-dinner18:04
*** hockeynut_afk has quit IRC18:05
openstackgerritMerged openstack/requirements: bump keystonemiddleware in upper constraints  https://review.openstack.org/27218318:06
*** hockeynut_otr has joined #openstack-infra18:06
*** sridhar_ram has quit IRC18:06
*** abregman is now known as abregman|mtg18:07
openstackgerritMonty Taylor proposed openstack-infra/project-config: Add centos7 nodes to OVH  https://review.openstack.org/27267818:08
*** jsavak has joined #openstack-infra18:08
mordredjeblair: ^^ there is just the addition of centos7 to ovh18:08
*** yaume_ has quit IRC18:09
jeblairmordred: you want to try both at once?18:10
jeblairmordred: why not try the dib first?18:10
mordredjeblair: we have some jobs that use each type18:10
*** yaume has joined #openstack-infra18:10
jeblairnow i wonder, why do we have a -dib and non-dib version of that?18:10
*** achanda has joined #openstack-infra18:10
*** thorst_ has joined #openstack-infra18:11
*** yaume has quit IRC18:11
mordredthat's my problem - I do not know why we have both - so I figured that if we can build both in ovh, we can divorce the problem of dib/non-dib centos7 from the hp issue18:11
jeblairyeah, i guess that sounds okay18:11
jeblairi hope someone comes along at some point and does care18:11
mordredif we cannot build both in ovh, then we will obviously need to get centos7-dib solid18:11
*** jistr has quit IRC18:11
*** zeih_ has quit IRC18:11
*** BobBall is now known as BobBall_AWOL18:11
mordred(since I am confident in our ability to upload centos images to ovh)18:11
mordredianw: ^^ when you are around18:12
jeblairi feel like using a snapshot image on ovh is retrograde18:12
*** esikachev has joined #openstack-infra18:12
openstackgerritJulien Danjou proposed openstack-infra/project-config: gnocchi: add openstack-server-release-jobs  https://review.openstack.org/27268018:12
*** sriram has left #openstack-infra18:13
jeblairmordred, greghaynes: i think we just switched back to slow mode on hpcloud18:14
greghaynesquick, hit the tubo button rest endpoint18:14
openstackgerritMerged openstack-infra/devstack-gate: Use devstack subunit stream in output  https://review.openstack.org/23444718:14
jeblairgreghaynes, mordred: basically, my process is i check strace and see if things scroll by fast or slow18:15
mordredjeblair: that's an excellent process18:15
*** dguitarbite has quit IRC18:15
openstackgerritSpencer Krum proposed openstack-infra/system-config: Bump haproxy to 1.4.0  https://review.openstack.org/27268118:15
nibalizerjeblair: heh18:16
jeblairmordred, greghaynes: restarted the builder, this upload is fast18:16
*** dguitarbite has joined #openstack-infra18:16
greghaynesjeblair: maybe we can remove all but centos7 from hpcloud? Seems like any time spent on babysitting hpcloud isn't well spent time18:17
jeblairgreghaynes: mordred ^ that might be a good compromise18:17
mordredjeblair: kk18:17
*** gokrokve has quit IRC18:17
*** unicell has joined #openstack-infra18:17
mordredjeblair: add to the centos7-on-ovh patch?18:17
jeblairmordred: i aprvd it, so make a followup18:18
mordredbtw - what are we keeping precise around for?18:18
jeblairmordred, greghaynes: i'll adjust my process to only upload centos images to hpcloud18:18
*** harlowja has joined #openstack-infra18:18
jeblairno idea18:18
jeblairmordred: maybe kilo or something?18:18
mordredoh - infra puppet apply :)18:19
jeblairheh or that18:19
jeblairyeah, many of our servers are still precise18:19
*** thorst_ has quit IRC18:19
mordredI _think_ we're actually upgraded on almost all of them now18:19
nibalizerhttp://puppetboard.openstack.org/fact/lsbdistcodename mordred nope18:19
mordrednope. I'm wrong18:19
nibalizerwe're more precise than aanything else18:19
*** thorst_ has joined #openstack-infra18:19
mordredwell, let's not start that project yet18:20
nibalizerbut some of this stuff probably is pretty easy18:20
jeblairit was handy when we had the recent security updates :)18:20
nibalizerapps.o.o and cacti.o.o should just go18:20
jeblair(which did not apply to precise)18:20
*** stevemar_znc is now known as stevemar18:20
nibalizeroh the whole elasticsearch cluster is on precise18:20
nibalizerjenkins masters and logstash workers18:21
*** e0ne has joined #openstack-infra18:21
jeblairi'm telnetting to geard and canceling upload jobs18:21
jeblairwhich i just remembered is a thing18:21
*** fawadk has quit IRC18:24
*** thorst_ has quit IRC18:24
*** MarkAtwo_ has quit IRC18:25
*** mgould has quit IRC18:25
jprgreghaynes: is that commit for disk-image-builder?18:25
*** jsavak has quit IRC18:25
*** eil397 has joined #openstack-infra18:26
greghaynesjpr: yes18:26
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove everything from HP except centos7  https://review.openstack.org/27268418:26
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove devstack-precise and devstack-precise-check  https://review.openstack.org/27268518:26
jprgreghaynes: btw, my manual build of dpc image worked fine from the ubuntu account. no messup of /dev18:26
mordredjeblair, nibalizer: ^^18:26
mordredjeblair: turns out we do not use devstack-precise, only bare-precise18:27
openstackgerritMerged openstack-infra/project-config: Add centos7 nodes to OVH  https://review.openstack.org/27267818:28
jeblairmordred: removal of devstack-precise-2-node snuck into the first patch.  should we care enough to reroll?18:28
*** hockeynut_otr has quit IRC18:28
mordredjeblair: nah18:29
*** achanda has quit IRC18:29
jeblairk, noted and +2d18:30
jeblairnibalizer: ^?18:30
mordredjeblair: I do not see anything in jenkins jobs that use centos7-dib18:30
mordredjeblair: but I might be reading something wrong18:30
*** angdraug has quit IRC18:30
mordredjeblair: looked in jjb and zuul layuot18:30
jeblairmordred: i wonder if the problem is that centos7-dib wasn't working _at all_ ?18:30
*** sabeen3 has quit IRC18:31
jeblairi guess we're about to find out18:31
*** rossella_s has quit IRC18:31
jeblair(this makes my babysitting centos7 dib uploads to hpcloud seem even more pointless)18:31
*** rossella_s has joined #openstack-infra18:31
mordredjeblair: double check me on the "we don't use them" - but if I'm right, I suggest we kill those too and just leave the centos7 snapshot images in hp18:32
jprgreghaynes: testing with dib 1.6.1.dev118:34
jeblairmordred: i agree, nothing uses devstack-centos7-dib18:34
openstackgerritMonty Taylor proposed openstack-infra/project-config: Remove devstack-centos7-dib from HP  https://review.openstack.org/27269118:35
mordredjeblair: there you go then18:35
*** jsavak has joined #openstack-infra18:35
jeblairmordred: that will mean no dib images in hpcloud, so i can just drop what i'm doing entirely :)18:35
mordredjeblair: yup18:35
*** zeih has joined #openstack-infra18:35
*** Swami has quit IRC18:36
*** jsavak has quit IRC18:36
nibalizermordred: we still have a couple devstack-precises in nodepool right now18:36
*** jsavak has joined #openstack-infra18:36
nibalizerya we need to do the thing where we set min-ready to 018:37
*** gokrokve has joined #openstack-infra18:37
nibalizerthen let it kill them off, then we can remove it entirely18:37
*** flepied has joined #openstack-infra18:37
mordrednibalizer: or maybe just delete them if you're looking at it atm18:37
jeblairlet's merge the changes first, then update the config (which we have to do manually), then delete the nodes/images we don't need any more18:38
jprgreghaynes: foo.  seemed like it was getting farther but failed in the end with the same mv command evacuating /dev18:38
jeblairnibalizer: stack at https://review.openstack.org/272691 is all yours if you want to review18:38
openstackgerritMonty Taylor proposed openstack-infra/project-config: Turn off HPCloud  https://review.openstack.org/27267418:39
openstackgerritMonty Taylor proposed openstack-infra/project-config: Cleanup after ourselves post HP sunset  https://review.openstack.org/27269218:39
jeblairnibalizer: can we skip the set to 0 first?18:39
mordredand there are those for the future18:39
*** abregman|mtg is now known as abregman18:39
nibalizerjeblair: so in gozer we would always set it to 0 with config changes18:40
nibalizerbecause if I just type delete right now it will make more18:40
jeblairnibalizer: yeah, i'm volunteering to fix things manually18:40
nibalizeroh okay18:40
nibalizersure then18:40
*** slogan621 has joined #openstack-infra18:40
jeblairnibalizer: puppet is not running on that host anyway, so we will even need to make the config changes manually (which i'm also volunteering to do)18:41
*** flepied1 has joined #openstack-infra18:41
greghaynesjeblair: for actually fixing nodepool->gear disconnects it might be worth trying colocating a separate geard on the nodepool host just for builders18:41
*** jsavak has quit IRC18:41
*** gokrokve has quit IRC18:42
jeblairgreghaynes: we'll have to teach nodepool to connect to two different geards18:42
greghaynesjeblair: yep18:42
*** jsavak has joined #openstack-infra18:42
greghaynesjeblair: That should be pretty easy18:42
jeblairgreghaynes: (i mean, for different purposes; it can already do so for ha purposes)18:42
slogan621If I create an openstack project (e.g., openstack/foo), which is unofficial, is it required that contributors sign the individual contributor agreement, and/or the corporate contributor agreement?18:42
*** ybathia has joined #openstack-infra18:42
jeblairgreghaynes: however, i'd like to reserve that option for later18:43
*** ashleighfarnham has quit IRC18:43
greghaynesjeblair: What alternative are you thinking?18:43
jeblairslogan621: see this change, which is not yet official, but some version of it is likely to be and should hopefully answer your question: https://review.openstack.org/26982318:43
*** sridhar_ram has joined #openstack-infra18:44
jeblairgreghaynes: well, i'd like to find the cause :)18:44
*** flepied has quit IRC18:44
openstackgerritMerged openstack-infra/project-config: Remove everything from HP except centos7  https://review.openstack.org/27268418:44
jeblairgreghaynes: let me rephrase that -- i would like the cause of the problem to be found :)18:44
greghayneshah, yep, we probably should do that18:45
*** zz_zz_zz_ja has quit IRC18:45
*** vgridnev has quit IRC18:45
jeblairgreghaynes: probably first steps are some more logging on the builder side (maybe gear debug logging); then if that doesn't tell us anything, we may need to turn on geard debug logging too.  that must be done with care because it is a torrent18:45
*** baoli has quit IRC18:45
greghaynesjeblair: when we dug through logs geard had no idea about the disconnect until it tried to write to the connection to nodepool. We couldn't find the initial traceback on the nodepool side from when it disconnected18:46
greghaynesjeblair: I am not sure we have seen any disconnects from the builder18:46
jeblairnibalizer, mordred: i'm going to take a few mins to prepare for eating lunch at my keyboard; biab and will apply changes then18:47
*** BobBall_AWOL has quit IRC18:48
*** achanda has joined #openstack-infra18:48
*** ashleighfarnham has joined #openstack-infra18:48
*** sdake has quit IRC18:51
openstackgerritMerged openstack-infra/project-config: Remove devstack-precise and devstack-precise-check  https://review.openstack.org/27268518:51
*** trown is now known as trown|lunch18:51
*** ihrachys has quit IRC18:52
*** sdake has joined #openstack-infra18:53
*** angdraug has joined #openstack-infra18:53
*** pvaneck has joined #openstack-infra18:54
slogan621jeblair: thanks. If you would, look at http://docs.openstack.org/infra/manual/creators.html - the section Request Signing of ICLA says that I can specify in the project if contributors are required to sign the ICLA if the project is unofficial. It implies that no corporate license agreement is required, either.18:56
*** sigmavirus24_awa is now known as sigmavirus2418:57
slogan621I assume that would effectively kill any chance of the project passing through governance, but my goal is to get code contributed, not necessarily be official. And I'm struggling with a company that has an issue t=with signing the corporate agreement.18:57
*** jaosorior_ has joined #openstack-infra18:57
openstackgerritMerged openstack-infra/project-config: Remove devstack-centos7-dib from HP  https://review.openstack.org/27269118:58
*** diazjf has joined #openstack-infra18:58
*** krtaylor has quit IRC18:58
slogan621how would I go about getting a binding interpretation on this? What it says seems clear, contributors to unofficial projects can be allowed without signing anything.18:59
*** flepied has joined #openstack-infra18:59
*** gokrokve has joined #openstack-infra18:59
pleia2meeting time :)19:00
*** ybathia has quit IRC19:00
*** flepied1 has quit IRC19:01
*** jsavak has quit IRC19:02
*** jsavak has joined #openstack-infra19:03
*** krtaylor has joined #openstack-infra19:03
*** amitgandhinz has quit IRC19:04
*** shardy_ has quit IRC19:04
*** gokrokve has quit IRC19:04
*** amitgandhinz has joined #openstack-infra19:05
*** kzaitsev_mb has quit IRC19:06
*** dkranz has quit IRC19:08
*** erikwilson has joined #openstack-infra19:08
openstackgerritMerged openstack/requirements: Updated from generate-constraints  https://review.openstack.org/27190219:08
*** tmcpeak has joined #openstack-infra19:09
tmcpeakhey all, it seems like that thing that pushes new versions to PyPI based on annotated tags might not have worked or is stuck, where's the right place to check?19:09
*** hdd has joined #openstack-infra19:09
tmcpeakfor Bandit specifically19:10
*** dkranz has joined #openstack-infra19:10
openstackgerritgreghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support  https://review.openstack.org/27238419:11
greghaynesjeblair: ^ that should help some, I still need to do the periodic cleanup for jobs which fall through the cracks19:11
zaropuppetboard.o.o seems broken, can any infra-root folks please take a look?19:11
greghaynesI also really want to test it a bit more19:11
openstackgerritSven Anderson proposed openstack-dev/pbr: Extend compile_catalog command to compile all .po files  https://review.openstack.org/27156219:12
zaronv, looks like it just came back online.19:12
nibalizerzaro: it goes up and down19:12
nibalizerthere is an item in the meeting agjenda to talk about it19:13
pleia2it's on the agenda for today19:13
zarois there a way to get older reports from puppetboard?19:13
*** dims has joined #openstack-infra19:13
*** dkehn has quit IRC19:14
*** arahal_ has joined #openstack-infra19:15
ianwgreghaynes: i get the feeling gate-tempest-dsvm-ironic-pxe_ssh-dib is broken, but maybe only on ovh.  who is responsible for this job?19:15
greghaynesianw: it is, there is an ironic launchpad bug up for it19:16
greghaynesianw: last I heard they were swimming in failures and trying to dig out19:16
jeblairslogan621: i would recommend writing to jbryce (jonathan bryce) at the openstack foundation with your concerns19:16
*** Sukhdev has joined #openstack-infra19:17
ianwgreghaynes: oh, ok; can we kick it to non-voting while it's sorted out?19:17
greghaynesianw: eh, I'd rather help them debug it if I had time...19:17
*** zeih has quit IRC19:17
*** ybathia has joined #openstack-infra19:17
tmcpeakinfra wizards - can somebody please point me to the process that combs the openstack git repo for annotated commits and pushes to PyPI19:17
*** dtardivel has quit IRC19:17
tmcpeakor however that works, the things that does that?19:18
tmcpeakseems like it hasn't picked up my 0.17.1 Bandit release19:18
greghaynesianw: looks like https://review.openstack.org/#/c/272564/ is a potential fix19:18
greghaynesianw: https://bugs.launchpad.net/ironic/+bug/1537754 is the bug19:18
openstackLaunchpad bug 1537754 in Ironic "Unable to associate floating IP to fixed IP for instance ..." [High,Triaged]19:18
*** r-daneel has quit IRC19:18
*** ybathia has quit IRC19:19
*** ybathia has joined #openstack-infra19:19
*** arahal_ has quit IRC19:19
openstackgerritMichael Krotscheck proposed openstack-infra/infra-specs: Unified Mirrors  https://review.openstack.org/25267819:19
slogan621jeblair: thanks19:20
ianwgreghaynes: well that actually appears to be a different problem to http://logs.openstack.org/62/271862/5/check/gate-tempest-dsvm-ironic-pxe_ssh-dib/340baed/ ...19:20
*** dkehn has joined #openstack-infra19:20
greghaynesianw: oh, a different fail for that job19:20
*** dprince has quit IRC19:21
greghaynesianw: https://etherpad.openstack.org/p/IronicWhiteBoard is useful for finding possible bug references19:21
*** achanda has quit IRC19:21
openstackgerritMichael Krotscheck proposed openstack-infra/infra-specs: Unified Mirrors  https://review.openstack.org/25267819:22
AJaegerianw: ironic team is and they're working on it...19:22
AJaegerianw: all of ironic is blocked19:22
*** sabeen1 has joined #openstack-infra19:22
*** achanda has joined #openstack-infra19:23
*** dims has quit IRC19:24
ianwAJaeger / greghayes : ok, sounds like SNAFU19:24
greghaynesianw: yep. I think they could definitely use some help19:25
ianwSpamapS / greghaynes : if I could get some eyes on https://review.openstack.org/#/c/271862/ modulo the ironic issues, that would be good.  required for devstack to work on F2319:25
*** Swami has joined #openstack-infra19:25
*** achanda has quit IRC19:25
greghaynesianw: ok19:25
AJaegerianw, greghaynes https://review.openstack.org/270356 could use some review love as well, please - but is blocked like others...19:26
greghaynesAJaeger: ah, that19:27
jrollianw: greghaynes: we have a revert of a neutron commit that seems suspect, was waiting for jenkins19:27
* jroll checks up on it19:27
ianwjroll: latest comment in there seems to say it's not that ...19:28
greghaynesjroll: I saw neutron dev say they think it is suspect19:28
jrollI'm just saying what I know right now.19:28
jrollI see the comments19:28
*** diana_clarke has quit IRC19:28
*** simondodsley has quit IRC19:28
greghaynesjroll: hah, ok :)19:28
jrolland I guess I sit and wait for ryan to post a different patch19:28
docaedoAny chance someone would like to take a quick look at an update for the app-catalog periodic job? Not urgent, but maybe some infra folk would like a quick break from doing all the hard thinking they've been up to lately :) https://review.openstack.org/#/c/269935/19:29
*** sabeen3 has joined #openstack-infra19:29
*** thorst_ has joined #openstack-infra19:29
jrollianw: the last logs you posted is another bug I'm working on but is more intermittent19:30
AJaegerthanks, ianw and greghaynes !19:30
*** bpokorny_ has joined #openstack-infra19:30
jrollgreghaynes: ianw: I guess I'm not opposed to making our job nv on dib though, I don't want to block y'all19:31
*** bpokorny_ has quit IRC19:31
greghaynesjroll: eh, that job has been really stable, it only seems to fail for actual breaks (like this)19:31
greghaynesso I like it19:31
*** sabeen1 has quit IRC19:31
*** bpokorny_ has joined #openstack-infra19:31
*** mkoderer_cloud has quit IRC19:31
jrollgreghaynes: okay, our jobs aren't super stable lately is why I mention it19:32
greghaynesjroll: well, relative to dib tests...19:32
jrollfair enough :)19:32
*** thorst_ has quit IRC19:33
*** thorst_ has joined #openstack-infra19:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Don't use internap for 2-node labels"  https://review.openstack.org/26976319:34
*** bpokorny has quit IRC19:34
AJaegerjeblair: I've rebased your change, is it safe to merge this now ? ^19:34
tmcpeakhey all, can somebody please help me figure out why my annotated tag didn't get pushed to PyPI19:34
jeblairAJaeger: afaik :)19:35
AJaegertmcpeak: which repo? Which tag?19:35
AJaegerjeblair: then let's merge ;)19:35
*** rlandy has quit IRC19:35
tmcpeakAJaeger: thank you, https://github.com/openstack/bandit/releases/tag/0.17.119:35
AJaegertmcpeak: nit git.openstack.org is our master ;)19:36
jeblairAJaeger: ++19:36
tmcpeakAJaeger: ahh, I learn something new e'ryday :)19:36
*** thorst_ has quit IRC19:38
AJaegerstrange, 0.17.1 is not in my local repo ;(19:39
tmcpeakAJaeger: huh?19:39
AJaegernot sure what's broken there...19:39
tmcpeakwhere's your local repo?19:39
*** daemontool_ has joined #openstack-infra19:40
greghaynesjpr: Any updates on that revert?19:40
greghaynesjpr: oh, I missed, you said it still failed19:40
jprgreghaynes: yeah. it didn't work (noted above) :/19:40
greghaynesjpr: can you paste some logs?19:40
*** rlandy has joined #openstack-infra19:41
greghaynesjpr: and also that reverted just adding set -o pipefail, correct?19:41
jprin a meeting.  i'll get the log out in a few.19:41
greghaynesjpr: ok, ty19:41
*** dprince has joined #openstack-infra19:42
AJaegeryolanda: your move of the log files resulted in an extra logs directory, see http://logs.openstack.org/logs/ and http://logs.openstack.org/19:42
*** daemontool has quit IRC19:43
*** eil397 has quit IRC19:43
*** adrienverge has joined #openstack-infra19:43
*** ociuhandu has quit IRC19:44
AJaegeranybody that can help me to find the logs of a release job, please?19:45
*** ashleighfarnham has quit IRC19:46
*** eil397 has joined #openstack-infra19:46
*** salv-orl_ has quit IRC19:46
AJaegerinfra-root, I need some help - bandit has according to http://git.openstack.org/cgit/openstack/bandit a 0.17.1 tag - but I do not see that in my local tree. My origin is https://git.openstack.org/openstack/bandit19:46
AJaegerthis is on two machines ;( I do not see the 0.17.1 tag at all. Do we have a corruption on one of our gits?19:47
* AJaeger checks out again from scratch...19:47
*** dizquierdo has quit IRC19:47
tmcpeakAJaeger: thanks for looking into it19:48
AJaegerfresh checkout is fine ;(19:48
zaronibalizer: is there a way to get older than last ten puppet reports ?19:48
*** esikachev has quit IRC19:50
*** HeOS has joined #openstack-infra19:50
nibalizerum i think so19:50
nibalizerin puppetboard do you mean?19:50
*** gokrokve has joined #openstack-infra19:52
*** edmondsw has quit IRC19:53
nibalizerya it looks like our version doesn't support the config option but more recent versions do19:53
*** BobBall_AWOL has joined #openstack-infra19:55
*** gokrokve has quit IRC19:56
*** trown|lunch is now known as trown19:56
AJaegerhttps://jenkins.openstack.org/view/All/job/bandit-pypi-both-upload/ does not show the job at all, last run was for 0.17.0. I need some help on what's going on here...19:57
nibalizeralso related to puppetdb crashing http://paste.openstack.org/show/485071/19:58
*** rcernin has quit IRC19:58
*** rfolco has quit IRC19:59
*** aysyd has quit IRC19:59
pleia2nobody died while I was chairing \o/20:00
*** rockyg has joined #openstack-infra20:00
pleia2(then again mordred's plane was still landing, so maybe the jury is out on that)20:01
AJaegerpleia2: just bandit 0.17.1 release confuses me.20:01
*** EricGonc_ has quit IRC20:01
AJaegerMy existing repos do not show the change nor the tag when pulling, a new checkout looks fine.20:01
AJaegerBut worse: No release to pypi done - see jenkins link above20:01
pleia2elizabeth@r2d2$:~/bandit(master)$ git tag | grep 0.17.120:02
*** gokrokve has joined #openstack-infra20:02
pleia2^^ fresh clone20:02
zaropleia2: nice job today20:02
AJaegeranybody with an old clone?20:02
pleia2zaro: thank you20:02
*** jsavak has quit IRC20:02
*** ashleighfarnham has joined #openstack-infra20:02
*** EricGonczer_ has joined #openstack-infra20:03
pleia2I'm off to lunch now20:03
*** jsavak has joined #openstack-infra20:03
*** matrohon has joined #openstack-infra20:03
zaronibalizer: so how can review puppet reports for review.o.o around 1/15/2016 ?20:03
*** austin81 has joined #openstack-infra20:03
AJaegertmcpeak: I can't find the logs, so this needs infra-root help. Looks like the job never run and I don't know why..20:06
tmcpeakAJaeger: ok cool, anything I should do to get the ball rolling on that?20:06
AJaegertmcpeak: ask here - I'll leave soon...20:06
*** vgridnev has joined #openstack-infra20:06
pabelangerjeblair: so, centos-7 dib worked as expected locally. will upload to clouds to see what happens20:07
tmcpeakAJaeger: ok will do, thanks for taking a look20:07
*** kzaitsev_ws has quit IRC20:07
*** kzaitsev_ws has joined #openstack-infra20:07
*** ashleighfarnham has quit IRC20:10
*** matrohon has quit IRC20:10
*** sdake has quit IRC20:10
rbrndt_Having trouble in a gate job, horizon install fails with "+ django-admin.py compress --force20:11
rbrndt_2016-01-26 17:18:47.906 | /opt/stack/new/devstack/lib/horizon: line 153: django-admin.py: command not found"20:11
*** sridhar_ram has quit IRC20:11
*** diazjf has quit IRC20:11
rbrndt_Not sure how to solve this or how its related to my changes in https://review.openstack.org/#/c/269805/620:11
prometheanfiremordred: why do source and repo install seem switched up to me?   https://github.com/openstack/diskimage-builder/tree/master/elements/simple-init/install.d20:11
prometheanfirerepo install uses 'pip install /tmp/glean.git'      source install uses 'pip install glean'20:12
zaroyolanda: ping20:13
*** pradk has quit IRC20:14
*** pradk_ has joined #openstack-infra20:14
*** matrohon has joined #openstack-infra20:14
*** ldnunes_ has quit IRC20:14
*** pradk_ is now known as pradk20:14
openstackgerritgreghaynes proposed openstack-infra/gear: Guard against double shutdown in client  https://review.openstack.org/27271720:15
openstackgerritgreghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support  https://review.openstack.org/27238420:15
*** krtaylor has quit IRC20:16
nibalizerzaro: im not sure we have those20:17
nibalizerwhy do you need them?20:17
*** rguillebert has quit IRC20:17
nibalizerif its super critical me or another rooter can make api requests against puppetdb directly but its kindof a lot of work20:17
*** notnownikki has quit IRC20:18
*** EricGonc_ has joined #openstack-infra20:18
*** adrienverge has quit IRC20:18
nibalizerHunner: i think our custom 'submit to puppetdb' python script is causing this http://paste.openstack.org/show/485071/ maybe? thats puppetdb.log20:18
*** ashleighfarnham has joined #openstack-infra20:19
*** flwang1 has joined #openstack-infra20:20
*** smarcet has quit IRC20:21
*** EricGonczer_ has quit IRC20:21
*** gildub has joined #openstack-infra20:22
AJaegerpabelanger: I've updated the talk description, please tell me whether you like it or whehter it needs more work.20:22
*** rguillebert has joined #openstack-infra20:22
AJaegerpabelanger: thanks for pulling this together20:22
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215620:23
*** ashleighfarnham has quit IRC20:23
*** ashleighfarnham has joined #openstack-infra20:24
Hunnernibalizer: Well, it sounds like the report it's trying to submit already exists?20:24
HunnerI don't think the python script can help that20:25
*** dane-fichter_ has quit IRC20:26
*** nikhil has quit IRC20:27
*** nikhil has joined #openstack-infra20:28
*** BobBall_AWOL has quit IRC20:29
*** diazjf has joined #openstack-infra20:31
*** derekh has joined #openstack-infra20:31
*** rcernin has joined #openstack-infra20:33
*** flwang has left #openstack-infra20:33
tmcpeakhey all, I'm trying to figure out why Bandit 0.17.1 hasn't automatically been pushed to PyPI by OpenStack CI despite my release of an annotated tag.  AJaeger took a look earlier and seems to think something might be wrong with the jobs doing it.  Any infra-root available to take a look please?20:33
*** jsavak has quit IRC20:33
*** notnownikki has joined #openstack-infra20:33
*** jsavak has joined #openstack-infra20:34
*** gokrokve has quit IRC20:34
jprgreghaynes: uploaded image build log from the failed build with 1.6.1.dev120:36
*** harlowja_ has joined #openstack-infra20:36
*** gokrokve has joined #openstack-infra20:36
*** harlowja has quit IRC20:37
*** yamahata has quit IRC20:37
*** hashar has joined #openstack-infra20:37
*** signed8b_ has joined #openstack-infra20:38
clarkbtmcpeak: http://logs.openstack.org/f4/f4961732992df5b9560b268114ddf8b1b53501f7/release/bandit-tarball/beb1b8c/console.html#_2016-01-26_18_55_44_31320:40
tmcpeakclarkb: huh, any idea why that would be?  http://git.openstack.org/cgit/openstack/bandit/tag/?h=0.17.120:41
clarkbnot sure why that happened and I am in a van in anothwr country unable to debug further20:41
*** sridhar_ram has joined #openstack-infra20:41
tmcpeakclarkb: good times :)20:42
AJaegerclarkb: thanks for that link. So, that confirms my problem: udpating a local checkout does not get me 0.17.1, only a fresh checkout does.20:42
AJaegeris the repo somehow broken?20:42
clarkbso things to check20:43
clarkbis there a branch with that name?20:43
*** sdake has joined #openstack-infra20:44
tmcpeakthere's an annotated tag with that name yes20:44
AJaegerno branch for git branch --remote20:44
*** harlowja has joined #openstack-infra20:45
clarkbI dont think you can have ambiguous overlapping names20:45
*** harlowja_ has quit IRC20:45
tmcpeakclarkb: what do you mean?20:45
openstackgerritTim Burke proposed openstack-infra/meetbot: Make timestamps link to their own anchors  https://review.openstack.org/27272620:46
tmcpeakthis is the same tag naming format we've been using for a while now20:46
clarkbI mean you can only have one thing called 0.17.120:46
tmcpeakI believe that's what we have20:46
openstackgerritTim Burke proposed openstack-infra/meetbot: Add __str__ methods for items  https://review.openstack.org/27272720:46
*** esikachev has joined #openstack-infra20:46
openstackgerritRichard Theis proposed openstack/requirements: Bump openstacksdk to 0.7.4  https://review.openstack.org/27264520:46
prometheanfiremordred: is is possible to even run the simple-init datasource from git? it states it's missing the git repo (didn't seem to fetch)20:46
*** salv-orlando has joined #openstack-infra20:47
tmcpeakthere is only one "0.17.1" tag if that's what you mean clarkb20:47
*** sdake has quit IRC20:47
clarkbI mean only one ref at all20:47
clarkbbranch tag etc20:48
pabelangerianw: regarding 272590 I would be also okay with sane defaults for gearman, like localhost or something20:48
prometheanfiremordred: you can test with prepending your dib call with DIB_DEFAULT_INSTALLTYPE=repo20:48
tmcpeakyeah, we only deal in the 'master' branch in Bandit, that should be fine20:48
pabelangerAJaeger: roger20:48
*** sdake has joined #openstack-infra20:49
AJaegerpabelanger: feel free to edit...20:49
pabelangerAJaeger: looks good to me20:49
prometheanfiremordred: ah, because the source-repo element checks for INSTALL_TYPE being source, so that switch is breaking things20:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Remove gate-openstack-ansible-lint  https://review.openstack.org/27273120:50
*** signed8__ has joined #openstack-infra20:50
*** esikachev has quit IRC20:51
tmcpeakok so it seems clear that it wasn't able to grab that tagged version, which AJaeger also found locally20:51
*** zul has quit IRC20:52
tmcpeakthe question is why20:52
AJaegerthe strange thing is that the top of my repo is: Merge "This fixes the blacklist to look like a plugin so we can filter it"20:53
*** signed8b_ has quit IRC20:53
*** pradk has quit IRC20:53
*** derekh has quit IRC20:53
*** krtaylor has joined #openstack-infra20:54
*** pradk has joined #openstack-infra20:54
*** matrohon has quit IRC20:54
tmcpeakAJaeger: yeah so I did something a little unconventional20:54
AJaegertmcpeak: when I run git show on 0.17.1, it shows me a commmit:20:54
AJaegerand when I run git show on that  one I get20:54
AJaegerfatal: bad object c364408faa25ccdc378b72c0a9e2150d20a94cba20:54
*** diazjf has quit IRC20:54
AJaegerbut that's on my existing repo - it works on the fresh clone20:55
*** bknudson has joined #openstack-infra20:55
*** nmagnezi_ has quit IRC20:55
tmcpeakAJaeger: I took 0.17.0 that was released in December20:55
AJaegergive me a few minuts to check something...20:55
tmcpeakcherry picked the commit for one new feature, and released 0.17.1 from that20:55
clarkbdid you push a commit that is not in gerrit?20:56
tmcpeakso there are a bunch of things that happened since December that 0.17.1 doesn't have, but it does have a fairly recent commit20:56
clarkber push a tag for a commit20:56
AJaegertmcpeak: git log 0.17.0..HEAD |grep c364408faa25ccdc378b72c0a9e2150d20a94cba20:57
*** david_cu has joined #openstack-infra20:57
AJaegerThat finds nothing for me.20:57
AJaegerBut c36... is the one you tagged20:57
AJaegerAnd that's on the fresh checkout. So, you tagged something that is not in master it seems20:57
tmcpeakclarkb: no, the commit that I cherry-picked is in Gerrit20:57
AJaegeron the other hand, https://review.openstack.org/#/c/271465/ merged20:58
clarkbyes but the resulting commit isnt the same commit20:58
clarkbcherry picks alwayd make a new commit20:58
*** matrohon has joined #openstack-infra20:58
AJaegertmcpeak: check 271465, it has a different commit ID then what you tagged20:58
clarkbso uf you did not push that commit to gerrit and get it merged that may be the issue20:58
tmcpeakclarkb: ahh20:58
AJaegertmcpeak: start with a clean repo again: git reset --hard origin/master20:59
AJaegerand tag 0.17.2 now, don't reuse 0.17.1...20:59
*** hdd has quit IRC20:59
tmcpeakok, then we'll have to create a branch I suppose20:59
AJaegertmcpeak: ah, you want to apply a single patch on top of 0.17.0? ;(20:59
tmcpeakAJaeger: ok, can I just delete 0.17.1?20:59
AJaegeruse a branch...20:59
AJaegertmcpeak: you cannot20:59
tmcpeakAJaeger: ok, cool, I'll give that a shot21:00
AJaegertmcpeak: write up first what you do and have it sanity-checked by clarkb or others...21:00
tmcpeakI'll leave 0.17.1, shouldn't hurt anything21:00
tmcpeakshould it?21:00
* AJaeger has to leave now...21:00
*** salv-orlando has quit IRC21:00
tmcpeakAJaeger: ok, sounds good, thanks for all the help!21:00
AJaegerclarkb: thanks!21:00
*** daemontool has joined #openstack-infra21:00
*** daemontool_ has quit IRC21:03
tmcpeakclarkb: ok so what's the best way to get out of this cleanly?21:05
tmcpeakif you have time please21:05
tmcpeakI'm lost :)21:05
*** diazjf has joined #openstack-infra21:06
*** BobH_ has joined #openstack-infra21:06
clarkbyou have to run a change through code review, get it merged and tag that commit21:06
*** derekh has joined #openstack-infra21:06
BobH_Greetings. Having an issue with a devstack I just got and running tempest.21:07
clarkbyou cant push arbitrary commits to gerrit21:07
BobH_Anyone see this while running run_tempst.sh -s: RuntimeError: no suitable implementation for this system, looks like monotonic is throwing an exception21:07
*** daemontool_ has joined #openstack-infra21:07
jprgreghaynes: I added another log of a build from a non-nodepool account. don't know why this makes a difference but it definitely succeeds. https://bugs.launchpad.net/diskimage-builder/+bug/1538135/+attachment/4557107/+files/dib-cli-image.log.gz21:07
openstackLaunchpad bug 1538135 in diskimage-builder "disk-image-create deleting system /dev during image create " [Undecided,New]21:07
tmcpeakclarkb: ok, so I'll need to make a new branch for this? I can't just do a tag in master?21:07
*** Sukhdev has quit IRC21:07
clarkbyou can do a tag on master, it just has to me merged from.gerrits perpsective21:08
*** esker has joined #openstack-infra21:08
tmcpeakclarkb: ok I have a bunch of commits that have happened from the time I'm trying to cherry pick against, so practically I'll probably need a branch, huh?21:09
*** jsavak has quit IRC21:09
tmcpeakclarkb: also I can't remove this tag I screwed up?21:09
*** zz_ja has joined #openstack-infra21:09
jeblairgreghaynes, mordred, nibalizer: okay, i'm going to make the changes to nodepool.o.o now21:10
*** jsavak has joined #openstack-infra21:10
*** aeng has joined #openstack-infra21:10
*** daemontool has quit IRC21:10
mordredjeblair: I'm excited21:11
*** ccrouch has joined #openstack-infra21:11
*** amrith is now known as _amrith_21:11
jeblairthanks to getting the secrets out of there, that basically just means copy it from project-config :)21:12
mordredjeblair: woot!21:12
clarkbtmcpeak: if you cant tag a current commit on master then yes21:13
*** ashleighfarnham has quit IRC21:13
tmcpeakclarkb: ok cool, fair enough21:13
prometheanfiremordred: works once I switch them around21:13
*** piet has quit IRC21:13
*** esikachev has joined #openstack-infra21:14
jeblairgreghaynes: http://paste.openstack.org/show/485080/21:14
*** kgiusti has left #openstack-infra21:17
jeblairgreghaynes: (i rm-rfd that dir by hand to keep it moving)21:18
AJaegertmcpeak: could you read http://docs.openstack.org/infra/manual/drivers.html and tell us if there's anything missing that would have avoided the problems you have, please? Changes are the best way to improve ...21:19
* AJaeger really waves good-bye now21:19
*** sdake has quit IRC21:19
tmcpeakAJaeger: ok, will do.  thank you21:20
prometheanfiremordred: I'm going to submit this, let me know if you find anything, it's kinda large... http://paste.openstack.org/show/48508221:20
*** ashleighfarnham has joined #openstack-infra21:21
*** esikachev has quit IRC21:21
*** hdd has joined #openstack-infra21:23
*** boris-42 has quit IRC21:23
*** matrohon has quit IRC21:24
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Protect deleteNode from missing images  https://review.openstack.org/27274221:25
*** matrohon has joined #openstack-infra21:26
*** bpokorny_ has quit IRC21:26
jeblairmordred: NotFound: No Image matching {'name': 'CentOS 7'}. (HTTP 404)21:27
*** achanda has joined #openstack-infra21:27
jeblairmordred: (apparently that is not in bhs1)21:27
*** bpokorny has joined #openstack-infra21:27
mordredjeblair: piddle - I may have capsed wrong21:27
*** rcernin has quit IRC21:28
mordredjeblair: one sec21:28
mordredjeblair: p. "Centos 7"21:28
mordredjeblair: I'll send patch21:28
*** bpokorny has quit IRC21:28
*** bpokorny has joined #openstack-infra21:29
jeblairmordred: k, i've manually updated, will let you know if i see anything else21:29
*** hdd has quit IRC21:29
*** salv-orlando has joined #openstack-infra21:30
*** achanda has quit IRC21:30
*** julim has quit IRC21:30
jeblairweird.  at one point nodepool could delete images without providers, but not any more21:30
jeblair(it still has the error messag saying it's going to delete an image with no provider)21:31
*** tiswanso has quit IRC21:32
*** jcoufal_ has quit IRC21:32
*** tiswanso has joined #openstack-infra21:33
*** vgridnev has quit IRC21:33
*** Sukhdev has joined #openstack-infra21:34
openstackgerritMonty Taylor proposed openstack-infra/project-config: Change capitalization of CentOS 7 to Centos 7  https://review.openstack.org/27274421:35
*** baoli has joined #openstack-infra21:35
*** piet has joined #openstack-infra21:35
*** gtmanfred is now known as dontcare21:36
*** sridhar_ram has quit IRC21:36
*** trown is now known as trown|outttypeww21:37
ianwmordred / jeblair : so thanks for getting these snapshot builds working.  FYI i've been working on centos-minimal dib builds which should work everywhere.  the issue with the dib images previously was that they weren't using glean and didn't do networking properly on rax.  but that should be fixed now21:37
jeblairianw: oh great21:37
jeblairianw: (i don't know that we have it working, btw)21:37
jeblairianw: we just wanted to try to have something in place for shutting down hpcloud21:38
*** baoli_ has joined #openstack-infra21:38
ianwyeah, mea culpa on not getting to this.  i've been running with f23 and dropped ball on the hp retirement21:38
jeblairianw: hopefully soon we'll have an answer about whether centos7 dib works on ovh21:38
ianwand sidetracked with things like pip breaking the world21:38
ianwit *should*, if dhcp works21:38
jeblairif so, we can drop the snapshot there, then look at rax21:38
jeblairianw: i don't think the last 2 weeks have gone as anyone expected21:39
*** jaosorior_ has quit IRC21:39
*** hashar has quit IRC21:40
*** baoli has quit IRC21:40
jeblairmordred: our image lists are _much_ shorter now21:40
*** baoli_ has quit IRC21:41
mordredjeblair: yah - the 5-different-hps was quite multiplicative wasn't it?21:41
*** baoli has joined #openstack-infra21:41
*** ayoung has quit IRC21:51
jeblairokay, i think things are settling down21:51
jeblairhpcloud is only spinning up centos7 nodes, and the others are winding down21:51
openstackgerritMerged openstack-infra/project-config: Change capitalization of CentOS 7 to Centos 7  https://review.openstack.org/27274421:51
*** alivigni has quit IRC21:53
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Fix typo in debug log  https://review.openstack.org/27275021:54
nibalizerjeblair: cool21:56
*** e0ne has quit IRC21:57
*** ayoung has joined #openstack-infra21:58
*** yamahata has joined #openstack-infra21:59
*** yamahata has quit IRC22:01
*** sdake has joined #openstack-infra22:01
greghaynesjeblair: ah, I forgbot about that delete bug with all the other breakage22:01
*** yamahata has joined #openstack-infra22:01
*** ccrouch has quit IRC22:02
greghaynesjpr: I dont understand - what change made your build succeed?22:02
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Clarify debug statements in image-upload  https://review.openstack.org/27275422:03
tmcpeakclarkb: still around?22:03
clarkbtmcpeak: sort of22:03
jprgreghaynes: i'm trying to figure that out.  about to run the same commands from the nodepool account so I can diff22:03
*** erikwilson has quit IRC22:03
greghaynesjpr: ah, ok22:03
tmcpeakclarkb: how do I create a new branch for Bandit?22:03
tmcpeakI'd know how to do it if it wasn't gerrit controlled22:04
*** ccrouch has joined #openstack-infra22:04
*** FallenPegasus has joined #openstack-infra22:04
*** tonytan4ever has quit IRC22:05
nibalizerjeblair: i don't know if you saw, but we dropped tempest tests to 78 minutes with a 16vcpu omfra cloud node22:05
jeblairhrm, now it's trying to upload a dib image that has been deleted22:05
jeblairnibalizer: hrm, any idea what the bottleneck is?22:05
clarkbtmcpeak: looks like someone from the release team will have to do it based on your acl22:05
tmcpeakok fair enough22:05
*** gokrokve has quit IRC22:06
clarkbtmcpeak: this should be documented in the infra manual let me get you a link22:06
*** salv-orl_ has joined #openstack-infra22:06
tmcpeakclarkb: thank you22:06
nibalizerjeblair: nope22:06
nibalizerbut d-g is largely a black box to me22:06
jeblairnibalizer: you think the delay is in d-g?22:06
*** regXboi has quit IRC22:06
tmcpeakclarkb: I found it, thank you22:06
*** diazjf has quit IRC22:07
nibalizerjeblair: i mean my understanding is pretty limited22:07
jeblairnibalizer: if you want to separate job setup time from actual tempest runtime, tempest outputs the total time spent running tests at the end22:08
nibalizerI couldn't back of the hand walk you through what safe-devstack-vm-gate-wrap.sh does22:08
*** Daisy has joined #openstack-infra22:08
jeblairnibalizer: i'd start with that -- compare the runtime of tempest and see if the bulk of the extra time as compared to other providers is job setup or test execution22:08
*** salv-orlando has quit IRC22:09
*** dprince has quit IRC22:09
*** diazjf has joined #openstack-infra22:10
jeblairnibalizer: (i'm assuming it's execution because the host is slow, but best to double check that first.  also, double check that tempest ran 8 parallel processes (it's supposed to do #cpus/2))22:10
jeblairnibalizer: (it outputs that info in the same place as the runtime)22:10
*** bexelbie has quit IRC22:10
openstackgerritgreghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support  https://review.openstack.org/27238422:11
greghaynesjeblair: did you see ^22:11
jeblairgreghaynes: not yet22:11
openstackgerritBrant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job  https://review.openstack.org/26499122:11
*** burgerk has quit IRC22:12
*** Daisy has quit IRC22:13
*** shashank_hegde has joined #openstack-infra22:14
jeblairgreghaynes, nibalizer, mordred, ianw: the centos7 dib image state on nodepool was missing the actual image files, so was unable to upload anything to ovh.  i have deleted them and it just kicked off a dib build of centos722:15
mordredjeblair: neat22:15
jeblairianw, mordred: i don't think we updated the logging config when we added the centos7 snapshot to ovh, so that's going to a black hole right now22:16
openstackgerritJames E. Blair proposed openstack-infra/system-config: Update nodepool log config  https://review.openstack.org/27275722:17
*** jsavak has quit IRC22:17
jeblairianw, mordred: ^22:17
jeblairnibalizer: i think clarkb said something about this change needing an update for a pidfile: https://review.openstack.org/27154122:17
*** dontcare is now known as gtmanfred22:17
*** sridhar_ram has joined #openstack-infra22:18
mordredjeblair: crappit22:18
jeblairnibalizer: but he didn't leave a comment on the review, so i'm not positive22:18
*** jsavak has joined #openstack-infra22:18
greghaynesthere was an issue where nodepool-builder and nodepoold attempted to sharea the same pid file22:18
greghaynesI have no idea whether that was resolved or not22:19
*** matrohon has quit IRC22:19
nibalizerjeblair: looking at it we set NAME to nodepool-builder and set PIDFILE to $NAME.pid22:19
nibalizerso I think we're ok22:19
clarkbnibalizer: jeblair the pidfile thing is fixed22:19
clarkbthe current thing is we need to split the logfile confogs22:20
jeblairclarkb: has spoken! :)22:20
nibalizerclarkb: back to drinking you! /me swats22:20
jeblairclarkb: a comment in gerrit to that effect would be swell :)22:21
*** sdake has quit IRC22:21
*** erikwilson has joined #openstack-infra22:21
ianwjeblair: should be going to image.log22:23
jeblairianw: ah indeed thanks22:23
jeblair2016-01-26 21:45:16,837 INFO nodepool.image.build.ovh-gra1.devstack-centos7: You are required to change your password immediately (password aged)22:23
jeblair2016-01-26 21:45:16,842 INFO nodepool.image.build.ovh-gra1.devstack-centos7: Changing password for root.22:23
*** erikwilson has quit IRC22:23
*** daemontool has joined #openstack-infra22:23
jeblairianw, mordred: i'm going to say that getting snapshots to work on ovh will pose a whole new set of challenges.22:24
*** davidlenwell has quit IRC22:24
jeblairianw, mordred: i believe we should abandon them and focus only on dib22:24
mordredjeblair: I agree22:24
*** ashleighfarnham has quit IRC22:24
*** calebb has quit IRC22:24
ianwfair enough.  with ovh, i haven't got an account yet as they require a prepayment on the credit card22:25
*** davidlenwell has joined #openstack-infra22:25
jeblairianw: i will let you know the results of the dib build on ovh as soon as its ready22:25
jeblairianw: do you think we should try uploading dib to rax now too?22:26
ianwjeblair: no, it won't work because it's not using glean/growroot etc.  so it will try to dhcp and get no address22:26
*** rlandy has quit IRC22:26
ianwi think using centos-minimal is the way forward, but i've been having issues with that i'm working on22:26
*** erlon has quit IRC22:26
jeblairso fingers crossed for centos7-dib on ovh then?22:26
*** derekh has quit IRC22:27
*** daemontool_ has quit IRC22:27
ianwi think that should work, because it was working on hpcloud with dhcp22:27
*** rlandy has joined #openstack-infra22:27
openstackgerritJames E. Blair proposed openstack-infra/project-config: Remove centos7 snapshots from ovh  https://review.openstack.org/27276122:30
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add flavor name filter to centos7-dib ovh  https://review.openstack.org/27276222:30
jeblairianw, mordred, nibalizer: ^22:30
openstackgerritJames E. Blair proposed openstack-infra/system-config: Further update nodepool log config  https://review.openstack.org/27276322:31
*** EricGonc_ has quit IRC22:31
*** gokrokve has joined #openstack-infra22:31
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215622:32
nibalizerjeblair: +2, can +a if you have trouble finding one22:32
*** sabeen3 has quit IRC22:35
*** jsavak has quit IRC22:35
zaronibalizer: maybe i don't need the logs, just some puppet help.22:36
*** gokrokve has quit IRC22:36
nibalizerzaro: sure how can I help22:36
*** jsavak has joined #openstack-infra22:36
zaronibalizer: puppet report says this gets created on every puppet check, https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/init.pp#n45122:36
nibalizeroh yea22:36
*** sabeen has joined #openstack-infra22:36
nibalizerone of the interns fixed that in the zuul mdule22:37
zarotrying to figure out why?22:37
zaroref to fix?22:37
nibalizertimothyb89: iirc22:37
*** abregman has quit IRC22:37
zarodid it fix the issue?22:38
*** sdake has joined #openstack-infra22:38
zarocool, thanks!22:38
jprgreghaynes: posted a log from the cli run of dib in the nodepool account that might help point to the issue.  the differences start to appear around the point the image is populated, in the vicinity of the mv -t command22:39
*** sabeen3 has joined #openstack-infra22:39
*** e0ne has joined #openstack-infra22:39
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases  https://review.openstack.org/27276722:39
openstackgerritgreghaynes proposed openstack-infra/nodepool: Don't mistake manifest dir's for images  https://review.openstack.org/27276822:39
greghaynesjeblair: ^ fix for delete failures22:39
*** tphummel has quit IRC22:39
*** daemontool_ has joined #openstack-infra22:39
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add a job to send automated announcements of releases  https://review.openstack.org/27276722:40
jeblairgreghaynes: thx22:40
*** tphummel has joined #openstack-infra22:40
*** ashleighfarnham has joined #openstack-infra22:40
*** sabeen has quit IRC22:41
*** dkranz has quit IRC22:42
*** daemontool has quit IRC22:42
*** thorst_ has joined #openstack-infra22:44
*** sdake has quit IRC22:46
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Tell puppet to check for cgid instead of cgi  https://review.openstack.org/27276922:47
*** tonytan4ever has joined #openstack-infra22:47
*** tiswanso has quit IRC22:47
*** calebb has joined #openstack-infra22:47
*** thorst_ has quit IRC22:48
jeblairgreghaynes: http://paste.openstack.org/show/485087/22:49
openstackgerritMerged openstack-infra/nodepool: Protect deleteNode from missing images  https://review.openstack.org/27274222:49
greghaynesjpr: hrm, looked at the log, that one looks like it succeeded?22:49
jeblairgreghaynes: that was the error that led me to delete the dib images in the first place; maybe there was no problem22:49
*** ccrouch has quit IRC22:49
jeblairother than whatever that bug is22:49
openstackgerritMerged openstack-infra/project-config: Remove centos7 snapshots from ovh  https://review.openstack.org/27276122:50
openstackgerritMerged openstack-infra/project-config: Add flavor name filter to centos7-dib ovh  https://review.openstack.org/27276222:50
jprgreghaynes: the one prefixed with nodepool- took my system /dev away22:50
*** sdake has joined #openstack-infra22:50
*** amitgandhinz has quit IRC22:50
greghaynesjpr: oh, I just have dib-cli-image.log, where is the nodepool- one?22:51
zaroclarkb: that reply to jenkins bug was pretty confusing, you should just make nodepool use the entire python-jenkins library.22:52
*** weshay has quit IRC22:52
jprgreghaynes: hmm, I've attached a dib-cli-image-2.log and nodepool-dib-cli-image.log to the ticket.22:53
jprwill follow up posts22:53
*** tmcpeak has left #openstack-infra22:53
jprthe last two are the identical commands save for the paths for the accounts22:54
jpr*with follow up posts22:54
*** boris-42 has joined #openstack-infra22:54
greghaynesjpr: ah, I see now22:54
*** dims has joined #openstack-infra22:55
openstackgerritDoug Fish proposed openstack-infra/project-config: Add JavaScript gate jobs to LBaaS dashboard  https://review.openstack.org/27277122:55
greghaynesjpr: I have one more request if you have a bit of time22:55
greghaynesjpr: Can you modify the disk-image-create executable and add a set -x at the top of it, then rerun nodepool and grab that log22:56
greghaynesjpr: that should add a ton of debug info22:56
jprsure.  working on a parallel effort so I may bounce a bit22:56
zaroclarkb: gerrit add reviewer takes forever to suggest your name for some reason.22:57
*** ccrouch has joined #openstack-infra22:57
*** achanda has joined #openstack-infra22:58
*** tonytan4ever has quit IRC22:59
*** doug-fish has quit IRC23:00
*** sdake has quit IRC23:01
openstackgerritAthlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter  https://review.openstack.org/27215623:01
*** e0ne has quit IRC23:02
jeblairgreghaynes: ah, that error is because we are looking for a raw but we only have qcow223:02
greghaynesjeblair: which error?23:03
greghaynesso many errors today23:03
jeblairBuilderInvalidCommandError: Unable to find image file for id 176212 to upload23:03
greghaynesah, makes sense23:04
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Add more debug logging to image upload  https://review.openstack.org/27277523:06
jeblairlogging i added to discover that ^23:07
jeblairokay, so now -- why isn't there a raw image there?23:07
greghaynesjeblair: who consumes raw images?23:08
jeblairgreghaynes: ovh apparently23:09
nibalizermordred: I think puppetdb is ooming23:09
nibalizeris why it keeps restarting23:09
greghaynesand we didnt add that provider recently, correct? There is an issue where if you build before you add the provider, then you try to upload the image it will not have built all the needed image types23:09
nibalizerJAVA_ARGS="-Xmx192m that makes sense23:10
greghaynesI have to afk for a bit23:10
jeblairgreghaynes: this image build is brand new, after the provider (and the image was added to that provider)23:10
nibalizerthats the java I know23:10
jeblairgreghaynes: however...23:10
jeblairgreghaynes: is that in the builder now?23:10
jeblairgreghaynes: and does the builder ever reload the config?23:10
jeblairgreghaynes: perhaps the builder was out of sync with the nodepool config23:11
greghaynesjeblair: oh. it doesnt reload config23:11
*** tongli has quit IRC23:11
greghaynesthat thought had never occured to me23:11
greghaynesok, so new feature to work on23:11
jeblairokay, i'm not going to fix that now.  i'm going to rely on the fact that i have since restarted the builder, and then delete that dib image and let it build it again23:11
greghaynesYep, I can poke at that when I return23:11
*** baoli has quit IRC23:12
*** baoli has joined #openstack-infra23:13
*** sdake has joined #openstack-infra23:13
*** tongli has joined #openstack-infra23:13
jeblair2016-01-26 23:14:26,108 INFO nodepool.builder: Running disk-image-create -x -t raw --no-tmpfs  -o /opt/nodepool_dib/176213 centos7 vm openstack-repos nodepool-base node-devstack cache-bindep23:14
jeblairlooks better23:14
*** zz_dimtruck is now known as dimtruck23:14
*** Sukhdev has quit IRC23:17
*** jsavak has quit IRC23:19
*** notnownikki has quit IRC23:21
openstackgerritMerged openstack/requirements: update to oslo.config 3.4.0  https://review.openstack.org/27260723:21
*** diazjf has quit IRC23:22
*** calebb has quit IRC23:24
ianwjeblair / clarkb : all git hosts now centos7 right?23:25
ianwjust looking at some system-config bits and old work-arounds we had in there23:25
*** gokrokve has joined #openstack-infra23:28
*** sigmavirus24 is now known as sigmavirus24_awa23:28
*** darrenc is now known as darrenc_afk23:28
*** derekh has joined #openstack-infra23:29
*** sdake has quit IRC23:31
*** gokrokve has quit IRC23:33
*** jcoufal has joined #openstack-infra23:33
*** darrenc_afk is now known as darrenc23:36
*** dimtruck is now known as zz_dimtruck23:37
*** ganso has quit IRC23:39
*** sabeen3 has quit IRC23:39
*** piet has quit IRC23:39
jprgreghaynes: i've added the log with additional debug23:41
jeblairianw: yes23:41
jeblairianw: we're uploading the centos7-dib images to ovh now23:42
*** ashleighfarnham has quit IRC23:43
*** rajinir has quit IRC23:45
*** nelsnelson has quit IRC23:47
*** nelsnelson has joined #openstack-infra23:47
*** zz_dimtruck is now known as dimtruck23:47
*** rbrndt_ has quit IRC23:49
*** rbrndt has joined #openstack-infra23:50
*** Qiming has joined #openstack-infra23:50
*** vponomaryov has quit IRC23:52
*** vponomaryov has joined #openstack-infra23:53
*** flepied has quit IRC23:54
*** diana_clarke has joined #openstack-infra23:54
openstackgerritShu Muto proposed openstack/os-testr: remove python 3.3 trove classifier  https://review.openstack.org/26452123:55
*** ashleighfarnham has joined #openstack-infra23:57
*** yamamoto_ has joined #openstack-infra23:58
*** austin81 has left #openstack-infra23:58
*** achanda has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!