| *** xchu has joined #openstack-infra | 00:00 | |
| mordred | clarkb: so - I'm going to have to spin up a vm to work further on dib - I've managed to tickle a bug in dib by running a debian host to build ubuntu images, which seems to be a combination that is not (yet) tested | 00:04 |
|---|---|---|
| mordred | clarkb: seems like it should be a straightforward bug - but I'd rather make progress on teh task at hand... | 00:04 |
| clarkb | right you are feeling why chroots wont work well for the general case :) | 00:04 |
| mordred | clarkb: nod. otoh - did you know that debian does per-user tmp dirs? | 00:05 |
| mordred | clarkb: TMP=/tmp/user/1000 | 00:05 |
| clarkb | I did not know | 00:05 |
| mordred | me either | 00:05 |
| clarkb | makes sense though | 00:05 |
| lifeless | mordred: have you filed a bug for me ? | 00:06 |
| mordred | lifeless: no, I have not | 00:06 |
| *** oomichi has joined #openstack-infra | 00:06 | |
| mordred | lifeless: sorry, I forget you like those :) | 00:06 |
| lifeless | mordred: are you going to do so right now? Or do I have to fly over there and sit on you until you do ? | 00:06 |
| mordred | doing | 00:06 |
| lifeless | good choice :) | 00:07 |
| *** yamahata has quit IRC | 00:08 | |
| *** jhesketh has joined #openstack-infra | 00:08 | |
| mordred | lifeless: https://bugs.launchpad.net/diskimage-builder/+bug/1330290 | 00:09 |
| uvirtbot | Launchpad bug 1330290 in diskimage-builder "Building ubuntu images on debian host fails because of tmpdir" [Undecided,New] | 00:09 |
| lifeless | thanks | 00:09 |
| mordred | lifeless: thank you! I do not believe I would have noticed the madness in the dir path | 00:09 |
| *** ianw has quit IRC | 00:26 | |
| *** ianw has joined #openstack-infra | 00:26 | |
| openstackgerrit | A change was merged to openstack-infra/gitdm: Update gitdm to use gerrit 2.8 workflow names https://review.openstack.org/99466 | 00:27 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Create user to catch email if $sysadmin == [] https://review.openstack.org/100118 | 00:34 |
| *** matsuhashi has joined #openstack-infra | 00:35 | |
| *** matsuhashi has quit IRC | 00:35 | |
| *** matsuhashi has joined #openstack-infra | 00:35 | |
| *** sarob_ has joined #openstack-infra | 00:38 | |
| *** sarob_ has quit IRC | 00:42 | |
| anteaya | upgraded the lappie to trusty, so far nothing has gone boom | 00:42 |
| StevenK | anteaya: I currently have an upgrade running on my desktop. | 00:47 |
| StevenK | So far the only thing broken while the upgrade is running is some fonts in Firefox. | 00:48 |
| anteaya | cool | 00:51 |
| anteaya | I had something with the gtk pixbuf or something | 00:51 |
| anteaya | keep seeing that error fly by | 00:51 |
| anteaya | so far nothing in the install feels buggy | 00:51 |
| anteaya | *shrug* | 00:51 |
| *** nosnos has joined #openstack-infra | 00:55 | |
| morganfainberg | anteaya, yeah the only issue i had w/ trusty on the laptop was EFI boot (but this was a clean install) | 01:04 |
| *** Ryan_Lane has joined #openstack-infra | 01:04 | |
| *** CaptTofu_ has joined #openstack-infra | 01:04 | |
| *** Ryan_Lane has quit IRC | 01:08 | |
| anteaya | efi? I _think_ I am still bios | 01:09 |
| anteaya | did you get it fixed? | 01:09 |
| *** melwitt has joined #openstack-infra | 01:10 | |
| *** wenlock_ has joined #openstack-infra | 01:11 | |
| morganfainberg | anteaya, yeah i used the Mac boot ISO, | 01:13 |
| morganfainberg | anteaya, this laptop didn't want to let me install in 'bios' mode (just outright failed) | 01:13 |
| morganfainberg | anteaya, took me a couple extra hours when i was installing one weekend, but it works well once using the right iso | 01:14 |
| anteaya | which was the right iso for you? | 01:14 |
| morganfainberg | anteaya, http://cdimage.ubuntu.com/releases/14.04/release/ubuntu-14.04-desktop-amd64+mac.iso | 01:15 |
| anteaya | cool | 01:15 |
| morganfainberg | anteaya, but if yours is working inbios mode, i prob. wouldn't switch it up :) | 01:16 |
| anteaya | yeah, just did an inplace upgrade | 01:18 |
| anteaya | so far so good | 01:18 |
| morganfainberg | my experience is 2 OSs really succeed at that well | 01:18 |
| morganfainberg | ubuntu | 01:18 |
| morganfainberg | and os X | 01:18 |
| morganfainberg | most anything else I find a reinstall to be better | 01:18 |
| * anteaya nods | 01:21 | |
| anteaya | never driven os X | 01:21 |
| anteaya | so I have no experience | 01:22 |
| morganfainberg | i like it, but that's because i've been using it for a looong time (and desktop linux has only recently been really viable) | 01:22 |
| morganfainberg | recently = last ~4yrs maybe | 01:22 |
| morganfainberg | imo | 01:22 |
| anteaya | kk | 01:24 |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Filter out WIP patches by default in changes list. https://review.openstack.org/100121 | 01:26 |
| lifeless | anteaya: if you have an HP laptop it can do either EFI or bios boot mode - check your bios to see | 01:28 |
| anteaya | k, thanks | 01:29 |
| anteaya | next boot up | 01:29 |
| *** wenlock_ has quit IRC | 01:36 | |
| mnaser | maybe this isnt much of an infra issue but someone with experience might chime in | 01:37 |
| *** CaptTofu_ has quit IRC | 01:37 | |
| mnaser | anyone ever ran 14.04 with havana? or should i keep installing 12.04 till i get icehouse in | 01:37 |
| *** CaptTofu_ has joined #openstack-infra | 01:37 | |
| anteaya | havana was tested with 12.04 | 01:39 |
| anteaya | I don't believe we test past releases with new versions of os's | 01:40 |
| mnaser | i don't see why it wouldn't work, but i assume the deps/etc probably will result in the install not going through | 01:40 |
| mnaser | this makes it hard, oh well | 01:40 |
| anteaya | it is possible | 01:40 |
| anteaya | I would definitely spend some time testing it | 01:40 |
| anteaya | and be prepared for it not to work | 01:40 |
| *** alugovoi has joined #openstack-infra | 01:41 | |
| mnaser | anteaya: in that case, rather not deal with that type of trouble.. better spend resources keeping things running | 01:41 |
| * anteaya nods | 01:41 | |
| anteaya | you sound like a wise person | 01:41 |
| *** CaptTofu_ has quit IRC | 01:42 | |
| *** sweston has joined #openstack-infra | 01:42 | |
| anteaya | keep in mind we are still testing everything on precise, or centos for py26 and perhaps something else | 01:43 |
| anteaya | infra has not moved to trusty yet for testing | 01:43 |
| anteaya | we want to, but it hasn't happened yet | 01:43 |
| morganfainberg | anteaya, i use VMs for all testing | 01:45 |
| *** blamar has joined #openstack-infra | 01:45 | |
| morganfainberg | and vagrant (usually) | 01:45 |
| morganfainberg | it's way easier to control the environment and not pollute my workstation/laptop's install | 01:45 |
| *** pcrews has quit IRC | 01:45 | |
| *** blamar has quit IRC | 01:47 | |
| morganfainberg | mnaser, i would even recommend staying with 12.04 until Juno (current development cycle) simply because 12.04 has a lot of drive time and as anteaya said, infra doesn't test on trusty | 01:48 |
| *** alexpilotti has joined #openstack-infra | 01:48 | |
| mnaser | morganfainberg: hmm, sounds pretty good actaully, but, my concern here might be what to do when the next cycle comes in | 01:49 |
| mnaser | not sure how easy it is to go from 12.04 to 14.04 | 01:49 |
| mnaser | (ex: if this is centos 5 -> centos 6, it's a big problem :p) | 01:49 |
| morganfainberg | mnaser, i've had great luck upgrading ubuntu. but it also depends on how you're deploying OpenStack. | 01:49 |
| mnaser | for example, centos 5 to 6 is completely unsupported and "impossible" to put in other terms | 01:50 |
| mnaser | i think upgrading to 14.04 from 12.04 is something that can be done smoothly | 01:50 |
| morganfainberg | mnaser, previous job *** we would quiesce a hypervisor, then install on the new OS as part of the upgrade. so in the case of when we did os upgrades, we made sure OpenStack would run on both, then upgraded each hypervisor | 01:50 |
| morganfainberg | mnaser, but that was because we had very controlled package lists we wanted per release | 01:51 |
| mnaser | well, i'm planning to kinda roll out with puppet packages / ubuntu repos | 01:51 |
| *** sarob_ has joined #openstack-infra | 01:51 | |
| mnaser | life should be much simpler then | 01:51 |
| morganfainberg | mnaser, doesn't mean i wouldn't evacuate the hypervisor then reinstall w/ the latest ubuntu release then. (personal preference if I were to be deploying that manner) | 01:52 |
| morganfainberg | mnaser, but i've def. seen good upgrade paths for ubuntu | 01:52 |
| mnaser | morganfainberg: im thinking continue with 12.04, then when juno is out, upgrade to 14.04 first, then roll out juno after | 01:52 |
| mnaser | and i agree with you.. it's been pretty smooth | 01:53 |
| morganfainberg | mnaser, provided we get some good drive time testing on 14.04 in the check/gate, that would be my approach as well | 01:53 |
| morganfainberg | it isn't like 12.04 will become obsolete overnight or support juno badly if we don't get the drive time on the testing | 01:53 |
| morganfainberg | for 14.04 that is. | 01:54 |
| *** koolhead17 has joined #openstack-infra | 01:54 | |
| * morganfainberg re-reads that sentence and ... has no idea how to make it more gramatically correct - sorry for bad typing... it's sunday and i'm fighting with finding good flight times for travel :) | 01:54 | |
| mnaser | morganfainberg / anteaya: thank you very much for bouncing the ideas around.. and thanks for your help 😊 | 01:55 |
| morganfainberg | mnaser, of course! happy to help~ | 01:55 |
| morganfainberg | ! | 01:55 |
| * mnaser has to do migrations | 01:55 | |
| mnaser | migrations to kvm, then migrating from nova-network to quantum/neutron | 01:55 |
| mnaser | over 500 vms | 01:55 |
| *** sarob_ has quit IRC | 01:55 | |
| anteaya | mnaser: may they go smoothly | 01:56 |
| mnaser | heh, thank you :) | 01:57 |
| mnaser | i'm worried.. but i think it'll go over smoothly | 01:57 |
| *** sweston has quit IRC | 01:58 | |
| anteaya | positive outlook, off to a good start | 01:59 |
| *** zhiyan_ is now known as zhiyan | 02:00 | |
| *** aysyd has joined #openstack-infra | 02:10 | |
| *** zhiyan is now known as zhiyan_ | 02:18 | |
| *** praneshp_ has joined #openstack-infra | 02:24 | |
| *** praneshp has quit IRC | 02:26 | |
| *** praneshp_ is now known as praneshp | 02:26 | |
| *** zhiyan_ is now known as zhiyan | 02:27 | |
| *** fifieldt__ is now known as fifieldt | 02:30 | |
| *** arnaud__ has joined #openstack-infra | 02:33 | |
| *** alugovoi has quit IRC | 02:34 | |
| *** penguinRaider has quit IRC | 02:34 | |
| *** yamahata has joined #openstack-infra | 02:36 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Create user to catch email if $sysadmin == [] https://review.openstack.org/100118 | 02:37 |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Create user to catch email if $sysadmin == [] https://review.openstack.org/100118 | 02:41 |
| *** arnaud__ has quit IRC | 02:45 | |
| openstackgerrit | James Polley proposed a change to openstack-infra/config: Add definitions for a job to check vlan config https://review.openstack.org/100076 | 02:45 |
| *** alexpilotti has quit IRC | 02:46 | |
| *** sarob_ has joined #openstack-infra | 02:51 | |
| *** CaptTofu_ has joined #openstack-infra | 02:53 | |
| *** sarob_ has quit IRC | 02:56 | |
| *** koolhead17 has quit IRC | 02:58 | |
| *** alexpilotti has joined #openstack-infra | 02:58 | |
| *** jhesketh has quit IRC | 03:13 | |
| *** alexpilotti has quit IRC | 03:16 | |
| *** adalbas has quit IRC | 03:16 | |
| *** shayneburgess has joined #openstack-infra | 03:16 | |
| anteaya | lifeless: thanks, I found it, uefi with or without csm, currently I am using the legacy bios, but good to know | 03:18 |
| *** jhesketh has joined #openstack-infra | 03:26 | |
| *** aysyd has quit IRC | 03:27 | |
| *** crc32 has joined #openstack-infra | 03:27 | |
| *** nosnos has quit IRC | 03:36 | |
| *** CaptTofu_ has quit IRC | 03:42 | |
| *** CaptTofu_ has joined #openstack-infra | 03:42 | |
| *** shayneburgess has quit IRC | 03:44 | |
| *** CaptTofu_ has quit IRC | 03:46 | |
| *** shayneburgess has joined #openstack-infra | 03:48 | |
| *** shayneburgess has quit IRC | 03:57 | |
| *** amotoki has joined #openstack-infra | 03:57 | |
| *** gema has quit IRC | 03:58 | |
| *** otter768 has quit IRC | 03:58 | |
| *** gema has joined #openstack-infra | 03:59 | |
| *** otter768 has joined #openstack-infra | 04:00 | |
| *** arnaud__ has joined #openstack-infra | 04:02 | |
| *** pcrews has joined #openstack-infra | 04:03 | |
| *** morganfainberg has quit IRC | 04:04 | |
| *** morganfainberg has joined #openstack-infra | 04:04 | |
| *** morganfainberg has quit IRC | 04:05 | |
| *** morganfainberg has joined #openstack-infra | 04:06 | |
| *** yfried_ has quit IRC | 04:07 | |
| *** pcrews has quit IRC | 04:09 | |
| *** otter768 has quit IRC | 04:14 | |
| *** arnaud__ has quit IRC | 04:15 | |
| *** nosnos has joined #openstack-infra | 04:15 | |
| *** yaguang has joined #openstack-infra | 04:15 | |
| *** melwitt has quit IRC | 04:15 | |
| *** arnaud__ has joined #openstack-infra | 04:24 | |
| *** mrodden has joined #openstack-infra | 04:25 | |
| *** crc32 has quit IRC | 04:29 | |
| *** crc32 has joined #openstack-infra | 04:30 | |
| *** signed8bit has joined #openstack-infra | 04:34 | |
| *** Longgeek has joined #openstack-infra | 04:39 | |
| *** Longgeek has quit IRC | 04:40 | |
| *** Longgeek has joined #openstack-infra | 04:40 | |
| *** arnaud__ has quit IRC | 04:42 | |
| *** crc32 has quit IRC | 04:43 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/devstack-gate: Make one copy of grenade log files https://review.openstack.org/100131 | 04:44 |
| openstackgerrit | Isaku Yamahata proposed a change to openstack-infra/config: Add tacker project on StackForge https://review.openstack.org/97435 | 04:47 |
| *** sarob_ has joined #openstack-infra | 04:51 | |
| *** sarob_ has quit IRC | 04:56 | |
| *** otherwiseguy has joined #openstack-infra | 04:57 | |
| *** yfried_ has joined #openstack-infra | 05:02 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add in support for 'recheck jenkins' https://review.openstack.org/100133 | 05:02 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/devstack-gate: Cleanup of grenade and old/new log copy https://review.openstack.org/99852 | 05:03 |
| *** signed8bit has quit IRC | 05:04 | |
| *** signed8bit has joined #openstack-infra | 05:05 | |
| *** otherwiseguy has quit IRC | 05:08 | |
| *** otherwiseguy has joined #openstack-infra | 05:12 | |
| *** koolhead17 has joined #openstack-infra | 05:13 | |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Don't crash on comments on unchanged files https://review.openstack.org/99563 | 05:14 |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Filter out WIP patches by default in changes list. https://review.openstack.org/100121 | 05:14 |
| openstackgerrit | lifeless proposed a change to stackforge/gertty: Only show verified reviews by default. https://review.openstack.org/100138 | 05:14 |
| *** terryw has joined #openstack-infra | 05:17 | |
| *** otherwiseguy has quit IRC | 05:20 | |
| *** e0ne has joined #openstack-infra | 05:21 | |
| *** ildikov has quit IRC | 05:21 | |
| *** sweston has joined #openstack-infra | 05:28 | |
| *** signed8bit has quit IRC | 05:34 | |
| *** e0ne has quit IRC | 05:37 | |
| *** basha has joined #openstack-infra | 05:44 | |
| *** ildikov has joined #openstack-infra | 05:48 | |
| *** sarob_ has joined #openstack-infra | 05:51 | |
| *** penguinRaider has joined #openstack-infra | 05:52 | |
| *** zhiyan is now known as zhiyan_ | 05:55 | |
| *** sarob_ has quit IRC | 05:55 | |
| *** zhiyan_ is now known as zhiyan | 05:56 | |
| *** matsuhashi has quit IRC | 06:00 | |
| *** matsuhashi has joined #openstack-infra | 06:01 | |
| *** _nadya_ has joined #openstack-infra | 06:05 | |
| *** e0ne has joined #openstack-infra | 06:07 | |
| *** _nadya_ has quit IRC | 06:15 | |
| *** Ryan_Lane has joined #openstack-infra | 06:16 | |
| *** e0ne has quit IRC | 06:18 | |
| *** arnaud__ has joined #openstack-infra | 06:18 | |
| *** e0ne has joined #openstack-infra | 06:18 | |
| *** arnaud__ has quit IRC | 06:18 | |
| *** penguinRaider has quit IRC | 06:21 | |
| *** arnaud__ has joined #openstack-infra | 06:21 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/devstack-gate: Cleanup of grenade and old/new log copy https://review.openstack.org/99852 | 06:23 |
| *** e0ne has quit IRC | 06:25 | |
| *** amotoki has quit IRC | 06:26 | |
| *** matsuhas_ has joined #openstack-infra | 06:38 | |
| *** alexpilotti has joined #openstack-infra | 06:38 | |
| *** alexpilotti has quit IRC | 06:38 | |
| *** jcoufal has joined #openstack-infra | 06:38 | |
| *** matsuhashi has quit IRC | 06:38 | |
| *** Guest47184 has joined #openstack-infra | 06:46 | |
| *** rgerganov_ has joined #openstack-infra | 06:47 | |
| *** dkehn_ has joined #openstack-infra | 06:54 | |
| *** ihrachyshka has joined #openstack-infra | 06:57 | |
| *** Clabbe has joined #openstack-infra | 06:58 | |
| *** dkehnx has quit IRC | 06:59 | |
| *** rdopieralski has joined #openstack-infra | 07:00 | |
| *** matsuhas_ has quit IRC | 07:01 | |
| *** vtapia has joined #openstack-infra | 07:01 | |
| *** yfried_ is now known as yfried | 07:03 | |
| *** _nadya_ has joined #openstack-infra | 07:04 | |
| *** penguinRaider has joined #openstack-infra | 07:04 | |
| *** matsuhashi has joined #openstack-infra | 07:04 | |
| *** e0ne has joined #openstack-infra | 07:04 | |
| *** chandan_kumar has quit IRC | 07:05 | |
| *** ociuhandu has joined #openstack-infra | 07:05 | |
| *** vtapia has left #openstack-infra | 07:05 | |
| *** amotoki has joined #openstack-infra | 07:06 | |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack-infra/config: Add XStatic-* projects with packaged static files for Horizon https://review.openstack.org/95716 | 07:10 |
| *** asettle has quit IRC | 07:12 | |
| ttx | reed: "other" is what's not being incubated or integrated. | 07:13 |
| *** rcarrillocruz has quit IRC | 07:13 | |
| *** rcarrillocruz has joined #openstack-infra | 07:13 | |
| *** e0ne has quit IRC | 07:14 | |
| *** Ryan_Lane has quit IRC | 07:15 | |
| *** Ryan_Lane has joined #openstack-infra | 07:15 | |
| *** basha has quit IRC | 07:15 | |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** _nadya_ has quit IRC | 07:16 | |
| *** pas-ha has joined #openstack-infra | 07:20 | |
| *** unicell has joined #openstack-infra | 07:21 | |
| *** flaper87|afk is now known as flaper87 | 07:24 | |
| *** penguinRaider has quit IRC | 07:25 | |
| *** andreykurilin_ has joined #openstack-infra | 07:27 | |
| *** freyes has joined #openstack-infra | 07:29 | |
| *** jaypipes has joined #openstack-infra | 07:36 | |
| *** jaypipes has quit IRC | 07:36 | |
| *** jlibosva has joined #openstack-infra | 07:38 | |
| *** zehicle_at_dell has quit IRC | 07:40 | |
| mattoliverau | I'm calling it a day, night all. | 07:41 |
| *** zehicle_at_dell has joined #openstack-infra | 07:41 | |
| *** Ryan_Lane has quit IRC | 07:42 | |
| *** tkelsey has joined #openstack-infra | 07:42 | |
| *** katyafervent_awa is now known as katyafervent | 07:42 | |
| pas-ha | hi all, is there any way to login to gerrint currently? launchpad openid is not working, and google's is also not accepted (OpenID provider not permitted by site policy.) | 07:44 |
| pas-ha | what are actually those permitted my site policy? | 07:45 |
| pas-ha | s/my/by/ | 07:45 |
| *** _nadya_ has joined #openstack-infra | 07:47 | |
| openstackgerrit | Victor Sergeyev proposed a change to openstack/requirements: Add oslo.db library https://review.openstack.org/91407 | 07:48 |
| *** afazekas_ has joined #openstack-infra | 07:48 | |
| *** salv-orlando has joined #openstack-infra | 07:49 | |
| *** sarob_ has joined #openstack-infra | 07:51 | |
| *** rcarrill` has joined #openstack-infra | 07:53 | |
| *** rcarrillocruz has quit IRC | 07:55 | |
| *** sarob_ has quit IRC | 07:55 | |
| *** jistr has joined #openstack-infra | 07:57 | |
| tchaypo | Again? | 07:59 |
| tchaypo | that's exciting | 07:59 |
| tchaypo | Hrm, it's working for me | 08:00 |
| pas-ha | I've managed to relogin to launchpad (after some annoying errors) and now login to gerrit works | 08:00 |
| pas-ha | so may be we need to put such hint somewhere | 08:01 |
| *** hashar has joined #openstack-infra | 08:02 | |
| *** dmitryme has joined #openstack-infra | 08:04 | |
| *** jpich has joined #openstack-infra | 08:04 | |
| *** markmc has joined #openstack-infra | 08:05 | |
| *** yolanda has quit IRC | 08:06 | |
| *** jlibosva has quit IRC | 08:09 | |
| *** jlibosva has joined #openstack-infra | 08:11 | |
| *** vponomaryov has joined #openstack-infra | 08:13 | |
| *** jgallard has joined #openstack-infra | 08:13 | |
| ogelbukh | hi, is the topic still relevant? hasn't lp restored openid yet? | 08:17 |
| *** sarob_ has joined #openstack-infra | 08:18 | |
| *** fbo_away is now known as fbo | 08:20 | |
| pas-ha | ogelbukh, it is restored to some extend, but it seems one needs to relogin to launchpad to be able to login to gerrit | 08:21 |
| *** hashar has quit IRC | 08:22 | |
| *** pas-ha has left #openstack-infra | 08:22 | |
| tchaypo | /topic | 08:22 |
| *** sarob_ has quit IRC | 08:22 | |
| tchaypo | well done me | 08:22 |
| *** lukego has joined #openstack-infra | 08:22 | |
| tchaypo | lifeless: did you get added to the ACL for the topic bot? | 08:23 |
| *** IvanBerezovskiy has joined #openstack-infra | 08:23 | |
| lukego | “testr list-tests” is failing for my 3rd party CI. possibly related to the setuptools issue over the weekend. anybody know the resolution? http://paste.openstack.org/show/84129/ | 08:24 |
| *** freyes has quit IRC | 08:25 | |
| *** jlibosva1 has joined #openstack-infra | 08:32 | |
| *** jlibosva has quit IRC | 08:35 | |
| *** freyes has joined #openstack-infra | 08:39 | |
| *** andreykurilin_ has quit IRC | 08:41 | |
| *** isviridov|away is now known as isviridov | 08:42 | |
| openstackgerrit | Victor Sergeyev proposed a change to openstack/requirements: Add oslo.db library https://review.openstack.org/91407 | 08:45 |
| *** arnaud__ has quit IRC | 08:47 | |
| BobBall | sdague: Can I please beg for an urgent review of https://review.openstack.org/#/c/100174/ pls? The XenServer CI has been broken by a nova change and that devstack patch needs to land to get it back running. | 08:47 |
| *** pelix has joined #openstack-infra | 08:48 | |
| *** sarob_ has joined #openstack-infra | 08:51 | |
| *** Guest47184 is now known as Hal_ | 08:51 | |
| *** Hal_ has quit IRC | 08:52 | |
| *** Hal_ has joined #openstack-infra | 08:52 | |
| *** sarob_ has quit IRC | 08:56 | |
| *** zhiyan is now known as zhiyan_ | 08:56 | |
| *** habib has joined #openstack-infra | 08:57 | |
| *** habib has joined #openstack-infra | 08:58 | |
| *** jp_at_hp has joined #openstack-infra | 08:58 | |
| *** rdopieralski has quit IRC | 08:59 | |
| SergeyLukjanov | oh, I'm return back after 4 days of non-reading IRC | 09:02 |
| SergeyLukjanov | ttx, I've +1'd your changes to release-tools | 09:04 |
| ttx | SergeyLukjanov: kewl, will approve now | 09:04 |
| ttx | SergeyLukjanov: The secodn change I'll probably wait until Swift RC and make sure they work well before committing them | 09:05 |
| SergeyLukjanov | sdague, lifeless, derekh, ttx, sorry folks, I've been on holidays and not reading irc last four days | 09:08 |
| SergeyLukjanov | ttx, yup, it makes sense | 09:08 |
| ttx | SergeyLukjanov: good to see you're actually human | 09:08 |
| SergeyLukjanov | ttx :) | 09:09 |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 09:09 |
| SergeyLukjanov | ttx, I think you'll see it a lot of times this summer :) | 09:10 |
| *** dizquierdo has joined #openstack-infra | 09:11 | |
| *** ihrachyshka has quit IRC | 09:15 | |
| *** yolanda has joined #openstack-infra | 09:18 | |
| yolanda | clarkb, mordred, how do we want to manage the image replacement/removal on glance/nodepool? at the moment we upload the image to glance with the real name, for example "bare-precise", without any timestamp suffix | 09:19 |
| yolanda | so right now if we update the glance image again, we end up with two images named in the same way, and that causes trouble. Shall we overwrite the glance image in this case, of shall we use suffixes for each upload? | 09:20 |
| *** cnesa has quit IRC | 09:23 | |
| *** adarazs has left #openstack-infra | 09:26 | |
| *** xchu has quit IRC | 09:28 | |
| *** jamielennox is now known as jamielennox|away | 09:29 | |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Wait for ACL creation in manage-projects https://review.openstack.org/94684 | 09:30 |
| *** praneshp has quit IRC | 09:30 | |
| *** davidhadas_ has quit IRC | 09:32 | |
| *** rdopieralski has joined #openstack-infra | 09:32 | |
| *** kashyap has quit IRC | 09:34 | |
| openstackgerrit | Isaku Yamahata proposed a change to openstack-infra/config: Add tacker project on StackForge https://review.openstack.org/97435 | 09:39 |
| *** ihrachyshka has joined #openstack-infra | 09:40 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add jshint job for tuskar-ui https://review.openstack.org/96473 | 09:44 |
| *** weshay has quit IRC | 09:45 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump minimum hacking version to 0.9.2 https://review.openstack.org/99815 | 09:45 |
| *** rcarrillocruz has joined #openstack-infra | 09:46 | |
| *** mrda is now known as mrda-away | 09:47 | |
| *** rcarrill` has quit IRC | 09:48 | |
| *** _nadya_ has quit IRC | 09:51 | |
| *** rgerganov_ is now known as rgerganov | 09:56 | |
| sdague | BobBall: done | 10:03 |
| *** ociuhandu has quit IRC | 10:04 | |
| sdague | any other deprecated things happening for xen that should be cleaned up? | 10:04 |
| *** ociuhandu has joined #openstack-infra | 10:04 | |
| *** Hal_ has quit IRC | 10:05 | |
| *** openstackgerrit has quit IRC | 10:06 | |
| BobBall | They were the only config options I saw - but did you have something particular in mind? | 10:06 |
| BobBall | + thanks | 10:06 |
| *** openstackgerrit has joined #openstack-infra | 10:07 | |
| *** davidhadas has joined #openstack-infra | 10:09 | |
| *** ociuhandu has quit IRC | 10:09 | |
| *** rlandy has joined #openstack-infra | 10:13 | |
| sdague | BobBall: nope, just curious if you are seeing any other deprecation warnings | 10:14 |
| sdague | so we could get ahead of those | 10:14 |
| BobBall | I don't think so, but I'll check again | 10:15 |
| BobBall | I'm very cross with myself for missing this one... (and frustrated with those on the review for ignoring it - but that's a different story :P) | 10:15 |
| *** _nadya_ has joined #openstack-infra | 10:17 | |
| BobBall | a few services are using the deprecated auth fragments and there are a bunch of glance deprecation warnings. I don't think any of them are XS specific | 10:20 |
| openstackgerrit | A change was merged to openstack-infra/config: Unbreak tripleo projects https://review.openstack.org/99778 | 10:22 |
| *** matsuhashi has quit IRC | 10:24 | |
| *** jgallard has quit IRC | 10:24 | |
| *** matsuhashi has joined #openstack-infra | 10:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add a solum-spec repo to stackforge https://review.openstack.org/98237 | 10:25 |
| *** unicell has quit IRC | 10:29 | |
| *** matsuhashi has quit IRC | 10:33 | |
| *** matsuhashi has joined #openstack-infra | 10:34 | |
| BobBall | sdague: you seen the increase in the zuul job queue? | 10:34 |
| BobBall | looks like something bad is about to happen... | 10:34 |
| *** openstackgerrit has quit IRC | 10:35 | |
| sdague | looks like we had a requirements proposal go out | 10:35 |
| BobBall | oh - is that expected then? | 10:36 |
| BobBall | fair enough :) | 10:36 |
| * BobBall just saw a big spike in the graph | 10:36 | |
| BobBall | and typically big spikes are bad :P | 10:36 |
| *** openstackgerrit has joined #openstack-infra | 10:36 | |
| *** matsuhashi has quit IRC | 10:41 | |
| sdague | yeh, proposal bot should maybe wait | 10:41 |
| *** Longgeek has quit IRC | 10:41 | |
| *** matsuhashi has joined #openstack-infra | 10:41 | |
| *** Longgeek has joined #openstack-infra | 10:42 | |
| sdague | plus proposal bot is not really supposed to propose hacking | 10:43 |
| ianw | sdague: when you're in a reviewing mood, https://review.openstack.org/#/c/100131/ is a follow on from one you merged to stop rm'ing grenade logs. i believe it's causing two copies of the logs now. i think it's kind of logical to stick those logs in /grenade/ | 10:44 |
| sdague | ianw: why? we don't do that with devstack logs on devstack runs | 10:44 |
| *** Longgeek has quit IRC | 10:46 | |
| sdague | also that seems to lose all the logs again | 10:47 |
| ianw | sdague: it's already making /opt/stack/logs/grenade and putting it's localrc in there, so seems logical to have the logs next to it. | 10:47 |
| ianw | sdague: http://logs.openstack.org/52/99852/3/check/check-grenade-dsvm/3b67817/logs/grenade/ looks right? | 10:47 |
| *** rdopieralski has quit IRC | 10:49 | |
| *** rdopieralski has joined #openstack-infra | 10:49 | |
| sdague | so I really don't think the grenade log should be further tucked down in a grenade run | 10:49 |
| ianw | i've gotten quite lost in all that copying stuff debugging the fedora jobs; personally i think https://review.openstack.org/#/c/99852/ makes things better, others may disagree... | 10:49 |
| *** rdopieralski has quit IRC | 10:49 | |
| *** rdopieralski has joined #openstack-infra | 10:49 | |
| *** rdopieralski has quit IRC | 10:50 | |
| *** rdopieralski has joined #openstack-infra | 10:51 | |
| *** sarob_ has joined #openstack-infra | 10:51 | |
| *** rdopieralski is now known as rdopiera | 10:52 | |
| *** pblaho has joined #openstack-infra | 10:52 | |
| ianw | sdague: well, you're ultimately the guy with the +2 :) i feel like keep things related together, but whatever | 10:54 |
| ianw | sdague: also, my internal ci is having some issues with the "convert from awk" patch | 10:54 |
| ianw | seems like something is hanging around and the ssh session i run devstack under isn't exiting cleanly | 10:55 |
| *** sarob_ has quit IRC | 10:55 | |
| tkelsey | Hello -infra people, could I please get some more eyeballs on this https://review.openstack.org/#/c/97250/ many thanks. | 10:57 |
| ianw | sdage: stack 1759 1751 98 00:49 ? 09:59:32 python ./tools/outfilter.py -o /opt/stack/logs/stack.sh.log.2014-06-16-004946.2014-06-16-004946.summary | 10:57 |
| *** Longgeek has joined #openstack-infra | 10:57 | |
| sdague | ianw: ok, so perhaps there is a signal handler we need to trigger the exit ? | 10:58 |
| ianw | sdague: yeah, it's just sitting there in read(0, "", 4096) | 10:58 |
| *** davidhadas___ has joined #openstack-infra | 11:03 | |
| ianw | sorry, chopped, read(0...) = 0 | 11:03 |
| ianw | sdague: maybe i'm missing something, but where's the break from the "while True" readline() loop? | 11:03 |
| sdague | yeh, so I thought that would signal down at the end, apparently not | 11:04 |
| sdague | just uploaded a new version that dumps out | 11:04 |
| sdague | lets see | 11:04 |
| ianw | sdague: ok, my tests eventually got going when the testing vm ran out of disk-space ... that raised and exception and got things moving :) but it took about 10 hours | 11:05 |
| *** davidhadas has quit IRC | 11:06 | |
| *** elitenudel2500 has joined #openstack-infra | 11:06 | |
| elitenudel2500 | Hey! | 11:06 |
| elitenudel2500 | Code Review - Error Server Error Cannot store contact information | 11:07 |
| elitenudel2500 | Anyone has any experience with solving this? | 11:07 |
| elitenudel2500 | (signing up for ICLA) | 11:07 |
| *** ominakov has joined #openstack-infra | 11:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: index grenade logs in elastic search https://review.openstack.org/100219 | 11:09 |
| sdague | ianw: can you try this latest patch locally? | 11:09 |
| *** yamahata has quit IRC | 11:10 | |
| sdague | ianw: interestingly enough, based on the times we see double time stamps, I think the new approach generates a lot less overhead because we aren't fork/execing date 100k times during a run. | 11:10 |
| *** habib has quit IRC | 11:12 | |
| elitenudel2500 | oh, i just saw the MOTD | 11:13 |
| elitenudel2500 | sorry :P | 11:13 |
| ianw | sdauge: my bot is running the change now ... tests won't pass due to other issues but it should at least fail out in ~ 40 mins | 11:14 |
| *** nosnos has quit IRC | 11:15 | |
| sdague | ianw: cool | 11:17 |
| sdague | ianw: it's back in check queue with the new changes, locally that seems to do the right thing | 11:18 |
| sdague | I guess in the gate the fact that we just kill everything means we didnt' see it | 11:18 |
| BobBall | oh... was just going to ask... did you kill the gate just now? :) I can't find any evidence my job is in the gate queue https://review.openstack.org/#/c/100174/ | 11:19 |
| BobBall | oh jeeze | 11:20 |
| BobBall | so I did a search and it wasn't there | 11:20 |
| BobBall | but now it is | 11:20 |
| BobBall | ignore me, please... | 11:20 |
| ianw | sdague: ok, so yeah rhel6.5 redhatci job with revision 5 failed, but it did exit out, which is good :) | 11:21 |
| sdague | ok, good :) | 11:22 |
| sdague | well then +1 on the devstack review would be good with that info | 11:23 |
| *** tnurlygayanov has joined #openstack-infra | 11:23 | |
| ianw | sdague: will do, i'll let redhatci post on that one. i think if it was rebased to TOT rhel6.5/centos6.5/f20 would work with redhatci ... just rhel7 has some issues ATM | 11:25 |
| sdague | ianw: so next up, I was going to take some of the exit debugging in grenade and put them into a 'worlddump' utility in devstack | 11:25 |
| sdague | the process dump, disk free list, if there are other things about the state of the box you think would be handy to dump for debug purposes, let me know | 11:26 |
| ianw | sdague: so almost like redhat's sos or vmware's vm-support bundle? | 11:27 |
| ianw | yeah, that's all good stuff. various ip link / route / namespace output is always helpful | 11:28 |
| *** matsuhashi has quit IRC | 11:29 | |
| *** matsuhashi has joined #openstack-infra | 11:29 | |
| ianw | also something i remember vm-support doing was taking two snapshots 15 seconds apart or so, can help you know if things are coming or going | 11:29 |
| ianw | (not of everything, but process listing, etc) | 11:29 |
| sdague | ianw: yeh, basically. We've been inlining these things adhoc in the console, but that just confuses everything I think | 11:31 |
| sdague | so I think we should dump them out to timestamped files, which you can go look at | 11:31 |
| *** CaptTofu_ has joined #openstack-infra | 11:31 | |
| *** matsuhashi has quit IRC | 11:33 | |
| *** elitenudel2500 has quit IRC | 11:34 | |
| ianw | sdague: yeah totally. i've been toying with the idea of a bundle download of logs, it would often be helpful | 11:41 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/requirements: Add xstatic-packaged JavaScript libraries for Horizon https://review.openstack.org/97793 | 11:41 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/requirements: Add xstatic and xstatic-jquery for Horizon https://review.openstack.org/94337 | 11:41 |
| ianw | (please let me know if i'm just missing the existing way to do it :) | 11:41 |
| sdague | so the bundle for download would probably be best done with a wsgi script | 11:43 |
| sdague | markmc had requested that at summit | 11:44 |
| *** ociuhandu has joined #openstack-infra | 11:45 | |
| ianw | sdague: that was my thought too, no point keeping it. i can look into it but i'll have to bootstrap myself on the log servers ... i'm sure it's all in the config repo somewhere :) | 11:45 |
| *** mbacchi has joined #openstack-infra | 11:48 | |
| *** YorikSar has quit IRC | 11:49 | |
| *** YorikSar has joined #openstack-infra | 11:49 | |
| *** rcarrill` has joined #openstack-infra | 11:50 | |
| *** mugsie has joined #openstack-infra | 11:50 | |
| *** sarob_ has joined #openstack-infra | 11:51 | |
| *** rcarrillocruz has quit IRC | 11:52 | |
| *** habib has joined #openstack-infra | 11:55 | |
| *** sarob_ has quit IRC | 11:56 | |
| *** weshay has joined #openstack-infra | 11:58 | |
| openstackgerrit | Maxime Vidori proposed a change to openstack-infra/storyboard-webclient: Documentation improvement https://review.openstack.org/99775 | 11:59 |
| *** weshay has quit IRC | 12:01 | |
| *** weshay has joined #openstack-infra | 12:02 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: don't use screen for grenade https://review.openstack.org/100229 | 12:05 |
| sdague | ianw: yeh, os-loganalyze is a decent example | 12:05 |
| *** lcostantino has joined #openstack-infra | 12:09 | |
| *** ArxCruz has joined #openstack-infra | 12:09 | |
| *** Hal_ has joined #openstack-infra | 12:10 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: revert hacking to 0.8 series https://review.openstack.org/100231 | 12:10 |
| sdague | ttx: I'd like your take on that ^^^ | 12:11 |
| *** pblaho has quit IRC | 12:12 | |
| *** ildikov_ has joined #openstack-infra | 12:12 | |
| *** pblaho has joined #openstack-infra | 12:13 | |
| *** ildikov has quit IRC | 12:13 | |
| *** ildikov_ has quit IRC | 12:16 | |
| *** morganfainberg has quit IRC | 12:18 | |
| *** CaptTofu_ has quit IRC | 12:19 | |
| sdague | SergeyLukjanov: do you know how to do the force deleting delete thing? I wonder if we could get some more capacity back with that. Because we seem to have grown stale deleting nodes over the weekend | 12:19 |
| *** morganfainberg has joined #openstack-infra | 12:19 | |
| *** _nadya_ has quit IRC | 12:20 | |
| SergeyLukjanov | sdague, hey | 12:20 |
| *** cody-somerville has joined #openstack-infra | 12:20 | |
| *** rfolco has joined #openstack-infra | 12:20 | |
| SergeyLukjanov | sdague, "force deleting delete thing" of what? | 12:20 |
| *** _nadya_ has joined #openstack-infra | 12:20 | |
| *** dkranz_afk has quit IRC | 12:22 | |
| *** yaguang has quit IRC | 12:22 | |
| *** jgallard has joined #openstack-infra | 12:23 | |
| *** yamahata has joined #openstack-infra | 12:23 | |
| *** davidhadas___ has quit IRC | 12:24 | |
| sdague | nodepool nodes | 12:25 |
| *** andreykurilin_ has joined #openstack-infra | 12:26 | |
| *** dprince has joined #openstack-infra | 12:27 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Update and freeze requirements https://review.openstack.org/98491 | 12:28 |
| *** adalbas has joined #openstack-infra | 12:29 | |
| *** ArxCruz has quit IRC | 12:29 | |
| SergeyLukjanov | sdague, oh, does you mean that slaves are removing very slowly? | 12:29 |
| sdague | yeh | 12:29 |
| SergeyLukjanov | sdague, I have no credentials to our clouds | 12:30 |
| *** ArxCruz has joined #openstack-infra | 12:32 | |
| *** hdd_ has joined #openstack-infra | 12:33 | |
| *** miqui has joined #openstack-infra | 12:33 | |
| *** pblaho has quit IRC | 12:37 | |
| *** dkranz_afk has joined #openstack-infra | 12:38 | |
| *** pdmars has joined #openstack-infra | 12:38 | |
| *** CaptTofu_ has joined #openstack-infra | 12:39 | |
| *** ildikov has joined #openstack-infra | 12:39 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use root to write syslog from journalctl https://review.openstack.org/98283 | 12:40 |
| *** pdmars has quit IRC | 12:40 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Save sql logs https://review.openstack.org/94314 | 12:41 |
| *** zhiyan_ is now known as zhiyan | 12:44 | |
| *** pblaho has joined #openstack-infra | 12:44 | |
| sdague | so I think the new grenade fails on services not starting is just because hp cloud 1.1 is so much slower - eyJmaWVsZHMiOltdLCJzZWFyY2giOiJtZXNzYWdlOlwiTWVzc2FnaW5nVGltZW91dDogVGltZWQgb3V0IHdhaXRpbmcgZm9yIGEgcmVwbHkgdG8gbWVzc2FnZSBJRFwiIEFORCBtZXNzYWdlOlwibmV0d29ya1wiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiXG4iLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsIm9mZnNldCI6MCwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwibW9kZSI6InNjb3JlIiwiYW5hbHl6 | 12:45 |
| sdague | ZV9maWVsZCI6ImJ1aWxkX25vZGUifQ | 12:45 |
| *** pdmars has joined #openstack-infra | 12:47 | |
| openstackgerrit | Dan Prince proposed a change to openstack-infra/nodepool: Shuffle requests for fairer label allocation https://review.openstack.org/88223 | 12:50 |
| *** sweston has quit IRC | 12:52 | |
| *** izinovik has joined #openstack-infra | 12:53 | |
| *** julim has joined #openstack-infra | 12:55 | |
| *** jgallard has quit IRC | 12:56 | |
| *** jgallard has joined #openstack-infra | 12:57 | |
| izinovik | Hello. Does my problem with creating review on gerrit (review.openstack.org) is connected with disabled launchpad openid? Right now when I'm doing `git review' it bails out with this message: "fatal: ICLA contributor agreement requires current contact information. | 12:59 |
| izinovik | Please review your contact information: | 12:59 |
| izinovik | https://review.openstack.org/#/settings/contact | 12:59 |
| izinovik | fatal: The remote end hung up unexpectedly" | 12:59 |
| *** jlibosva1 is now known as jlibosva | 12:59 | |
| *** smarcet has joined #openstack-infra | 12:59 | |
| *** sandywalsh has quit IRC | 13:01 | |
| *** jgallard has quit IRC | 13:04 | |
| *** sandywalsh has joined #openstack-infra | 13:04 | |
| *** CaptTofu_ has quit IRC | 13:04 | |
| *** jgallard has joined #openstack-infra | 13:04 | |
| *** CaptTofu_ has joined #openstack-infra | 13:05 | |
| *** basha has joined #openstack-infra | 13:05 | |
| ttx | sdague: commented | 13:05 |
| *** basha has quit IRC | 13:05 | |
| sdague | ttx: ok :) | 13:06 |
| sdague | I think it's terribleness personally | 13:06 |
| sdague | ttx: did you look at my comment with some of the impacts of a hacking upgrade? | 13:06 |
| ttx | sdague: ah now it came after | 13:07 |
| ttx | no* | 13:07 |
| ttx | rereading | 13:07 |
| *** e0ne has quit IRC | 13:08 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 13:09 |
| *** CaptTofu_ has quit IRC | 13:09 | |
| *** ominakov has quit IRC | 13:09 | |
| srenatus | izinovik: I have seen that last week when the new person had not yet joined the openstack foundation with the same email address | 13:10 |
| *** ominakov has joined #openstack-infra | 13:10 | |
| srenatus | izinovik: https://wiki.openstack.org/wiki/GerritWorkflow#Account_Setup | 13:10 |
| *** mfer has joined #openstack-infra | 13:10 | |
| *** chuckC has joined #openstack-infra | 13:10 | |
| ttx | sdague: I tend to agree that this should be seen as gratuitous changes and therefore preferably pushed at low tide (probably between release and first milestone)... but I'm not sure it's a cycle issue. It's more of a gate load issue | 13:11 |
| sdague | ttx: they are coupled | 13:11 |
| sdague | because it steals nodes from check | 13:11 |
| *** dims has joined #openstack-infra | 13:12 | |
| sdague | the reason there is a 1 hrs wait for check nodes now is because of hacking proposal to requirements jobs | 13:12 |
| *** cody-somerville has quit IRC | 13:12 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 13:12 |
| sdague | but if no one else sees a problem, so be it | 13:13 |
| ttx | sdague: the slope is a bit dangerous here: we would classify changes by their level of gratuitousness and have a window where truly gratuitous change would land in te cycle | 13:13 |
| ttx | I'm ok to single out hacking there | 13:13 |
| ttx | and have specific rules around it | 13:13 |
| ttx | but I gear there will soon be no good moment for those | 13:13 |
| ttx | fear* | 13:13 |
| sdague | the right time is release week | 13:13 |
| sdague | there is also something else kind of wrong with it being in the requirements proposal bot right now | 13:14 |
| ttx | sdague: My point is... what you're actually after here is agreement on timing rules for hacking changes. | 13:14 |
| sdague | yep | 13:14 |
| *** penguinRaider has joined #openstack-infra | 13:14 | |
| sdague | and there is a mailing list thread on that | 13:14 |
| ttx | it's not a one-shot, it will come back | 13:15 |
| sdague | which I started at the same time of proposing the req change | 13:15 |
| sdague | because this is the same issue as the pep8 fatigue that we had, where we agreed one bump, first week of cycle, then we're done | 13:15 |
| ttx | sdague: oh we did ? Then yes, i would put that in the same bag | 13:16 |
| ttx | I think we are saying the same thing, we just mean slightly different things by "cycle" | 13:16 |
| sdague | ok | 13:16 |
| ttx | I don't think landing those now jeopardizes the release -- but I agree that we could have rules for optimizing landing time for gratuitous changes. | 13:17 |
| ttx | Will comment as such | 13:17 |
| sdague | cool | 13:17 |
| *** krtaylor has joined #openstack-infra | 13:18 | |
| sdague | though given that we've got fixed resources for machines and people (reviewers) I'm not sure how I understand having another openstack projects worth of changes to get through doesn't impact release :) | 13:18 |
| ttx | sdague: it means less things get merged. But that's not necessarily a bad thing. It impacts our velocity of things you may consider more useful. | 13:19 |
| *** pblaho is now known as pblaho|afk | 13:19 | |
| ttx | before j2 it doesn't really affect release quality (as would landing them during RC time, which would affect bugfixing, or during j-3, which would affect feature completeness) | 13:20 |
| ttx | It would definitely mean less features merged. | 13:20 |
| sdague | it impacts bug fixing now | 13:21 |
| ttx | as much as adding features. | 13:21 |
| sdague | sure | 13:21 |
| *** e0ne has joined #openstack-infra | 13:21 | |
| sdague | but for instance - http://status.openstack.org/elastic-recheck/gate.html - the top gate reset bug is something that I think I have a theory on | 13:22 |
| sdague | but I'll be waiting about 3 hrs to get the first results back | 13:22 |
| *** _nadya_ has quit IRC | 13:23 | |
| ttx | right, it impacts our velocity at doing more useful things. But I would not say it jeopardizes that much the "release" (understood as releasing something coherent) | 13:23 |
| *** james_li has joined #openstack-infra | 13:24 | |
| ttx | it impacts how much features we would ship in the release for sure | 13:24 |
| ttx | but tha's not a quality metric for me | 13:24 |
| *** trinaths has joined #openstack-infra | 13:25 | |
| sdague | ok | 13:25 |
| ttx | anyway, i think we are saying the same thing. | 13:25 |
| sdague | probably | 13:26 |
| *** crc32 has joined #openstack-infra | 13:26 | |
| *** pblaho|afk is now known as pblaho | 13:26 | |
| ttx | it's just that I don't think "screwing up the release" is the best argument. "Gratuitously impacting our velocity so it should be put in same bag as pep8" sounds better to me | 13:27 |
| ttx | now again, my answer would be different if we were past j2. | 13:27 |
| sdague | that's going to happen pretty quick | 13:27 |
| sdague | and once we're past j2, based on current capacity issues, patch round trips are going to take a couple of days | 13:27 |
| ttx | sdague: If that's the impact, I'm starting to question the value of upgrading hacking AT ALL | 13:28 |
| ttx | if it raelly screws up a whole development milestone timeframe, is the upgrade worth it ? | 13:28 |
| sdague | yeh, my back of the envelope math is there is no way this stuff gets through in j2 | 13:29 |
| sdague | as most projects haven't started yet | 13:29 |
| sdague | ttx: yeh, honestly, I'm in the same sort of boat. The impact of this one is it drags in a new flake8 | 13:30 |
| sdague | which means a ton of new rules from upstream | 13:30 |
| ttx | haven't looked into what it brings, but we may have passed a size where style checking rules need to be frozen because closing the gap on upgrade is just too costly | 13:30 |
| sdague | http://logs.openstack.org/68/96268/12/check/gate-python-novaclient-pep8/95a7699/console.html | 13:31 |
| sdague | that's a pretty good example | 13:31 |
| *** otherwiseguy has joined #openstack-infra | 13:32 | |
| ttx | I'll try to find the thread and comment there | 13:32 |
| sdague | actually, that's mostly self inflicted now that I see it | 13:33 |
| *** medieval1 has joined #openstack-infra | 13:33 | |
| *** jgrimm_ has joined #openstack-infra | 13:34 | |
| *** terryw has quit IRC | 13:35 | |
| *** CaptTofu_ has joined #openstack-infra | 13:39 | |
| *** jp_at_hp has quit IRC | 13:44 | |
| *** eharney has joined #openstack-infra | 13:45 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Update method improved in db api https://review.openstack.org/98760 | 13:45 |
| *** jp_at_hp has joined #openstack-infra | 13:47 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add tempest jobs with nova-v3 enabled https://review.openstack.org/99835 | 13:51 |
| *** sarob_ has joined #openstack-infra | 13:51 | |
| *** pdmars has quit IRC | 13:52 | |
| *** pdmars has joined #openstack-infra | 13:53 | |
| *** radez_g0n3 is now known as radez | 13:53 | |
| *** homeless has joined #openstack-infra | 13:53 | |
| *** sarob_ has quit IRC | 13:55 | |
| *** e0ne_ has joined #openstack-infra | 13:56 | |
| *** trinaths has quit IRC | 13:56 | |
| *** hdd_ has quit IRC | 13:57 | |
| *** maxbit has joined #openstack-infra | 13:58 | |
| *** e0ne has quit IRC | 13:59 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms https://review.openstack.org/95873 | 13:59 |
| *** mriedem has joined #openstack-infra | 13:59 | |
| *** otherwiseguy has quit IRC | 13:59 | |
| *** jistr has quit IRC | 13:59 | |
| *** ildikov has quit IRC | 14:01 | |
| *** jistr has joined #openstack-infra | 14:01 | |
| openstackgerrit | Victor Sergeyev proposed a change to openstack/requirements: Add oslo.db library https://review.openstack.org/91407 | 14:02 |
| *** alugovoi has joined #openstack-infra | 14:03 | |
| *** markmcclain has joined #openstack-infra | 14:04 | |
| *** koolhead17 has quit IRC | 14:05 | |
| *** cody-somerville has joined #openstack-infra | 14:05 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 14:05 |
| *** markmcclain1 has joined #openstack-infra | 14:06 | |
| *** dhellman_ has joined #openstack-infra | 14:06 | |
| *** dhellman_ has quit IRC | 14:06 | |
| *** zz_gondoi is now known as gondoi | 14:07 | |
| *** markmcclain has quit IRC | 14:08 | |
| openstackgerrit | Alexandre Viau proposed a change to openstack-infra/config: Added the Surveil project to gerritbot, zuul and stackforge config https://review.openstack.org/99746 | 14:10 |
| *** dhellmann has quit IRC | 14:12 | |
| *** dhellmann has joined #openstack-infra | 14:12 | |
| *** otherwiseguy has joined #openstack-infra | 14:14 | |
| *** afazekas_ has quit IRC | 14:16 | |
| *** crc32 has quit IRC | 14:18 | |
| *** otherwiseguy has quit IRC | 14:20 | |
| *** terryw has joined #openstack-infra | 14:20 | |
| *** gargola has joined #openstack-infra | 14:23 | |
| *** gargola has quit IRC | 14:25 | |
| *** gargola has joined #openstack-infra | 14:27 | |
| *** terryw has quit IRC | 14:27 | |
| sdague | fungi / clarkb: it would be good to quick approve this today - https://review.openstack.org/#/c/100219/ so I can start pruning grenade output from the console | 14:27 |
| fungi | sdague: i can try, though on a wireless modem watching the movers pack things so not really paying attention | 14:28 |
| sdague | sure | 14:28 |
| sdague | it's just an indexing rule for ES | 14:28 |
| sdague | so should be low risk | 14:29 |
| fungi | sdague: looks simple enough, agreed | 14:29 |
| *** rwsu has joined #openstack-infra | 14:29 | |
| fungi | sdague: size of those log files should be fairly small, right? | 14:30 |
| fungi | we're not going to overwhelm logstash/elasticsearch by adding them presumably? | 14:30 |
| fungi | approved. i guess it can be reverted if it does for some reason | 14:32 |
| *** _nadya_ has joined #openstack-infra | 14:32 | |
| fungi | assuming the only things it contains are the same lines echoed into the console currently, and you'll be culling those shortly thereafter, net impact should be near zero | 14:33 |
| *** cp16net_ has joined #openstack-infra | 14:34 | |
| *** chandan_kumar has joined #openstack-infra | 14:36 | |
| *** W00die has joined #openstack-infra | 14:36 | |
| *** doug-fish has joined #openstack-infra | 14:37 | |
| *** wenlock_ has joined #openstack-infra | 14:38 | |
| sdague | fungi: yeh, my hope is to have all the pieces in over the next 2 days | 14:38 |
| *** dkliban_afk is now known as dkliban | 14:38 | |
| anteaya | fungi: does anyone else have the ability to run your magic delete the nodes scripts, that you ran on Friday? | 14:38 |
| openstackgerrit | A change was merged to openstack-infra/config: index grenade logs in elastic search https://review.openstack.org/100219 | 14:38 |
| sdague | the problem is that it's grenade, devstack, and d-g changes that all need to in a specific order | 14:38 |
| *** amotoki_ has joined #openstack-infra | 14:39 | |
| sdague | most of which are written, a few are not | 14:39 |
| sdague | what's our rax quota anyway? | 14:39 |
| fungi | anteaya: there's nothing magic. i was just filtering nodepool list output for nodes in a delete state longer than an hour and passing their node ids to nodepool delete --now | 14:39 |
| sdague | because I was suprised how few rax nodes where in my timing lists | 14:40 |
| fungi | anteaya: anyone with root access on nodepool.o.o (i.e. infra root admins) can do that | 14:40 |
| *** kashyap has joined #openstack-infra | 14:40 | |
| fungi | sdague: pretty low. you can see the max-servers counts in nodepool.yaml.erb | 14:41 |
| fungi | well, i say "pretty low" but really only low in relation to hpcloud | 14:41 |
| sdague | can we get that bumped? | 14:41 |
| sdague | especially as the rax nodes are faster than the hp cloud ones now | 14:41 |
| anteaya | fungi: awesome, thanks, since it looks like we may need to do that before too long | 14:41 |
| fungi | they lowered it recently due to capacity issues, but mordred might be able to sweet talk some more out of pvo | 14:41 |
| *** esker has joined #openstack-infra | 14:41 | |
| fungi | possibly in a different region under less load | 14:42 |
| *** ildikov has joined #openstack-infra | 14:42 | |
| sdague | ok | 14:44 |
| *** W00die has quit IRC | 14:45 | |
| sdague | ok, I'm going to go get a bike ride before it gets too too hot today | 14:45 |
| anteaya | sdague: happy cycling | 14:46 |
| jeblair | good morning | 14:46 |
| anteaya | gooooooood morning, jeblair | 14:46 |
| anteaya | welcome back | 14:46 |
| anteaya | and welcome back | 14:46 |
| sdague | this is probably a key fix for grenade failures, hopefully check results are in by the time I get back - https://review.openstack.org/#/c/100253/ | 14:46 |
| mordred | aroo? | 14:46 |
| mordred | morning jeblair ! | 14:47 |
| * jeblair starts gertty (with 212 sync actions) | 14:48 | |
| mordred | jeblair: while you were gone, everything broke. welcome back - you can fix it all now! | 14:48 |
| jeblair | mordred: i don't think i can fix launchpad openid | 14:48 |
| mordred | jeblair: blasphemy | 14:48 |
| *** basha has joined #openstack-infra | 14:49 | |
| *** alugovoi has quit IRC | 14:50 | |
| *** prad_ has joined #openstack-infra | 14:50 | |
| *** sarob_ has joined #openstack-infra | 14:51 | |
| *** CaptTof__ has joined #openstack-infra | 14:54 | |
| *** CaptTofu_ has quit IRC | 14:55 | |
| *** rkukura has quit IRC | 14:56 | |
| *** sarob_ has quit IRC | 14:56 | |
| *** W00die has joined #openstack-infra | 14:57 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard-webclient: Including UX Feedback on menu and nav. https://review.openstack.org/99209 | 14:57 |
| *** jistr has quit IRC | 14:58 | |
| *** jistr has joined #openstack-infra | 14:59 | |
| *** sarob_ has joined #openstack-infra | 14:59 | |
| *** hogepodge has joined #openstack-infra | 15:02 | |
| *** _nadya_ has quit IRC | 15:03 | |
| *** ihrachyshka has quit IRC | 15:03 | |
| *** dkranz_afk has quit IRC | 15:05 | |
| *** dizquierdo has quit IRC | 15:05 | |
| *** changbl has joined #openstack-infra | 15:06 | |
| *** yfried has quit IRC | 15:06 | |
| SergeyLukjanov | jeblair, hey | 15:07 |
| *** pcrews has joined #openstack-infra | 15:07 | |
| jeblair | SergeyLukjanov: o/ | 15:07 |
| jeblair | storyboard meeting in #openstack-meeting-3 | 15:08 |
| *** dkranz_afk has joined #openstack-infra | 15:08 | |
| *** atiwari has joined #openstack-infra | 15:08 | |
| *** rkukura has joined #openstack-infra | 15:08 | |
| *** Guest8031 is now known as mgagne | 15:08 | |
| *** mgagne has joined #openstack-infra | 15:08 | |
| *** enikanorov_ has joined #openstack-infra | 15:09 | |
| *** julim has quit IRC | 15:12 | |
| jeblair | heh, gerrty is now "down to" 333 sync tasks (from 212) | 15:13 |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/nodepool: Build images using diskimage-builder https://review.openstack.org/46482 | 15:14 |
| *** sarob_ has quit IRC | 15:16 | |
| *** moted has joined #openstack-infra | 15:16 | |
| *** sarob_ has joined #openstack-infra | 15:16 | |
| *** alugovoi has joined #openstack-infra | 15:16 | |
| *** jaypipes has joined #openstack-infra | 15:20 | |
| *** sarob_ has quit IRC | 15:21 | |
| *** zul has quit IRC | 15:22 | |
| *** zul has joined #openstack-infra | 15:23 | |
| *** dangers_away is now known as dangers | 15:27 | |
| *** markmc has quit IRC | 15:28 | |
| *** basha has quit IRC | 15:28 | |
| *** enikanorov__ has joined #openstack-infra | 15:29 | |
| *** dkehn_ is now known as dkehnx | 15:29 | |
| *** alexpilotti has joined #openstack-infra | 15:30 | |
| *** enikanorov_ has quit IRC | 15:32 | |
| *** gondoi is now known as zz_gondoi | 15:34 | |
| *** IvanBerezovskiy has left #openstack-infra | 15:38 | |
| *** afazekas_ has joined #openstack-infra | 15:38 | |
| *** rdopiera has quit IRC | 15:40 | |
| *** denis_makogon_ has joined #openstack-infra | 15:41 | |
| *** denis_makogon has quit IRC | 15:42 | |
| *** denis_makogon_ is now known as denis_makogon | 15:42 | |
| *** dmakogon_ has joined #openstack-infra | 15:42 | |
| clarkb | o/ fyi I will probably not be super here today between cold and world cup | 15:45 |
| *** unicell has joined #openstack-infra | 15:47 | |
| anteaya | :( | 15:47 |
| anteaya | take care of your cold | 15:47 |
| anteaya | clarkb: can you filter nodepool list output for nodes that have been deleting for longer than an hour and pass their node ids to nodepool delete --now | 15:49 |
| anteaya | we see to be in constant delete state like last week | 15:49 |
| anteaya | s/see/seem | 15:49 |
| anteaya | boris-42: is this you? https://review.openstack.org/#/c/100243/3 | 15:50 |
| *** kmartin has joined #openstack-infra | 15:51 | |
| *** basha has joined #openstack-infra | 15:52 | |
| *** zhiyan is now known as zhiyan_ | 15:52 | |
| *** bearhands is now known as comstud | 15:53 | |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/requirements: Add Pint for Horizon https://review.openstack.org/97224 | 15:54 |
| *** basha has quit IRC | 15:55 | |
| *** jcoufal has quit IRC | 15:56 | |
| *** sweston has joined #openstack-infra | 15:57 | |
| *** jlibosva has quit IRC | 15:57 | |
| *** ominakov has quit IRC | 15:59 | |
| *** dkliban is now known as dkliban_brb | 15:59 | |
| *** Longgeek has quit IRC | 16:00 | |
| krotscheck | So should we permit projects to be created that don’t have repositories? Or is it ok to create repositories for projects with the knowedge that they’ll never be used? (Context: UX wants access to storyboard, storyboard uses review.projects.yaml to preload, which also creates git repos). | 16:02 |
| krotscheck | Third option: Create a storyboard.projects.yaml file | 16:02 |
| *** otherwiseguy has joined #openstack-infra | 16:03 | |
| *** e0ne_ has quit IRC | 16:03 | |
| rainya | jeblair, are you in this morning sir? | 16:05 |
| clarkb | krotscheck: I think it is ok. lp for exanple has trackers for things without repos | 16:05 |
| *** freyes has quit IRC | 16:06 | |
| *** afazekas_ has quit IRC | 16:06 | |
| krotscheck | clarkb: I don’t really have an opinion, it’s just that the discussion petered out before a consensus was reached. | 16:06 |
| *** marun has joined #openstack-infra | 16:06 | |
| *** pblaho is now known as pblaho|afk | 16:07 | |
| *** otherwiseguy has quit IRC | 16:07 | |
| *** hemna_ is now known as hemna | 16:08 | |
| *** arnaud__ has joined #openstack-infra | 16:08 | |
| zaro | morning | 16:08 |
| anteaya | morning zaro | 16:08 |
| mordred | clarkb: yeah - we don't have jeepyb knowledge of things not having repos right now | 16:09 |
| mordred | gooooooooooooooooooooooooooooooooooooooooool | 16:11 |
| zaro | mordred: ansible | 16:11 |
| morganfainberg | mordred, hehe | 16:11 |
| zaro | morganfainberg: is your suggestion to replace jenkins with ansible? | 16:11 |
| mordred | zaro: no. not at all | 16:12 |
| zaro | sorry meant for mordred ^ | 16:12 |
| morganfainberg | zaro, i'm used to that by now. :P | 16:12 |
| mordred | zaro: it's simply that the yaml format for describing tasks and actions is SO CLOSE | 16:12 |
| mordred | that if we aligned it somehow, it might leave us with yaml chunks that could be used by either | 16:12 |
| *** pblaho|afk is now known as pblaho | 16:12 | |
| *** jergerber has joined #openstack-infra | 16:12 | |
| mordred | zaro: now, there _could_ be opportunities for making use of the in ways that jhesketh suggested in the email | 16:15 |
| *** blamar has joined #openstack-infra | 16:16 | |
| mordred | zaro: but I think first step is just figuring out if effort towards alignment is meaningful in any way | 16:16 |
| zaro | mordred: i must have missed his email. will take a look. thanks. | 16:18 |
| *** marcoemorais has joined #openstack-infra | 16:20 | |
| *** dkranz_afk is now known as dkranz | 16:23 | |
| *** gyee has joined #openstack-infra | 16:26 | |
| anteaya | krtaylor: going to be at today's third-party meeting? | 16:28 |
| *** jgallard has quit IRC | 16:31 | |
| *** shayneburgess has joined #openstack-infra | 16:31 | |
| *** alugovoi has quit IRC | 16:31 | |
| *** blamar has quit IRC | 16:31 | |
| mordred | gooooooooooooooooooooooooooooool | 16:32 |
| anteaya | can we get some nodes starting to be deleted please? | 16:33 |
| *** marcoemorais has quit IRC | 16:33 | |
| anteaya | the nodes that have been deleting for over an hour? | 16:33 |
| *** marcoemorais has joined #openstack-infra | 16:33 | |
| *** markmcclain1 has quit IRC | 16:33 | |
| anteaya | 186 in check | 16:33 |
| *** marcoemorais has quit IRC | 16:33 | |
| *** basha has joined #openstack-infra | 16:34 | |
| *** marcoemorais has joined #openstack-infra | 16:34 | |
| *** basha has quit IRC | 16:35 | |
| *** praneshp has joined #openstack-infra | 16:36 | |
| *** trinaths has joined #openstack-infra | 16:36 | |
| *** sweston has quit IRC | 16:37 | |
| *** jpich has quit IRC | 16:37 | |
| *** tcammann has quit IRC | 16:38 | |
| *** zehicle_at_dell has quit IRC | 16:39 | |
| jogo | anteaya: good news is the nova fix for this is in the gate | 16:39 |
| anteaya | jogo: woooo | 16:39 |
| jogo | anteaya: so all we need is rax to land the patch | 16:39 |
| anteaya | yay | 16:40 |
| jogo | anteaya: wich may be a while :/ | 16:40 |
| anteaya | before the end of the week do you think? | 16:40 |
| anteaya | I don't know how long it takes for rax to consume master | 16:40 |
| jogo | anteaya: doubtful | 16:40 |
| jogo | comstud said up to a month | 16:40 |
| anteaya | well progress anyway | 16:40 |
| anteaya | ouch | 16:40 |
| *** pblaho is now known as pblaho|afk | 16:40 | |
| *** zzelle_ has joined #openstack-infra | 16:40 | |
| jogo | anteaya: do you know of any plans to get more quota in general? | 16:40 |
| Alex_Gaynor | The only nova change I see in the gate is a deprecation warning, is that causing big issues for some reasons? | 16:40 |
| anteaya | something happened at 0:00utc saturday | 16:40 |
| jogo | anteaya: yeah, they may try to speed it up | 16:41 |
| anteaya | all the nodes stuck in delete state all of a sudden ceased at 0:00 utc saturday | 16:41 |
| anteaya | it was really odd | 16:41 |
| anteaya | I haven't seen any mention that anyone from rax checked logs to see why that was | 16:41 |
| *** dkliban_brb is now known as dkliban | 16:42 | |
| anteaya | in backscroll I think sdague asked monty if he could ask for more quota but if there was a response from monty, I didn't see it | 16:42 |
| jogo | anteaya: so a 'nova show' on the nodes that broke will give some insight | 16:42 |
| clarkb | I doubt we get more quota from hp | 16:43 |
| clarkb | we are already pushing the limits on what they give us | 16:43 |
| anteaya | jogo: it is failing in check | 16:43 |
| jogo | clarkb: too bad | 16:43 |
| jogo | anteaya: ? | 16:43 |
| anteaya | https://review.openstack.org/#/c/99796/ | 16:43 |
| anteaya | look at it, I see it as failing in the check queue | 16:44 |
| jogo | anteaya: thanks | 16:44 |
| anteaya | np | 16:44 |
| *** pblaho|afk is now known as pblaho | 16:44 | |
| jogo | well I guess it will be a while before this lands then | 16:44 |
| anteaya | :( | 16:44 |
| *** Hal_ has quit IRC | 16:45 | |
| *** pblaho has quit IRC | 16:45 | |
| jogo | clarkb: are there any thoughts around making the check queue two stages | 16:45 |
| clarkb | I havent no. | 16:45 |
| jogo | short running !integration jobs -> integration jobs. | 16:45 |
| jogo | that give folks less useful feedback if they fail pep8 or forget to update a unittest | 16:46 |
| mordred | goooooooooooooooooooooooooooooooooooooooooooooooooooooooool | 16:46 |
| jogo | but would free up some jobs | 16:46 |
| jogo | not sure if the tradeoff is worth it | 16:46 |
| clarkb | jogo they could just run them locally... | 16:46 |
| mordred | clarkb: I agree- I do not think we're getting any more quota from hp or rax right now | 16:46 |
| krtaylor | anteaya, yes, I'll be there, need anything? | 16:46 |
| *** jistr has quit IRC | 16:46 | |
| mordred | anteaya: ^^ | 16:46 |
| jogo | clarkb: and if they don't we don't spend hours of machine hours on there patch | 16:47 |
| jogo | mordred: thoughts ^ | 16:47 |
| anteaya | krtaylor: just wanted to know if you would be there | 16:47 |
| anteaya | thanks | 16:47 |
| jogo | mordred: too bad we can't get more quota we seem to be maxing out our quota every day now | 16:48 |
| anteaya | mordred can either you or clark make nodepool delete stuck nodes? | 16:48 |
| anteaya | please | 16:48 |
| mordred | jogo: we've gone back and forth on running non-integration patches first | 16:48 |
| jogo | mordred: and what is the latest? | 16:49 |
| mordred | jogo: I'm usually the champion of the idea, and I usually wind up beign wrong | 16:49 |
| *** flaviof has joined #openstack-infra | 16:49 | |
| clarkb | problem with it is you end up doing more iterations | 16:49 |
| clarkb | resulting in more cpu time | 16:49 |
| mordred | jogo: the biggest issue is that the unittests take as long as the integration tests (which is crazypants and shoudl diaf btw - unittests should never take more than 5 minutes really) | 16:49 |
| Alex_Gaynor | I think it also promotes people not running tests locally, and incentivizing that seems wrong | 16:49 |
| mordred | jogo: but they do - which means we double the best time through the gate | 16:49 |
| clarkb | anteaya: I have not yet booted machine with keys | 16:49 |
| sdague | clarkb: do you think that's still true? | 16:49 |
| anteaya | clarkb: ah | 16:50 |
| mordred | Alex_Gaynor: one of the problems is that we have a ton of quasi-functional tests in our unittests | 16:50 |
| jogo | clarkb: wow neutron takes 24 minutes | 16:50 |
| anteaya | clarkb: we need some nodes freed up to run tests | 16:50 |
| mordred | Alex_Gaynor: what I'd love to see is for more projects to break out functional and unittests | 16:50 |
| sdague | Alex_Gaynor: it's also not clear that the current incentives create the behavior that we think we want | 16:50 |
| mordred | and then run functional tests against an install similar to how the swift functional tests go | 16:50 |
| jogo | mordred: so we could do this for check queue only and not gate. but your argument is still valid | 16:50 |
| flaviof | hi folks. Trying to submit a change but it failed due to something unrelated to my patch: https://review.openstack.org/#/c/100130/ | 16:50 |
| *** ramashri has joined #openstack-infra | 16:51 | |
| flaviof | logs: http://people.redhat.com/~iwienand/100130/ | 16:51 |
| *** melwitt has joined #openstack-infra | 16:51 | |
| *** matjazp has joined #openstack-infra | 16:51 | |
| flaviof | is there someone in oslab in this channel who could help me? | 16:51 |
| mordred | Alex_Gaynor: but that's not going to happen anytime soon | 16:51 |
| trinaths | fungi: Hi | 16:52 |
| anteaya | trinaths: he is busy moving this week | 16:52 |
| anteaya | trinaths: what is on your mind | 16:52 |
| *** sweston has joined #openstack-infra | 16:53 | |
| anteaya | flaviof: red hat ci is ianw | 16:53 |
| jogo | clarkb: so you are only right for some of the bigger projects | 16:53 |
| anteaya | paging ianw | 16:53 |
| flaviof | anteaya: thanks | 16:53 |
| jogo | clarkb: look at 98868,5 or 98918,4 | 16:53 |
| anteaya | flaviof: np | 16:53 |
| jogo | in check queue | 16:53 |
| *** james_li has quit IRC | 16:53 | |
| trinaths | anteaya: okay. doubt on improving the performance of CI and some guidelines on Multi node setup for CI | 16:54 |
| anteaya | trinaths: do you feel like bringing them up at the third party meeting? | 16:54 |
| trinaths | anteaya: yes. | 16:55 |
| *** ociuhandu_ has joined #openstack-infra | 16:55 | |
| anteaya | which starts in an hour and will benift others? | 16:55 |
| *** ociuhandu has quit IRC | 16:55 | |
| *** ociuhandu_ is now known as ociuhandu | 16:55 | |
| *** sweston has quit IRC | 16:55 | |
| jogo | Alex_Gaynor: so I can see how the incentives can be wrong, you fail unit tests and now get really quick results. we can always just delay the results if this becomes an issue | 16:55 |
| anteaya | great, do add an agenda item to the agenda | 16:55 |
| trinaths | anteaya: sure. kindly please add this to the agenda | 16:55 |
| anteaya | trinaths: you can do that | 16:55 |
| anteaya | trinaths: you know what you want to dicuss | 16:55 |
| trinaths | anteaya: okay. can I edit that page ?? | 16:55 |
| anteaya | yes | 16:55 |
| trinaths | anteaya,https://wiki.openstack.org/wiki/Meetings/ThirdParty | 16:56 |
| anteaya | in the top right corner you should see a button allowing you to sign in | 16:56 |
| anteaya | use the same email and password as signing into gerrit | 16:56 |
| jogo | clarkb: also I don't think this would make thinks slower in check queuee all the time | 16:56 |
| trinaths | anteaya: yes doing | 16:56 |
| *** alugovoi has joined #openstack-infra | 16:56 | |
| anteaya | trinaths: k | 16:56 |
| jogo | clarkb: if you drop down to the bottom of the check queue many jobs are finished there !integration testing and are waiting for tempest nodes | 16:56 |
| clarkb | jogo: the issue is in aggregate pipeline throughput | 16:57 |
| *** sarob has joined #openstack-infra | 16:57 | |
| *** sweston has joined #openstack-infra | 16:57 | |
| *** _nadya_ has joined #openstack-infra | 16:57 | |
| clarkb | as a reviewing and code submitter I want a full picture upfront to avoid fixing lots of little issues in lots of patchcsets | 16:57 |
| jogo | clarkb: if we assume that we will be using 100 of our quot at all times, I think on average things will be faster | 16:57 |
| jogo | clarkb: you should just run your unit and style checks locally first anyway | 16:57 |
| clarkb | jogo: agreed | 16:58 |
| clarkb | so this shouldn't matter :) | 16:58 |
| jogo | clarkb: I agree that I would prefer to get all the jobs, but we have a finite number of cloud | 16:58 |
| mordred | except | 16:58 |
| clarkb | maybe that needs to be a review item | 16:58 |
| clarkb | -1 did not run tests locally | 16:58 |
| mordred | any time we say "you should ..." | 16:58 |
| mordred | we fail | 16:58 |
| jogo | mordred: true | 16:58 |
| clarkb | mordred: right. but I am asserting teh current state is better than the proposed state | 16:59 |
| jogo | clarkb: I am not convinced either way | 16:59 |
| mordred | not agreeing or disagreeing with that - just making sure we don't fall down the trap of expecting different behavior from our devs | 16:59 |
| jogo | clarkb: if we did this for 98909,4 | 16:59 |
| mordred | I believe that it is worth doing the mental exercise to consider what needs to happen to exist in a world where we are at max quota at all times | 16:59 |
| jogo | we would have 20 extra nodes | 17:00 |
| jogo | mordred: ++ | 17:00 |
| mordred | because thus far we have operated under teh assumption tha we can throw more nodes at the problem | 17:00 |
| mordred | if we have reached the point where that is no longer an assumption, it warrants careful consideration | 17:00 |
| jogo | mordred: well said. | 17:00 |
| clarkb | jogo: you would have 20 extra nodes now | 17:00 |
| trinaths | anteaya: done | 17:01 |
| clarkb | jogo: but then they would push a patch to fix pep8 and py26 py27 fails | 17:01 |
| anteaya | trinaths: thank you | 17:01 |
| clarkb | then you push a patch to fix py26 but py27 fails | 17:01 |
| clarkb | then you push a patch to fix py27 and node devstack fials | 17:01 |
| *** shayneburgess has quit IRC | 17:01 | |
| jogo | clarkb: perhaps, as mordred said I think we need to do the full thought exercise | 17:01 |
| clarkb | you fix devstack but now you find that you don't work with neutron | 17:01 |
| clarkb | so you fix neutron | 17:01 |
| *** habib has quit IRC | 17:01 | |
| clarkb | and now yo uhave done 5 round trips when one would have been sufficient | 17:01 |
| anteaya | trinaths: no as chair of the meeting, I may move agenda items around a bit, but if it at least posted somewhere in the agenda, we will get to it (or try based on time) | 17:01 |
| anteaya | s/no as chair/now as chiar | 17:02 |
| clarkb | mordred: I agree | 17:02 |
| anteaya | chair | 17:02 |
| mordred | clarkb: what if ... | 17:02 |
| anteaya | I can't spell | 17:02 |
| *** matjazp has left #openstack-infra | 17:02 | |
| mordred | clarkb: we added smarts (*waves hands*) such that the first iteration of any patch gets submitted to all of the things immediately | 17:02 |
| jogo | clarkb: and how would your scenario play out currently? | 17:02 |
| mordred | but if, at _any_ point, you fail pep8 or unittests | 17:02 |
| mordred | that change gets put into a category which requires it goes through two-stage verification | 17:03 |
| mordred | of running pep8/unit first, then integration | 17:03 |
| clarkb | jogo: all of those tests would fail togetehr and you can address them in one new patchset | 17:03 |
| mordred | so basically, "run them locally because we're going to put you in jail if you don't" | 17:03 |
| jogo | so what I am saying is: | 17:03 |
| sdague | mordred: so we could do this another way | 17:03 |
| *** sparkycollier has joined #openstack-infra | 17:03 | |
| jogo | pep8,py26,py27 fail. you see all of those together | 17:03 |
| trinaths | anteaya: ok | 17:03 |
| sdague | which is implement priority flag on patches | 17:03 |
| jogo | then once pass those, you run all of integration | 17:04 |
| sdague | because the biggest issue about a full pipeline is fixes that actually hep the full pipeline can't get cpu time in a timely manner | 17:04 |
| sdague | and then it turns in to the manual promote process | 17:04 |
| clarkb | sdague: ++ | 17:04 |
| *** shayneburgess has joined #openstack-infra | 17:04 | |
| sdague | which isn't a process | 17:04 |
| sdague | it's super adhoc | 17:04 |
| clarkb | which was intentional at the time | 17:05 |
| *** marcoemorais has quit IRC | 17:05 | |
| clarkb | but maybe experience shows it sucks | 17:05 |
| clarkb | and we need something else | 17:05 |
| jogo | sdague: so this is a mostly orthoginal issue no? | 17:05 |
| *** marcoemorais has joined #openstack-infra | 17:05 | |
| sdague | jogo: maybe | 17:05 |
| sdague | except if critical fixes can still get through in a timely manner | 17:05 |
| *** marcoemorais has quit IRC | 17:05 | |
| mordred | I thnk they're different - one is about the general case of trying to not waste resources | 17:05 |
| sdague | than the queue grinding for everything else becomes less urgent | 17:05 |
| *** marcoemorais has joined #openstack-infra | 17:06 | |
| clarkb | anteaya: the first node I am attempting to delete is taking its sweet time... | 17:06 |
| mordred | I think queue grinding is still quite urgent, even if it's not 'important' patches | 17:06 |
| mordred | because that's still 48 hour merge windows for developers | 17:06 |
| sdague | mordred: sure | 17:06 |
| *** amotoki_ has quit IRC | 17:06 | |
| mordred | I don't disagree that something like what you're saying couldn't be helpful | 17:06 |
| *** marcoemorais has quit IRC | 17:06 | |
| mordred | wait. | 17:06 |
| sdague | but honestly, I think we fix that one by throwing out tests | 17:06 |
| mordred | that's too many double negatives | 17:07 |
| mordred | sdague: I think we probably do _many_ thinkgs | 17:07 |
| *** marcoemorais has joined #openstack-infra | 17:07 | |
| jogo | mordred: I agree queue griding is urgent. it slows down our velocity | 17:07 |
| sdague | mordred: sure | 17:07 |
| *** yfried has joined #openstack-infra | 17:07 | |
| anteaya | clarkb: I think fungi uses a loop | 17:07 |
| mordred | which is why I was saying earlier that I believe since our operating assumptions may have changed, that we may need to do a deep dive on design | 17:07 |
| clarkb | anteaya: yeah I was testing first | 17:07 |
| jeblair | we could disable clean check or lengthen the window | 17:07 |
| anteaya | clarkb: then if one is stubborn it moves on to easier pickings and comes back again later | 17:07 |
| anteaya | clarkb: ah | 17:07 |
| anteaya | clarkb: stubborn nodes | 17:08 |
| jogo | mordred: a great topic to hash out in Germany | 17:08 |
| mordred | since "throw more nodes at it" may have hit its ceiling | 17:08 |
| jeblair | anteaya, clarkb: that's what nodepool itself does... | 17:08 |
| sdague | jeblair: I think both of those things make the situation worse, no? | 17:08 |
| clarkb | jeblair: yes, the apparent issue is somethign in nova | 17:08 |
| clarkb | jeblair: and it confuses nodepool | 17:08 |
| clarkb | fix is proposed to nova so we should see it arrive at some point in the future at rax | 17:08 |
| *** ilyashakhat__ has joined #openstack-infra | 17:09 | |
| jogo | jeblair: so looking at status.o.o/zuul now, top of gate is <2 hours old | 17:09 |
| jogo | but top of check is 5 hours old, and we have 182 jobs in check | 17:10 |
| clarkb | though I didn't follow the converstaion about deleting nodes too closely as I was dealing with ES | 17:10 |
| *** trinaths has quit IRC | 17:10 | |
| jogo | so maybe running less check jobs would help | 17:10 |
| jogo | clarkb: TL;DR if a node is in deleting(error) you cannot delete it right now | 17:10 |
| sdague | jogo: so the spike for the check queue started with proposal bot proposing hacking 0.9 to all the projects again, because some other requirement landed | 17:12 |
| sdague | thus creating 40 - 50 patches in check failing pep8 | 17:12 |
| jogo | sdague: that will happen with every requirements check change no? | 17:12 |
| sdague | jogo: probably | 17:12 |
| sdague | which is terrible | 17:12 |
| jogo | yes | 17:12 |
| sdague | and why I think we should roll back hacking | 17:13 |
| sdague | but I seem to be the only one of that opinion | 17:13 |
| jogo | that arguement can be made for every requirments change | 17:13 |
| jogo | sdague: that spike may also be because of the weekend ending etc | 17:14 |
| sdague | jogo: no, it really wasn't | 17:14 |
| sdague | I was awake when it happened | 17:14 |
| sdague | the entire check queue filled up all at once, and it was all hacking proposal | 17:14 |
| jogo | but its not full of them know is it? | 17:15 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Logstash: Modifying rewrite rules to allow kibana 3 https://review.openstack.org/93167 | 17:15 |
| sdague | probably not | 17:16 |
| sdague | so it looks like a concrete issue right now is we don't have any: check-tempest-dsvm-f20 nodes | 17:17 |
| *** trinaths has joined #openstack-infra | 17:17 | |
| sdague | which means we can't complete devstack patches | 17:17 |
| *** ociuhandu has quit IRC | 17:19 | |
| *** UtahDave has joined #openstack-infra | 17:20 | |
| *** basha has joined #openstack-infra | 17:20 | |
| *** markmcclain has joined #openstack-infra | 17:22 | |
| openstackgerrit | Alexandre Viau proposed a change to openstack-infra/config: Fixed typo: 'project.yaml' -> 'projects.yaml' https://review.openstack.org/100305 | 17:24 |
| boris-42 | anteaya yep it's me | 17:26 |
| boris-42 | anteaya thanks for review | 17:26 |
| anteaya | okay | 17:26 |
| sdague | anyone able to look into nodepool about the fact that there are no: devstack-f20 nodes in rotation? | 17:26 |
| anteaya | boris-42: a lot of stackforge projects are requiring folks to sign the cla and I don't know if that _is_ what you want of if you just copy/pasted | 17:26 |
| boris-42 | anteaya I want | 17:27 |
| boris-42 | anteaya this project should be a part of integrated program | 17:27 |
| boris-42 | anteaya cause in case of success all other projects will use this service | 17:27 |
| bodepd | how do you guys handle generating/storing private keys that are configured via Puppet? | 17:27 |
| boris-42 | anteaya so yep I would like to force CLA | 17:27 |
| anteaya | boris-42: great, state as such in a comment on the patch so we can keep that in mind in subsequent reviews | 17:28 |
| anteaya | boris-42: thanks for making that clear | 17:28 |
| boris-42 | anteaya sure | 17:28 |
| boris-42 | anteaya btw | 17:28 |
| boris-42 | anteaya one question | 17:28 |
| *** Ryan_Lane has joined #openstack-infra | 17:28 | |
| boris-42 | anteaya why only logaas-ptl should be able to push tags? | 17:28 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Logstash: Modifying rewrite rules to allow kibana 3 https://review.openstack.org/93167 | 17:29 |
| jeblair | i just logged in with lp openid no probs | 17:29 |
| boris-42 | anteaya maybe it's okay if all core team is able to that? | 17:29 |
| anteaya | jeblair: yes that got fixed on sunday, the issue now is with statusbot | 17:30 |
| jeblair | anteaya: does that mean it's fixed or is there a more specific problem? | 17:30 |
| jeblair | oh | 17:30 |
| jeblair | anteaya: is anyone working on that? | 17:30 |
| anteaya | jeblair: we can't change the channels statuss back | 17:30 |
| clarkb | sdague: I think it has to do with the node allocator predominantly allocating precise nodes because that is wat we need the most of | 17:30 |
| clarkb | jeblair: ^ | 17:30 |
| clarkb | sdague: unfortunately when we say boot 2 nodes the changes of both of them failing are high | 17:30 |
| *** _nadya_ has quit IRC | 17:30 | |
| anteaya | jeblair: other than knowing it is an issue, I don't know of anyone working on that, no | 17:30 |
| clarkb | which makes the allocator even worse | 17:30 |
| jeblair | anteaya: wow. i think that's a super critical issue | 17:31 |
| anteaya | jeblair: kk, glad you are back then | 17:31 |
| *** julim has joined #openstack-infra | 17:31 | |
| jeblair | anteaya: i'm going to drop what i'm doing and fix that | 17:31 |
| anteaya | jeblair: kk | 17:31 |
| jeblair | i'm kind of surprised no one else thinks it is | 17:31 |
| anteaya | jeblair: well can any non-core do anything to help? | 17:31 |
| anteaya | Ill do what I can to help if there is anything I can do | 17:31 |
| clarkb | jeblair: lots of non core can change topics | 17:32 |
| clarkb | er anteaya ^ | 17:32 |
| anteaya | I don't have channel permissions | 17:32 |
| jeblair | clarkb: fixing them one at a time would be a very bad solution | 17:32 |
| anteaya | or a list of channel topics before it was changed | 17:32 |
| jeblair | clarkb: because i'm about to change them all back regardless of any that may have been fixed one at a time | 17:32 |
| anteaya | boris-42: so yes, tagging is best done by one person | 17:32 |
| boris-42 | anteaya heh okay | 17:33 |
| *** james_li has joined #openstack-infra | 17:33 | |
| anteaya | boris-42: thanks | 17:33 |
| sdague | clarkb: ok, well how do you feel about fast approving me moving f20 to experimental then? | 17:33 |
| *** nati_ueno has joined #openstack-infra | 17:33 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 17:33 |
| *** mmaglana has joined #openstack-infra | 17:34 | |
| clarkb | sdague: let me check one more thing otherwise ++ | 17:34 |
| boris-42 | anteaya ^ fixed that | 17:34 |
| *** pelix has quit IRC | 17:34 | |
| sdague | because right now devstack changes are actually blocked. And in blocking them, we block actually fixing the grenade gate issue | 17:34 |
| anteaya | boris-42: k thanks | 17:34 |
| anteaya | boris-42: I'll review again at the end of my day | 17:35 |
| clarkb | sdague: ok confirmed that both rax and hp have f20 images so it isn't a case of being pinned to one provider | 17:35 |
| clarkb | sdague: ++ to moving to experimental | 17:35 |
| boris-42 | anteaya ok thanks=) | 17:35 |
| *** UtahDave has quit IRC | 17:35 | |
| *** julim has quit IRC | 17:35 | |
| *** reed has joined #openstack-infra | 17:36 | |
| *** ilyashakhat__ has quit IRC | 17:36 | |
| sdague | clarkb: so does min ready degrade under this condition? | 17:37 |
| clarkb | sdague: yes | 17:37 |
| sdague | could we actually make min ready a hard floor? | 17:38 |
| clarkb | sdague: ebcause when we are at quota it becomes proportional | 17:38 |
| clarkb | sdague: you can't when at quota | 17:38 |
| sdague | why? it seems that min ready should be an actual floor, with head room to expand beyond that | 17:39 |
| *** julim has joined #openstack-infra | 17:39 | |
| sdague | or some other min item | 17:39 |
| clarkb | sdague: because if I can boot 1 node right now | 17:39 |
| clarkb | what do I do? | 17:39 |
| clarkb | I can't boot in ready nodes | 17:39 |
| *** cindyo has joined #openstack-infra | 17:39 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer Infrastructure | docs http://ci.openstack.org | bugs https://launchpad.net/openstack-ci/ | https://git.openstack.org/cgit/openstack-infra/config/tree/" | 17:39 | |
| sdague | oh, so I guess min ready should fall back to min "out there somewhere" | 17:39 |
| clarkb | sdague: we may need a proper PID controller loop | 17:39 |
| anteaya | clarkb: how soon can you get to a computer with server keys? | 17:40 |
| clarkb | anteaya: I am doing it now | 17:40 |
| mordred | jeblair: how did you do that? I tried to figure out how to do it and failed | 17:40 |
| clarkb | anteaya: but mostly by hand because they fail randomly | 17:40 |
| sdague | the issue right now is there are actually no fedora nodes anywhere, in use or ready | 17:40 |
| mordred | jeblair: so there is clearly knowledge I should learn | 17:40 |
| clarkb | sdague: correct because it is proportional devstack-precise wins every time | 17:40 |
| *** e0ne has joined #openstack-infra | 17:40 | |
| jeblair | mordred: stack TOPIC #openstack-meeting :OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings | 17:40 |
| anteaya | clarkb: thanks and :( | 17:40 |
| jeblair | grr | 17:40 |
| jeblair | mordred: grep TOPIC statusbot_debug.log.2014-06-02 | 17:41 |
| jeblair | mordred: then emacs | 17:41 |
| cindyo | @anteaya I am helping Craig get python-monascaclient onto stackforge. Do you need a pypi project for python-monascaclient before creating the stackforge repo? | 17:41 |
| mordred | jeblair: ah. ok. I did not think directly enough. *facepalm* | 17:41 |
| anteaya | cindyo: refresh my memory of the url for the patch? | 17:41 |
| jeblair | mordred: to transform those lines into "/m chanserv topic #channel foo" | 17:41 |
| anteaya | and welcome, cindyo | 17:41 |
| jeblair | mordred: a lhf bug would be to have statusbot record those externally to a yaml file and be smart about restarting... | 17:42 |
| jeblair | mordred: the fundamental problem is that if statusbot restarts during an alert, it loses state | 17:42 |
| cindyo | thanks! They current url is https://github.com/hpcloud-mon/python-monclient, but expecting to see github.com/stackforge/python-monascaclient | 17:42 |
| mordred | jeblair: yah. that bit I grokked | 17:42 |
| anteaya | cindyo: sorry the gerrit url for the patch | 17:42 |
| openstackgerrit | A change was merged to stackforge/gertty: Add ctrl-o to help dialog https://review.openstack.org/97753 | 17:42 |
| anteaya | cindyo: https://review.openstack.org/#/c/99767/ | 17:43 |
| clarkb | anteaya: looks like the erally old nodes are the ones that don't want to work so I ahev an upper bound on the time as well and am looping over that now | 17:43 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: demote f20 to experimental https://review.openstack.org/100309 | 17:44 |
| anteaya | cindyo: so yes, according to clarkb's comment on this file: https://review.openstack.org/#/c/99767/1/modules/openstack_project/files/zuul/layout.yaml | 17:44 |
| sdague | clarkb: there it is | 17:44 |
| cindyo | anteaya: https://review.openstack.org/99767 | 17:44 |
| clarkb | cindyo: anteaya: or remove the pypi jobs | 17:44 |
| anteaya | clarkb: go you | 17:44 |
| anteaya | clarkb: slight increase in running jobs, thanks | 17:45 |
| *** annegentle has joined #openstack-infra | 17:46 | |
| clarkb | sdague: +2, maybe we can get jeblair and mordred to review as well | 17:46 |
| clarkb | I can manually submit it if neccesary | 17:46 |
| sdague | clarkb: great | 17:46 |
| jeblair | sdague clarkb: is f20 voting? | 17:46 |
| sdague | jeblair: nope | 17:46 |
| anteaya | cindyo: do you have what you need for your next step? | 17:47 |
| jeblair | who's pushing f20? | 17:47 |
| sdague | ianw | 17:47 |
| sdague | which I support | 17:47 |
| cindyo | anteaya,clarkb To create a new pypi PKG-INFO needed for registration, I would have to build it on the old repo. Is that what you want before creating the new stack forge repo? We ususally build and register projects from the repo. | 17:47 |
| sdague | but right now it's got the side effect that we can't make progress on any d-g or devstack patches | 17:48 |
| jeblair | are we going to loose inertia on that with this? (eg, are we close to making it voting, etc?) | 17:48 |
| clarkb | cindyo: no, you can just register it directly in pypi | 17:48 |
| clarkb | rather than using setup.py | 17:48 |
| mordred | I'd like to have a longer conversation about the idea of that at all | 17:48 |
| mordred | but we don't have to do that right now | 17:48 |
| cindyo | OK I'll do that, and let you know when it's ready | 17:48 |
| sdague | jeblair: I don't know, but clarkb just explained that we're basically going to be always in this situation at full quota, which isn't a recipe for success | 17:49 |
| jeblair | sdague: it will correct... i'm mildly hesitant to make a change like this because the order in which jobs are being filled isn't the preferred one | 17:49 |
| anteaya | cindyo: if you add a link to the pypi registration as a comment on the patch, that will help all subsequent reviewers of the patch | 17:49 |
| mordred | tl;dr- I'd don't think we should be in the business of testing linux distros - we have too many other things on our plate. If we're discussing 86-ing postgres, I don't think we should be adding base linux distros | 17:49 |
| cindyo | anteay: ok will do | 17:49 |
| anteaya | cindyo: thanks, welcome to infra | 17:49 |
| cindyo | sorry anteaya my spelling is not great | 17:49 |
| mordred | no matter how much I like all of my friends in the redhat world | 17:49 |
| sdague | mordred: that I'm fine with as well | 17:49 |
| anteaya | cindyo: we have a lot of fun here, do let me know anytime you want to learn more | 17:50 |
| *** e0ne has quit IRC | 17:50 | |
| anteaya | cindyo: neither is mine | 17:50 |
| anteaya | cindyo: I forgive you any spelling errors in advance | 17:50 |
| clarkb | anteaya: fwiw I got stuck on a less old node so for loop is going to be slow | 17:50 |
| mordred | I'd be happier moving the gate off of ubuntu and on to $somethingelse that isn't tied to a vendor if that would make things 'fairer' | 17:50 |
| jeblair | sdague, clarkb: queue systems don't work like that -- they either ultimately satisfy demand or don't. | 17:50 |
| cindyo | haha! and thanks for the help | 17:50 |
| anteaya | cindyo: for irc nicknames type the first 3 letters and then tab | 17:50 |
| cindyo | anteaya: ahh | 17:51 |
| anteaya | clarkb: slow progress is better than none | 17:51 |
| anteaya | cindyo: :D | 17:51 |
| sdague | jeblair: if you believe we'll see f20 nodes show up for testing in the near future, I'm good with some other approach | 17:51 |
| jeblair | sdague: so if we're in a position where we will never catch up, then we need to make fundamental changes. if we're just "busy" right now, then the nodes will eventually be allocated and the backlog will clear (and quickly) | 17:51 |
| anteaya | clarkb: the blue team is slowly creeping up | 17:51 |
| anteaya | the backlog cleared on the weekend | 17:52 |
| sdague | well check queue is + 1.5 times it's capacity at the moment | 17:52 |
| *** basha has quit IRC | 17:52 | |
| anteaya | for some reason all the nodes that were in delete state suddenly cleared at 0:00 utc saturday and I have no idea why | 17:52 |
| sdague | a party being the pent up demand from how terrible everything has been the last 2 weeks | 17:52 |
| *** blamar has joined #openstack-infra | 17:52 | |
| clarkb | jeblair: sdague: I thin ka PID controller could fullfil the demand better | 17:52 |
| jeblair | sdague: the operative metric is the zuul job queue | 17:52 |
| sdague | jeblair: ok | 17:52 |
| mordred | yolanda: we may want to discuss this in here so that jeblair can chime in | 17:52 |
| clarkb | as it would be smarter about the change over time and realize it hasn't booted and f20 node ina while | 17:52 |
| *** _nadya_ has joined #openstack-infra | 17:53 | |
| clarkb | but I would have to go do reading and maths to figure it out | 17:53 |
| sdague | clarkb: or actually have a hard floor | 17:53 |
| sdague | of in-use / ready | 17:53 |
| *** dims has quit IRC | 17:53 | |
| jeblair | clarkb: yeah, there are two issues: f20 isn't booting fast enough to make sdague happy because he wants the changes he's looking at to merge | 17:53 |
| mordred | jeblair: yolanda and I were discussing the nodepool dib patch, which she's close to completing. she has it working except is missing glance delete image things | 17:53 |
| jeblair | clarkb: versus not being able to satisfy demand at all | 17:53 |
| mordred | jeblair: I was suggesting that perhaps nodepool.deleteImage may not need to change at all - since once it's an image in the cloud, it should ultimately be managed identically | 17:54 |
| clarkb | mordred: correct, but you need to delete the local qcow2 | 17:54 |
| sdague | jeblair: so if you look at all the long standing changes in check, the ones > 3 hrs | 17:54 |
| *** dims has joined #openstack-infra | 17:54 | |
| sdague | they are all waiting on f20 nodes | 17:54 |
| sdague | and from what I can tell, there are no f20 nodes anywhere | 17:54 |
| mordred | clarkb: so you're thinking we should keep the local qcow on disk for the same time it's live in teh cloud? | 17:55 |
| clarkb | mordred: yes | 17:55 |
| mordred | clarkb: perhaps add some code to deleteImage to also find the image on disk and delete it? | 17:55 |
| mordred | yolanda: ^^ | 17:55 |
| jeblair | clarkb: sdague's change will alter the behavior he's immediately seeing; however, it has a minimal impact on the second thing. if the second thing is true to the degree that you think we will never see f20 nodes boot, then the system is unustainable regardless. | 17:55 |
| clarkb | mordred: they aren't super expensive and having it local may be nice | 17:55 |
| *** ilyashakhat__ has joined #openstack-infra | 17:55 | |
| jeblair | mordred: later | 17:55 |
| clarkb | jeblair: correct | 17:55 |
| mordred | jeblair: yah. just making sure it was in this channel for logging/scrollback | 17:55 |
| clarkb | jeblair: understood | 17:56 |
| jeblair | sdague: yeah, so what i'm saying is that the expected behavior when the system is busy is that those changes will queue in zuul until nodepool is able to satisfy the f20 demand, and then they will clear out quickly once it does | 17:56 |
| jeblair | sdague: (i'm not saying it's ideal, just expected) | 17:56 |
| yolanda | i think the same code we have when we upload the image to glance, we will need to have it to delete it, otherwise we will be filling glance with images that aren't used | 17:56 |
| sdague | jeblair: ok, so can this be fixed by increasing min ready? Because we added it to 2 core jobs, but it's at such a low allocation it basically never creates them under load | 17:57 |
| jeblair | sdague: if, however, we've crossed the threshold that every queuing system has where production exceeds consumption, then it's all over regardless and we need to do less or have more (clearly we are close to that point). | 17:57 |
| mordred | jeblair: ++ | 17:57 |
| *** marcoemorais has quit IRC | 17:58 | |
| *** marcoemorais has joined #openstack-infra | 17:58 | |
| yolanda | mordred, clarkb ^ | 17:58 |
| *** marcoemorais has quit IRC | 17:58 | |
| jeblair | sdague: i think what i'm trying to say is that unless we have crossed that threshold, this is not a system problem, this is a human perception problem. is the human perception problem bad enough to make the change you have proposed? | 17:58 |
| *** marcoemorais has joined #openstack-infra | 17:58 | |
| mordred | yolanda: one sec ... | 17:59 |
| sdague | jeblair: it's actually a system problem in that the fix for the #1 gate reset issue is currently stuck | 17:59 |
| clarkb | yolanda: nodepool already delete images out of glance | 17:59 |
| clarkb | yolanda: I don't think you need to implement that | 17:59 |
| *** basha has joined #openstack-infra | 17:59 | |
| sdague | and it's stuck on a non voting job | 17:59 |
| sdague | maybe that's a perception problem, I don't know. But that feels like a system problem to me. | 18:00 |
| *** rustlebee is now known as russellb | 18:00 | |
| *** jp_at_hp has quit IRC | 18:00 | |
| mordred | I think that's a perception problem | 18:00 |
| jeblair | sdague: i know what you mean, and i'm not trying to minimize it -- i'm just saying that assuming we have not crossed the threshold, the job will eventually run (it may take way longer than we are comfortable with) | 18:01 |
| mordred | yah. that's what I Was trying to type but jeblair said it better | 18:01 |
| sdague | ok, so based on usage graphs last week. | 18:01 |
| sdague | that's going to be Saturday | 18:01 |
| sdague | so when I speak of system problem, maybe I'm thinking about it as a system as a whole | 18:02 |
| yolanda | clarkb, so we leave the images in glance without removing them? no problem in leaving the remaining instances into glance? | 18:02 |
| sdague | because if we can't actually land the patches that fix the system to be more productive in a timely manner, then I think we're in a bad place | 18:03 |
| clarkb | yolanda: yes as the existing image management deletes old images | 18:03 |
| yolanda | also in glance? | 18:03 |
| *** marcoemorais has quit IRC | 18:03 | |
| clarkb | yolanda: I believe it will just work as expected if yo upopulated the image list db table with a uuid and a time | 18:03 |
| yolanda | i need to test it | 18:03 |
| clarkb | yolanda: yes, when we snapshot images in rax and hpcloud they go into glance | 18:03 |
| jeblair | sdague: it looks like last week the job queue dropped below quota each night | 18:03 |
| *** marcoemorais has joined #openstack-infra | 18:03 | |
| yolanda | ok, so tomorrow i'll do a full test of it, spinning nodepool instances and deleting them, to check the results | 18:04 |
| jeblair | except the night of the 13/14th, where the # waiting got close to but did not touch 0 | 18:04 |
| mordred | sdague: I think what you're characterizing as a system problem is that fact that we have never designed the system to handle "land this patch urgently" | 18:04 |
| sdague | mordred: that may be the case. | 18:04 |
| mordred | we've designed it to handle "land all of these patches eventually" | 18:05 |
| sdague | right | 18:05 |
| mordred | this, of course, does not produce pleasing experiences in urgent time | 18:05 |
| mordred | times | 18:05 |
| sdague | which makes healing the system within the system hard | 18:05 |
| *** harlowja has joined #openstack-infra | 18:05 | |
| sdague | which is what we've been dealing with | 18:05 |
| jeblair | sdague: okay, so landing your patch will make the system more constant and less bursty, and also have a minor reduction in usage. so it will help (to a degree) with all the things we're seeing | 18:06 |
| jeblair | sdague: and as you say it will unblock a fix for a gate-failing bug (though the window minimizes that impact somewhat) | 18:07 |
| *** e0ne has joined #openstack-infra | 18:07 | |
| sdague | jeblair: the window minimizes it there, but patches are taking a lot more round trips through check because of it | 18:07 |
| jeblair | sdague: *nod* | 18:08 |
| jeblair | sdague: i haven't done a thorough analysis, but i'd expect it to recover by tomorrow | 18:08 |
| sdague | jeblair: I would be surprised :) | 18:08 |
| sdague | because there is a lot of pent up demand | 18:08 |
| jeblair | sdague: so i think our decision is to weigh that against what we lose by removing f20 | 18:08 |
| sdague | because while you were out we were pushing back on people approving not bug fix stuff | 18:09 |
| sdague | because of the state of the gate | 18:09 |
| jeblair | sdague: (we are at about +250 jobs from where we were 7 days ago and wating hit 0 for 6+ hours after that) | 18:09 |
| *** alugovoi has quit IRC | 18:09 | |
| jeblair | sdague: so is it worth dropping f20 to get that in before tomorrow? | 18:10 |
| *** sarob has quit IRC | 18:10 | |
| sdague | I think so. But I'm biased on the fact that all my work to make grenade debuggable requires devstack and/or d-g changes | 18:11 |
| sdague | and, honestly, knowing that this is the edge condition, I don't every want f20 in the devstack check queue | 18:11 |
| *** thedodd has joined #openstack-infra | 18:11 | |
| sdague | until there is a guaruntee that at least some nodes will be available | 18:11 |
| reed | can someone look at this patch today please? https://review.openstack.org/#/c/99481/ | 18:12 |
| jeblair | sdague: what kind of guarantee would you like? and do you have someone in mind that can provide nodes to satisfy that guarantee? | 18:12 |
| sdague | jeblair: I would like a min floor on node types. Which is ready + in-use. | 18:13 |
| sdague | or an allocator based on time waiting | 18:13 |
| sdague | so that the job that was waiting the longest in the check got it's needs filled | 18:14 |
| *** jaypipes has quit IRC | 18:14 | |
| mordred | sdague: I think that's enough of a departure in the current thing that it might be better to bucket into the aforementioned "let's talk about the gate in the world of finite cloud resources" conversation | 18:16 |
| mordred | rather than as a response to a current stimulus | 18:16 |
| cindyo | clarkb: I'm having a security issue registering my project online. I can login to pypi, but when I try to submit I get this error...The user name or password you entered for area “pypi” on pypi.python.org:443 was incorrect. Make sure you’re entering them correctly, and then try again. | 18:16 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Publish osprofiler on pypy and add check for requirments https://review.openstack.org/100052 | 18:17 |
| sdague | mordred: that's fine | 18:17 |
| cindyo | clarkb: i am using the same username password as i did to login. What other password is needed? | 18:17 |
| sdague | I'm trying to figure out how to get out of this box though where we have a fix | 18:17 |
| sdague | that will impact our capacity | 18:17 |
| sdague | which we can't get past tests | 18:17 |
| *** sarob_ has joined #openstack-infra | 18:17 | |
| sdague | because of our capacity allocator | 18:17 |
| clarkb | cindyo that should be it | 18:17 |
| clarkb | maybe you have a stale cookie? | 18:18 |
| sdague | it feels like a very small box, and makes me claustrophobic | 18:18 |
| mordred | yah. I hear that | 18:18 |
| cindyo | clarkb: I'll try clearing cookies, I hope it's not that my email has a '-' in it. I noticed some python programs have problems with that | 18:19 |
| *** harlowja has quit IRC | 18:19 | |
| sdague | so I'd like to not always feel like I'm in the cheese shop sketch when trying to fix gate bugs :) | 18:19 |
| *** harlowja has joined #openstack-infra | 18:19 | |
| sdague | because, that's very draining | 18:19 |
| mordred | totally. but I think when we're drained is probably the wrong time to start poking large holes - precisely because of the effects of the claustropphobia | 18:20 |
| mordred | I think it's important that we figure out how to decrease the amount of time people feel like they are stuck in tiny boxes | 18:20 |
| sdague | so demoting f20 back to experiemental, where it was a week ago, doesn't seem like the poking a large hole | 18:21 |
| *** basha has quit IRC | 18:21 | |
| sdague | maybe it was a couple of weeks ago, time doesn't really have meaning during that last sprinting towards fixes | 18:21 |
| sdague | mordred: and I do agree with the sentiment | 18:22 |
| jeblair | sdague: wfm | 18:22 |
| jeblair | clarkb: is your work on deleting nodes having an effect? i notice a change | 18:23 |
| jeblair | (zuul waiting is trending down, used nodes is trending up) | 18:23 |
| *** basha has joined #openstack-infra | 18:23 | |
| jeblair | apropos, i just got an email from cloudwatt :) | 18:24 |
| sdague | so I could also use some additional review on - https://review.openstack.org/#/c/100253/ because dtroyer_zz doesn't appear to be around, and I think that simple fix will dramatically impact life | 18:24 |
| *** _nadya_ has quit IRC | 18:24 | |
| reed | jeblair, did you win something? | 18:24 |
| jeblair | reed: i'll let you know after i translate it from french | 18:24 |
| sdague | because ER doesn't show the full scope of the problem as it's actually hit on a lot of different services | 18:24 |
| jeblair | ah, i'm to follow a link to complete my registration | 18:25 |
| clarkb | jeblair: yes it is helping | 18:25 |
| sdague | and it's hard to finger print (... I have a patch I'm working on to make it a ton more clear) | 18:25 |
| clarkb | helped last week too | 18:25 |
| sdague | clarkb: can you stick that in a nightly cron? :) | 18:25 |
| jeblair | clarkb: do you understand it enough to describe what needs to change in nodepool? | 18:25 |
| mtreinish | sdague: would that cause the failure file not being written thing we were seeing before? | 18:26 |
| sdague | mtreinish: yep | 18:26 |
| *** basha has quit IRC | 18:26 | |
| clarkb | jeblair: aiui nodepool needs to assume even less that a deleted node will be deleted | 18:26 |
| clarkb | basically nova api bug that makes deletes noop on backend | 18:27 |
| clarkb | so you need to be even more forceful | 18:27 |
| mtreinish | man those boxes must be slow then. 3 secs for bash to launch in a screen window | 18:27 |
| clarkb | jogo ^ | 18:27 |
| sdague | mtreinish: so the crux of the issue is that because we are stuffing into bash in screen, until the bash process actually spawns and gets to the tty it wont' take | 18:27 |
| jeblair | clarkb: i'm unsure how it could do that -- it does not assume it is deleted until it disappears from the server list | 18:27 |
| clarkb | right but it deletes on a slowish interval iirc | 18:28 |
| sdague | mtreinish: well it seems to be a bell curve. We actually had this problem before, just in very extreme cases | 18:28 |
| clarkb | I honestly didnt follow the issue last week | 18:28 |
| clarkb | I had the same thought you did | 18:28 |
| sdague | but now that the majority of our nodes are hp 1.1, and they are slower, it's happening a lot | 18:28 |
| mtreinish | sdague: yeah that makes sense | 18:28 |
| clarkb | but jogo and comstud indicate it isnt sufficient | 18:28 |
| jeblair | jogo: help us out here :) | 18:29 |
| comstud | hm | 18:29 |
| sdague | mtreinish: I made it 3 seconds to hopefully get us completely out of the bell curve | 18:29 |
| comstud | reading | 18:29 |
| sdague | or make it super rare | 18:29 |
| mtreinish | sdague: hopefully that'll be enough I guess it depends on how slow things really are | 18:29 |
| comstud | What "isn't sufficient" ? | 18:29 |
| mtreinish | sdague: what about removing screen from grenade? That's probably a lot of work | 18:29 |
| *** nati_ueno has quit IRC | 18:29 | |
| clarkb | comstud we already delete nodes over and over on an interval | 18:29 |
| comstud | oh | 18:29 |
| comstud | yeah | 18:30 |
| comstud | there's a nova bug... | 18:30 |
| clarkb | byt for some reason rax nodes must be manually deleted | 18:30 |
| *** nati_ueno has joined #openstack-infra | 18:30 | |
| comstud | such that deletes are ignored | 18:30 |
| mtreinish | sdague: won't you also need to backport that change to the devstack stable branches? | 18:30 |
| sdague | mtreinish: yeh, I thought we might be able to do that, it's not working | 18:30 |
| sdague | mtreinish: most of the fails have been on the new side | 18:30 |
| comstud | clarkb: Yeah, if HP doesn't ever error during terminate, you don't hit this bug | 18:30 |
| comstud | but we're erroring during terminate | 18:30 |
| sdague | so yes, but this will help | 18:30 |
| *** timrc-afk is now known as timrc | 18:30 | |
| jeblair | clarkb, comstud: so i think the current algo is: issue delete; wait 10 minutes for it to disappear; if it doesn't, try again in 1 minute. | 18:30 |
| comstud | After that point... the deletes are stuck and cannot be deleted again via the API | 18:30 |
| comstud | because of a nova bug | 18:30 |
| comstud | jeblair: a patch from jogo is merging to fix the nova bug.. | 18:31 |
| comstud | However, I don't have a timeframe for us deploying that fix yet. | 18:31 |
| jeblair | comstud: oh, so we're under the impression that clarkb manually running a bunch of deletes is fixing things... | 18:31 |
| comstud | After that point... your logic would work | 18:31 |
| jeblair | comstud: but if what you're saying is true, that's ineffective? | 18:31 |
| comstud | jeblair: We're having to manually reset task_states to NULL in the DB | 18:32 |
| comstud | after that point... | 18:32 |
| sdague | clarkb: as https://review.openstack.org/#/c/100253/ has dtroyer +A, can we jump it to the gate queue? Then we don't have to wait for the f20 node to show up | 18:32 |
| comstud | yes, either manual deletes or your auotmated job should work | 18:32 |
| sdague | clarkb: it's passed all the other tests | 18:32 |
| comstud | assuming we don't error on delete again and have to reset task states again... and repeat repeat | 18:32 |
| comstud | hehe | 18:32 |
| jeblair | comstud: ok, so behind the scenes someone is resetting something that is causing api deletes to be able to be attempted again, and after that point, our attempts may work? | 18:32 |
| comstud | correct | 18:32 |
| *** sparkycollier has quit IRC | 18:33 | |
| jeblair | clarkb: so it's possible that leaving nodepool to do this on its own may work and you may not need the manual deletes -- it might just happen 10 minutes faster your way? | 18:33 |
| comstud | that makes sense to me | 18:33 |
| harlowja | lifeless thanks for all the good comments on those oslo-specs | 18:34 |
| harlowja | will adjust them in a few | 18:34 |
| lifeless | harlowja: yw | 18:34 |
| *** rlandy has quit IRC | 18:34 | |
| jeblair | comstud: thanks for enlightenment :) | 18:34 |
| comstud | hehe np | 18:34 |
| comstud | So jogo's patch will revert task state properly on delete errors | 18:35 |
| jeblair | clarkb, sdague: want me to force-merge the f20 patch? | 18:35 |
| *** tkelsey has quit IRC | 18:35 | |
| comstud | But.. like I said, I'm not sure when we'll be deploying it. | 18:35 |
| clarkb | jeblair: these nodes were 2 hours to 31 hours in delete state | 18:35 |
| comstud | I'm going to raise the issue shortly internally | 18:35 |
| clarkb | pretty sure it wont happen 10 minutes faster | 18:35 |
| clarkb | jeblair: go for it | 18:35 |
| comstud | Until then, we'll have to manually reset task_states for you all | 18:35 |
| comstud | OR better.. just fix our delete issues :) | 18:36 |
| jeblair | clarkb: then we should debug nodepool further to find out if it's not doing the algorithm i described | 18:36 |
| comstud | (actually it's not specific to delete) | 18:36 |
| sdague | jeblair: force merge would be fine as well | 18:36 |
| comstud | i'll be back shortly if you need anything else | 18:36 |
| sdague | f20 doesn't run in the gate | 18:37 |
| * comstud -> errand | 18:37 | |
| sdague | so getting it into the gate queue would also be fine | 18:37 |
| openstackgerrit | A change was merged to openstack-infra/config: demote f20 to experimental https://review.openstack.org/100309 | 18:37 |
| sdague | jeblair: ok, how about a gate move of https://review.openstack.org/#/c/100253 now? | 18:38 |
| sdague | then we can let the check report whenever it feels like it | 18:39 |
| jeblair | sdague: do you know about nodepool's "test job" feature? | 18:39 |
| sdague | I guess not | 18:39 |
| jeblair | sdague: it can run a jenkins job on a node before marking it ready (which it will only do if the job passes) | 18:40 |
| sdague | ok | 18:40 |
| *** Ryan_Lane has quit IRC | 18:40 | |
| *** basha has joined #openstack-infra | 18:40 | |
| jeblair | sdague: sorry, i have to run | 18:42 |
| sdague | ok | 18:42 |
| sdague | clarkb / mordred: gate move of https://review.openstack.org/#/c/100253 now? - it should dramatically help on fail rates | 18:42 |
| ildikov | hi | 18:44 |
| clarkb | doesnt it need jenkins +1? | 18:44 |
| ildikov | I would like to ask some questions if someone has a few minutes for me | 18:44 |
| clarkb | ildikov best to just ask | 18:46 |
| *** nati_ueno has quit IRC | 18:46 | |
| ildikov | clarkb: ok, I will briefly describe the issue then, thanks :) | 18:46 |
| sdague | clarkb: if you look at the jenkins output everything except f20 has voted | 18:48 |
| ildikov | in Ceilo we have some weird log entries in the py26 gate jobs: http://logs.openstack.org/40/99140/1/check/gate-ceilometer-python26/84983c8/console.html.gz | 18:48 |
| sdague | and it passed all the rest of it | 18:48 |
| sdague | and it's been waiting for the f20 node for 4 hrs | 18:48 |
| clarkb | right but it wont queue without it | 18:48 |
| sdague | clarkb: if you do it manually it will | 18:48 |
| ildikov | the log is kind of full of "I/O operation on closed file" messages | 18:48 |
| sdague | fungi was doing that last week | 18:49 |
| clarkb | ah ok | 18:49 |
| ildikov | the posted log is the log of a failed gate job | 18:49 |
| sdague | turns out we can be in both queues at once :) | 18:49 |
| sdague | the check job will just show up and vote eventually | 18:49 |
| ildikov | the other strange issue is that the successfull gate job looks almost the same: http://logs.openstack.org/40/99140/1/check/gate-ceilometer-python26/16d92de/console.html | 18:49 |
| clarkb | looks like next reset is ~5 minutes | 18:49 |
| clarkb | can promote then | 18:50 |
| ildikov | it only contains the "running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1}" text in addition | 18:50 |
| sdague | clarkb: honestly, I don't even need promote | 18:50 |
| sdague | just putting it in the gate queue is probably good enough | 18:50 |
| clarkb | oh | 18:50 |
| sdague | given that it's pretty short | 18:50 |
| clarkb | not sure I can do that without promote | 18:50 |
| clarkb | but will look | 18:51 |
| sdague | I thought they were 2 different things | 18:51 |
| ildikov | I was wondering if any of these issues ring a bell for anyone here, as in local environments we couldn't reproduce them | 18:51 |
| *** crc32 has joined #openstack-infra | 18:51 | |
| sdague | but that being said, a promote in 5 mins is cool as well | 18:51 |
| *** nati_ueno has joined #openstack-infra | 18:51 | |
| sdague | ok, stepping away for a few | 18:51 |
| clarkb | ildikov: sounds like you have a race in yourtets | 18:52 |
| clarkb | ildikov: and multiple tests are using the same file object one of which uses it after it is closed | 18:52 |
| clarkb | ah yup enqueue vs promote. I will just wait for reset and promot | 18:52 |
| ildikov | clarkb: shouldn't it affect the py27 gate job too? | 18:52 |
| clarkb | ildikov: not necessarily if it is a race | 18:52 |
| clarkb | races by nature don't always show up | 18:53 |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 18:53 |
| ildikov | clarkb: the I/O error is always present in the py26 job for a long while now and I've never seen it in the py27 gate logs | 18:53 |
| clarkb | it may be a timing issue in py26 | 18:54 |
| *** basha has quit IRC | 18:55 | |
| clarkb | I can't know for sure without actually debugging the test but that is were i would start | 18:55 |
| *** Sukhdev has joined #openstack-infra | 18:56 | |
| ildikov | clarkb: ok, sounds reasonable | 18:56 |
| boris-42 | clarkb hi there | 18:56 |
| ildikov | clarkb: thanks | 18:57 |
| boris-42 | clarkb hm one dummy question, can I add coverage task on every check | 18:57 |
| boris-42 | clarkb otherwise it's a bit useless... | 18:57 |
| *** UtahDave has joined #openstack-infra | 18:57 | |
| clarkb | boris-42: you acn but we don't because its slow | 18:57 |
| adam_g | sdague, with some fixes that were finally synced out to all devstack slaves, the *dsvm-virtual-ironic jobs are looking stable again. http://no-carrier.net/~adam/openstack/ironic_gate_status.html there's also a patch up to get everything we need cached on slaves at build time. assuming that merges and we remain >90%, when would be a good time to talk about getting it voting again? | 18:57 |
| clarkb | boris-42: I don't understand why it is useless in post though | 18:57 |
| clarkb | boris-42: having it in post shows you a trend | 18:57 |
| clarkb | it can't be used for review but anyone can run that locally | 18:58 |
| clarkb | if we can solve the coverage is slow problem then sure | 18:58 |
| boris-42 | clarkb but having it before merging patch shows exactly place that are not covered | 18:58 |
| clarkb | boris-42: sure and in the past we have weighed that against test time | 18:58 |
| boris-42 | clarkb so I don't need to do this manually | 18:58 |
| *** doddstack has joined #openstack-infra | 18:58 | |
| clarkb | and consensus has been don't do it if it makes tests even slower | 18:58 |
| *** mrda-away is now known as mrda | 18:58 | |
| boris-42 | clarkb ok sounds reasonable | 18:59 |
| boris-42 | clarkb but post is still useless for me=) | 18:59 |
| boris-42 | ^_^ | 18:59 |
| ildikov | boris-42: I have a BP in Ceilo for having a coverage gate job and the outcome of the summit session about this was also to do it in post, because it's too slow :( | 19:00 |
| trinaths | sweston: hi | 19:00 |
| sweston | trinaths: hello | 19:00 |
| boris-42 | imho tempest jobs are much slower=) | 19:00 |
| anteaya | trinaths sweston mind chatting in -dev? | 19:00 |
| trinaths | anteaya: okay | 19:00 |
| *** thedodd has quit IRC | 19:00 | |
| ildikov | boris-42: you don't have to convince me =) | 19:01 |
| boris-42 | ildikov lol | 19:01 |
| *** e0ne has quit IRC | 19:01 | |
| clarkb | boris-42: they actually aren't when you compare to say neutron and nova | 19:02 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Add Storyboard puppet module https://review.openstack.org/100321 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Added Authorization Header flag to storyboard module https://review.openstack.org/100322 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Make storyboard run over ssl https://review.openstack.org/100323 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Load storyboard projects from projects.yaml https://review.openstack.org/100324 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Load storyboard superusers from yaml file https://review.openstack.org/100325 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Storyboard token expiration increased https://review.openstack.org/100326 | 19:03 |
| clarkb | boris-42: this is slowly getting better but for more than a year neutron unittests took longer than tempest | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Make storyboard idempotent https://review.openstack.org/100327 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Put a default section header above the TTL https://review.openstack.org/100328 | 19:03 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: StoryBoard module bounces apache on update https://review.openstack.org/100329 | 19:03 |
| boris-42 | ^ spamer | 19:03 |
| nibalizer | krotscheck: ^ want to take a gander at that | 19:03 |
| boris-42 | =) | 19:03 |
| *** e0ne has joined #openstack-infra | 19:03 | |
| boris-42 | clarkb hm didn't run coverage in nova=) | 19:03 |
| krotscheck | wat | 19:03 |
| boris-42 | clarkb but for small projects they are quite faaast | 19:03 |
| clarkb | right but small projects are not 90% of the development effort | 19:03 |
| jogo | clarkb: catching up from lunch, you are seeing delete issues in hp as well? | 19:04 |
| clarkb | jogo: we are not | 19:04 |
| nibalizer | krotscheck: that was my attempt to push the current state of the storyboard module embeded in infra/config into puppet-storybaord | 19:04 |
| nibalizer | which i thought was the plan | 19:04 |
| krotscheck | nibalizer: Right. | 19:04 |
| boris-42 | clarkb so probably then it's not a big deal to enable it in small projects at least? | 19:04 |
| *** Ryan_Lane has joined #openstack-infra | 19:04 | |
| clarkb | boris-42: ya I don't care what small projects do | 19:04 |
| clarkb | particularly stackforge | 19:04 |
| krotscheck | nibalizer: My current plan was actually this - https://review.openstack.org/#/c/98007/ | 19:04 |
| mordred | nibalizer: hey - you don't feel like figuring out something hard (or maybe easy but I can't be bothered to figure it out) | 19:04 |
| boris-42 | clarkb ahhh | 19:04 |
| jogo | clarkb: kk, you were just trying to understand the bug? | 19:04 |
| clarkb | we should keep the integrated projects as 1:1 as possible | 19:04 |
| boris-42 | clarkb nice then I am going to enable it in rally | 19:04 |
| *** cp16net_ has quit IRC | 19:04 | |
| boris-42 | =) | 19:04 |
| clarkb | jogo: yes beacuse it doesn't make sense given the nodepool behavior | 19:04 |
| nibalizer | mordred: sure poke it at me | 19:05 |
| jogo | clarkb: no it doesn't hehe. the bug is delete if an instane is in deleting state becomes a no-op | 19:05 |
| jogo | so if stuck in deleting,error. | 19:05 |
| *** cindyo has quit IRC | 19:05 | |
| clarkb | jogo: right but if I manually run it which uses the same API its fine | 19:05 |
| mordred | nibalizer: https://review.openstack.org/#/c/98656/ - but if you read https://wiki.openstack.org/wiki/Stackalytics/HowToRun | 19:05 |
| nibalizer | krotscheck: aha! i suspected that review existed but failed at finding it | 19:05 |
| mordred | nibalizer: they run theirs in nginx using uwsgi | 19:06 |
| jogo | mordred: so silly question. you say we aren't going to get more quota from rax or hp in the near future? Can the foundation just buy us quota? | 19:06 |
| jogo | clarkb: waaa | 19:06 |
| mordred | nibalizer: I'd like to keep using apache for consistency with our other things | 19:06 |
| jogo | clarkb: you can manually delete nodes stuck in deleting,error state? | 19:06 |
| mordred | nibalizer: but it does seem to be sluggish just running in mod_wsgi | 19:06 |
| mordred | nibalizer: so the question is "how does one use uswgi with apache similar to their nginx example" | 19:07 |
| * mordred now runs and hides | 19:07 | |
| nibalizer | haha mk | 19:07 |
| jogo | clarkb: because at any given time you will have instances doing normal deletes | 19:07 |
| *** dims has quit IRC | 19:07 | |
| nibalizer | krotscheck: i'll poke at that review in a sec | 19:07 |
| clarkb | jogo yup seems to work | 19:07 |
| anteaya | pleia2: are you around today? | 19:08 |
| clarkb | but I really dont have brain power to debug today | 19:08 |
| anteaya | or still conferencing? | 19:08 |
| mordred | nibalizer: the change could probably be done as a follow on patch - I think that one is landable as-is | 19:08 |
| *** _nadya_ has joined #openstack-infra | 19:08 | |
| *** reed has quit IRC | 19:09 | |
| *** sarob_ has quit IRC | 19:09 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add rally coverage check and post tasks https://review.openstack.org/100330 | 19:09 |
| *** trinaths has quit IRC | 19:09 | |
| nibalizer | mordred: are you refereing to storyboard or stackalytics? | 19:10 |
| jogo | clarkb: can you give me your logs from that? | 19:10 |
| mordred | nibalizer: stackalytics | 19:10 |
| *** sarob_ has joined #openstack-infra | 19:10 | |
| jogo | clarkb: in particular 'nova show' from before you successfully delete something by hand | 19:10 |
| *** ilyashakhat__ has quit IRC | 19:10 | |
| anteaya | so we agreed in the third-party meeting that the driverlog page will say ci exists instead of ci tested? since that is the actual status it is currently communicating | 19:10 |
| anteaya | sdague: ^ | 19:10 |
| jogo | and the delete command you use | 19:10 |
| anteaya | and I am going for a walk | 19:11 |
| *** sweston has quit IRC | 19:11 | |
| jogo | sdague: '[OpenStack-Infra] change frequency of "Imported Translations from Transifex" | 19:12 |
| jogo | from the -infra ML, that may not be a bad idea | 19:12 |
| clarkb | jogo next time things back up sure | 19:12 |
| jogo | clarkb: kk, well what command did you use? | 19:13 |
| jogo | clarkb: a simple 'nova delete UUID' ? | 19:13 |
| *** sarob_ has quit IRC | 19:14 | |
| *** reed has joined #openstack-infra | 19:14 | |
| clarkb | no nodepool delete --now | 19:15 |
| jogo | clarkb: oh hrm, have any nodepool logs then? | 19:16 |
| jogo | did that work for all the nodes stuck in delete? | 19:16 |
| *** dims has joined #openstack-infra | 19:18 | |
| *** eglynn-regus has joined #openstack-infra | 19:18 | |
| clarkb | not all nodes | 19:19 |
| *** lcostantino has quit IRC | 19:20 | |
| jogo | clarkb: any idea of what % stayed around? | 19:20 |
| clarkb | low ~2% | 19:21 |
| *** eglynn-office has quit IRC | 19:21 | |
| *** arnaud__ has quit IRC | 19:22 | |
| *** arnaud__ has joined #openstack-infra | 19:22 | |
| *** 6JTAACWFP has joined #openstack-infra | 19:22 | |
| *** blamar has quit IRC | 19:23 | |
| *** lcostantino has joined #openstack-infra | 19:23 | |
| *** medieval1 has quit IRC | 19:23 | |
| *** nati_ueno has quit IRC | 19:23 | |
| *** cnesa has joined #openstack-infra | 19:23 | |
| *** medieval1 has joined #openstack-infra | 19:23 | |
| *** nati_ueno has joined #openstack-infra | 19:24 | |
| harlowja | lifeless let me know when u start to want help on that rpm spec stuff (for pbr semeantic stuffs) | 19:24 |
| harlowja | lifeless or if u need unit tests or something else, i can help with | 19:24 |
| *** doug-fish has quit IRC | 19:24 | |
| harlowja | *when u get there* | 19:24 |
| jogo | clarkb: hmm yeah, next time you do it let me know so I can see the logs | 19:24 |
| jogo | very strange | 19:24 |
| *** vhoward has quit IRC | 19:24 | |
| *** doug-fish has joined #openstack-infra | 19:25 | |
| lifeless | harlowja: it just needs your reviews atm - did you see the recent updates, and also my query on using the release version component ? | 19:25 |
| *** miqui has quit IRC | 19:25 | |
| harlowja | lifeless will check that out | 19:25 |
| mordred | harlowja: https://review.openstack.org/#/c/96608/ | 19:25 |
| harlowja | mordred yup, thx | 19:26 |
| *** miqui has joined #openstack-infra | 19:26 | |
| jogo | clarkb: dumb question did you try nodepool delete without '--now'? | 19:26 |
| *** adalbas has quit IRC | 19:26 | |
| *** mestery has quit IRC | 19:27 | |
| *** adalbas has joined #openstack-infra | 19:27 | |
| *** vhoward has joined #openstack-infra | 19:27 | |
| clarkb | jogo no as that is a noop | 19:28 |
| clarkb | it marks the node for deletion in the db | 19:28 |
| clarkb | which was already done | 19:28 |
| *** lcostantino has quit IRC | 19:28 | |
| jogo | do you have any nodepool logs? | 19:29 |
| mordred | jogo: we have SO MANY | 19:29 |
| jogo | mordred: so I want a log that has an attempted delete in it | 19:30 |
| jogo | mordred: any answer to if the foundation can just buy us more cloud? | 19:30 |
| mordred | jogo: I have not asked | 19:30 |
| jogo | because the way clarkb describes things, there *may* be a nodepool bug | 19:31 |
| jogo | mordred: kk, because isn't that part of there mission or something? | 19:31 |
| clarkb | more likely the nova bug breaks a nodepool expectation | 19:32 |
| clarkb | but ya | 19:32 |
| sdague | anteaya: are the bugs re: not accounting for any of the infra jobs going to be addressed as well? | 19:32 |
| annegentle | dumb bug triaging question: can only doc-core mark a milestone in a Launchpad bug? | 19:33 |
| jogo | clarkb: anyway some nodepool logs should shine some light on this | 19:33 |
| *** CaptTof__ has quit IRC | 19:33 | |
| *** tkelsey has joined #openstack-infra | 19:33 | |
| *** tkelsey has quit IRC | 19:33 | |
| openstackgerrit | Dan Bode proposed a change to openstack-infra/config: Remove dep on wget package for apt::key https://review.openstack.org/100333 | 19:34 |
| *** CaptTofu_ has joined #openstack-infra | 19:34 | |
| *** _nadya_ has quit IRC | 19:34 | |
| jogo | clarkb: perhaps a log with 'Could not delete node' in it | 19:34 |
| ildikov | annegentle: according to how it works in case of Ceilometer, yes | 19:35 |
| jogo | clarkb: or 'Exception deleting node' | 19:35 |
| *** rfolco has quit IRC | 19:35 | |
| ildikov | annegentle: but now I checked it, I'm not allowed to set, just like the importance option, it is also available only for cores | 19:38 |
| *** CaptTofu_ has quit IRC | 19:38 | |
| jogo | clarkb: wow I can see where you ran 'nodepool delete --now' | 19:39 |
| *** sweston has joined #openstack-infra | 19:39 | |
| *** krotscheck has quit IRC | 19:41 | |
| *** krotscheck has joined #openstack-infra | 19:42 | |
| *** james_li has quit IRC | 19:44 | |
| *** lcostantino has joined #openstack-infra | 19:44 | |
| *** nati_ueno has quit IRC | 19:44 | |
| *** dprince has quit IRC | 19:46 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Publish osprofiler on pypy and add check for requirments https://review.openstack.org/100052 | 19:47 |
| annegentle | ildikov: ok, thanks for testing! That's what ours does too | 19:47 |
| *** ramashri has quit IRC | 19:48 | |
| *** annegentle has quit IRC | 19:48 | |
| *** rfolco has joined #openstack-infra | 19:48 | |
| SlickNik | Can I get some eyeballs on https://review.openstack.org/#/c/98517/ when folks get a chance? Much appreciated - thanks! | 19:52 |
| *** praneshp has quit IRC | 19:52 | |
| *** james_li has joined #openstack-infra | 19:53 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack-infra/config: Add new project loggas https://review.openstack.org/100243 | 19:57 |
| *** james_li_ has joined #openstack-infra | 20:00 | |
| *** matjazp has joined #openstack-infra | 20:00 | |
| *** sweston has quit IRC | 20:00 | |
| *** rcarrillocruz has joined #openstack-infra | 20:01 | |
| *** james_li has quit IRC | 20:02 | |
| *** james_li_ is now known as james_li | 20:02 | |
| *** sarob has joined #openstack-infra | 20:02 | |
| *** matjazp_ has joined #openstack-infra | 20:03 | |
| *** Sukhdev has quit IRC | 20:03 | |
| *** rcarrill` has quit IRC | 20:04 | |
| *** marcoemorais has quit IRC | 20:04 | |
| *** marcoemorais has joined #openstack-infra | 20:04 | |
| *** matjazp has quit IRC | 20:04 | |
| *** matjazp_ has quit IRC | 20:07 | |
| *** eharney has quit IRC | 20:07 | |
| *** crc32 has quit IRC | 20:07 | |
| *** matjazp has joined #openstack-infra | 20:08 | |
| bodepd | nibalizer: jesusaurus for the decoupling of modules project, are you considering throwing things away where there might be a better community alternative? | 20:08 |
| *** rkukura_ has joined #openstack-infra | 20:09 | |
| *** sarob has quit IRC | 20:09 | |
| *** dhellmann has quit IRC | 20:10 | |
| *** loquacities has quit IRC | 20:10 | |
| *** sarob has joined #openstack-infra | 20:10 | |
| *** loquacities has joined #openstack-infra | 20:11 | |
| *** rkukura has quit IRC | 20:11 | |
| *** rkukura_ is now known as rkukura | 20:11 | |
| *** dhellmann has joined #openstack-infra | 20:11 | |
| *** nati_ueno has joined #openstack-infra | 20:14 | |
| *** dims has quit IRC | 20:14 | |
| *** sweston has joined #openstack-infra | 20:15 | |
| *** crc32 has joined #openstack-infra | 20:15 | |
| *** dims has joined #openstack-infra | 20:15 | |
| *** annegentle_ has joined #openstack-infra | 20:16 | |
| *** ramashri has joined #openstack-infra | 20:16 | |
| *** sarob has quit IRC | 20:17 | |
| boris-42 | clarkb https://review.openstack.org/#/c/100330/ <- just adding check and post cover jobs | 20:17 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Shift elasticsearch retention to 10 days. https://review.openstack.org/100339 | 20:19 |
| *** nati_ueno has quit IRC | 20:19 | |
| clarkb | sdague: mordred jeblair ^ I had been doing that by hand | 20:19 |
| clarkb | I just did it as I noticed ES backlog was increasing again and oen of the nodes had ~50GB free | 20:20 |
| *** matjazp has quit IRC | 20:20 | |
| jeblair | clarkb: ack +2 | 20:20 |
| clarkb | I am currently checking sar numbers to make sure we don't have another bad volume | 20:20 |
| jogo | so I found a nodepool bug | 20:22 |
| jogo | a instance can be in the deleted state | 20:22 |
| jogo | but 'wiatForServerDeletion' doesn't accept that as deleted | 20:22 |
| jogo | 'waitFor' | 20:23 |
| jeblair | jogo: why should it? | 20:23 |
| clarkb | hrm its possible 07 has a bad volume too | 20:23 |
| jogo | jeblair: because an instance can be in deleted state and appear in nova list | 20:23 |
| clarkb | await of 7701.91 on 07 :/ | 20:23 |
| jeblair | jogo: what does that mean? | 20:23 |
| clarkb | mordred: any luck pinging pvo? | 20:24 |
| jogo | jeblair: it means the instance was just deleted, instances don't always vanish from nova list instantaneously | 20:24 |
| jeblair | jogo: in particular, what is the certainty that instance will actually be truly deleted with no further action ever necessary on the part of the user? and how is that instance accounted for in quota? | 20:24 |
| clarkb | I am not sure it is worth spinnig up nodes until things work | 20:24 |
| clarkb | much better to get in touch with rax | 20:24 |
| *** e0ne has quit IRC | 20:24 | |
| jeblair | jogo: sure, but how long do you expect it to stay there? it will keep checking every 5 seconds | 20:24 |
| jogo | jeblair: to the first part, 100 %, for the quota double checking | 20:25 |
| *** CaptTofu_ has joined #openstack-infra | 20:25 | |
| jogo | jeblair: I am double checking how long something stays in deleted for | 20:25 |
| *** lyxus has joined #openstack-infra | 20:26 | |
| bodepd | I'm confused about why the pip class does not install pip? | 20:27 |
| *** CaptTofu_ has quit IRC | 20:30 | |
| *** ramashri has quit IRC | 20:31 | |
| *** sarob_ has joined #openstack-infra | 20:32 | |
| *** ramashri has joined #openstack-infra | 20:32 | |
| *** sweston has quit IRC | 20:32 | |
| jeblair | jogo: ok cool; i would characterize the behavior you describe as a deliberate choice to avoid trusting nova responses in case they are wrong; depending on the answers to those 3 questions (confidence, time in state, quota), it may not be necessary | 20:32 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: display grenade summary instead of all of it https://review.openstack.org/99333 | 20:32 |
| jeblair | jogo: but if the tradeoffs are minor (does not count toward quota but is expected to remain in deleted state for < 10 mins), it may not be worth reversing that assumption | 20:33 |
| jeblair | jogo: if the tradeoffs are larger (could remain in state for hours), it may be a good idea | 20:34 |
| *** praneshp has joined #openstack-infra | 20:34 | |
| *** radez is now known as radez_g0n3 | 20:34 | |
| jeblair | jogo: though if it counts toward quota, we should leave it as is (so that nodepool knows not to spin up a replacement yet) | 20:34 |
| *** eharney has joined #openstack-infra | 20:34 | |
| *** nati_ueno has joined #openstack-infra | 20:34 | |
| jogo | jeblair: thats what I am trying to sort out | 20:34 |
| *** markmcclain has quit IRC | 20:35 | |
| jeblair | jogo: yup, and thanks. mostly brain dumping so i don't block you because i have to run afk again. :) | 20:35 |
| *** sweston has joined #openstack-infra | 20:35 | |
| jogo | jeblair: thanks | 20:36 |
| *** radez_g0n3 is now known as radez | 20:36 | |
| *** rcarrill` has joined #openstack-infra | 20:36 | |
| *** julim has quit IRC | 20:38 | |
| *** rcarrillocruz has quit IRC | 20:38 | |
| reed | hi guys, izinovik has a problem signing the iCLA on gerrit, I have run out of options and need some help on gerrit | 20:38 |
| *** nati_ueno has quit IRC | 20:39 | |
| anteaya | reed: what is the error message? | 20:40 |
| *** eharney has quit IRC | 20:40 | |
| *** eharney has joined #openstack-infra | 20:41 | |
| izinovik | anteaya, error message is following: | 20:41 |
| izinovik | fatal: ICLA contributor agreement requires current contact information. | 20:41 |
| izinovik | Please review your contact information: | 20:41 |
| izinovik | https://review.openstack.org/#/settings/contact | 20:41 |
| izinovik | fatal: The remote end hung up unexpectedly | 20:41 |
| anteaya | izinovik: hi | 20:41 |
| izinovik | while doing `git review' | 20:41 |
| izinovik | andreykurilin_, hello | 20:41 |
| anteaya | do you know about paste.o.o | 20:41 |
| izinovik | anteaya, hello | 20:41 |
| izinovik | yeah | 20:41 |
| anteaya | great thanks, so next time | 20:41 |
| izinovik | I mean yes | 20:41 |
| reed | izinovik, ah, so that's not the issue you listed | 20:41 |
| anteaya | so that is sign up with the foundation | 20:42 |
| anteaya | I do believe3 | 20:42 |
| reed | izinovik, go to https://review.openstack.org/#/settings/contact first | 20:42 |
| andreykurilin_ | izinovik, hi | 20:42 |
| reed | and follow the instructions on that page to sign the iCLA | 20:42 |
| reed | actually, it's here https://review.openstack.org/#/settings/agreements | 20:43 |
| izinovik | yes, i see that status of my agreement is 'verified' | 20:43 |
| reed | and if you go to https://review.openstack.org/#/settings/contact and add your contact information there, can you save them? | 20:44 |
| izinovik | after pressing 'Save changes' nothing bad happend | 20:45 |
| reed | ok... so try git review again now please | 20:45 |
| *** eharney has quit IRC | 20:45 | |
| *** alexpilotti has quit IRC | 20:45 | |
| *** sarob_ has quit IRC | 20:45 | |
| *** sarob_ has joined #openstack-infra | 20:46 | |
| *** vhoward has quit IRC | 20:46 | |
| izinovik | same result, ICLA requires current contact information | 20:46 |
| Alex_Gaynor | Is there a way to get emails on a particular issue besides adding yourself as areviewer? | 20:46 |
| *** eharney has joined #openstack-infra | 20:46 | |
| izinovik | my gitreview.username is set to izinovik | 20:46 |
| izinovik | is it right? | 20:47 |
| reed | izinovik, I think so but I'm no expert there... anteaya? | 20:47 |
| *** annegentle_ has quit IRC | 20:47 | |
| jesusaurus | bodepd: im open to switching to community modules if the transition is an easy one | 20:48 |
| anteaya | izinovik: go to settings | 20:48 |
| anteaya | what does it say your username is? | 20:48 |
| *** 6JTAACWFP has left #openstack-infra | 20:48 | |
| anteaya | Alex_Gaynor: not that I am aware | 20:48 |
| anteaya | Alex_Gaynor: you could set it as a watched repo | 20:49 |
| anteaya | Alex_Gaynor: and set your email settings for watched repos | 20:49 |
| Alex_Gaynor | anteaya: I definitely don't want every single nova change, just this one :-) | 20:49 |
| *** vhoward has joined #openstack-infra | 20:49 | |
| anteaya | Alex_Gaynor: ah | 20:49 |
| anteaya | just a reviewer then I think | 20:49 |
| Alex_Gaynor | whatever, adding myself as a reviewer is fine | 20:49 |
| anteaya | for just one patch | 20:49 |
| anteaya | kk | 20:49 |
| izinovik | anteaya, on 'settings' page I see that Username: izinovik, Email Address: izinovik@mirantis.com, Registered: May 12, 2014 3:24 PM | 20:50 |
| *** sarob_ has quit IRC | 20:50 | |
| *** marcoemorais has quit IRC | 20:50 | |
| *** e0ne has joined #openstack-infra | 20:50 | |
| anteaya | then that is your username | 20:50 |
| nibalizer | bodepd: i would like to see a default-to-upstream model | 20:51 |
| *** marcoemorais has joined #openstack-infra | 20:51 | |
| nibalizer | since maintaing our own modules for everything is blegh | 20:51 |
| anteaya | paste git review -v please izinovik | 20:51 |
| *** radez is now known as radez_g0n3 | 20:52 | |
| izinovik | anteaya, http://paste.openstack.org/show/84218/ | 20:52 |
| *** e0ne has quit IRC | 20:53 | |
| *** maxbit has quit IRC | 20:53 | |
| *** maxbit has joined #openstack-infra | 20:53 | |
| anteaya | izinovik: next time you paste please include the line with the command you ran | 20:53 |
| *** ihrachyshka has joined #openstack-infra | 20:53 | |
| izinovik | anteaya, ok | 20:54 |
| anteaya | izinovik: did you put in any contact information? | 20:54 |
| *** lukego has quit IRC | 20:55 | |
| izinovik | anteaya, where? in `git config --local` or in `git config --global` ? | 20:55 |
| anteaya | https://review.openstack.org/#/settings/contact | 20:56 |
| anteaya | did you put in any contact information | 20:56 |
| izinovik | anteaya, I only filled Full Name and Preferred Email | 20:56 |
| anteaya | I think it needs something more | 20:56 |
| izinovik | anteaya, all other field on page are empty | 20:57 |
| anteaya | like contact information | 20:57 |
| anteaya | yes | 20:57 |
| anteaya | it will be | 20:57 |
| anteaya | since that information is stored in another db | 20:57 |
| anteaya | not the gerrit db | 20:57 |
| anteaya | but like the message says | 20:57 |
| anteaya | it is looking for contact information | 20:57 |
| izinovik | anteaya, collegue of mine has only full name and preferred address and his `git review` work as expected | 20:57 |
| anteaya | izinovik: when did you collegue sign up? | 20:58 |
| *** amotoki_ has joined #openstack-infra | 20:58 | |
| izinovik | anteaya, I do not know | 20:58 |
| *** markmcclain has joined #openstack-infra | 20:58 | |
| anteaya | well the system is looking for some contact information | 20:58 |
| anteaya | you don't have to give it some | 20:58 |
| anteaya | but that is what it is looking for | 20:59 |
| izinovik | anteaya, strange I filled country and hitted 'save changes' and nothing happened | 20:59 |
| *** markmcclain1 has joined #openstack-infra | 20:59 | |
| anteaya | have you tried git review since you filled in country? | 20:59 |
| *** markmcclain1 has quit IRC | 20:59 | |
| *** sarob_ has joined #openstack-infra | 20:59 | |
| reed | I think the system needs also mailing address | 20:59 |
| anteaya | I don't know | 20:59 |
| anteaya | sometimes it just needs country | 21:00 |
| anteaya | or did at one point | 21:00 |
| reed | but I expect gerrit to throw a meaningful error message | 21:00 |
| *** marcoemorais has quit IRC | 21:00 | |
| anteaya | let's see what git review does now | 21:00 |
| anteaya | reed: it does | 21:00 |
| *** markmcclain1 has joined #openstack-infra | 21:00 | |
| *** marcoemorais has joined #openstack-infra | 21:00 | |
| anteaya | since this db isn't accessible by gerrit, it doesn't know what is in there | 21:00 |
| izinovik | anteaya, only after filling all fields and hitting 'save changes', as result I see message that "Contact information last updated on Jun 17, 2014 at 12:59 AM." | 21:00 |
| anteaya | izinovik: what does git review do now? | 21:01 |
| *** marcoemorais has quit IRC | 21:01 | |
| izinovik | anteaya, yes, now it works | 21:01 |
| anteaya | great | 21:01 |
| anteaya | welcome to openstack | 21:01 |
| anteaya | happy contributing | 21:01 |
| anteaya | reed: so we have no way of knowing what is or is not required in the contact info fields other than going through this painful process | 21:02 |
| *** mbacchi has quit IRC | 21:02 | |
| anteaya | since changes are made to requirements and I dont' know what they are | 21:02 |
| *** marcoemorais has joined #openstack-infra | 21:02 | |
| anteaya | apparently now all fields must have content | 21:02 |
| izinovik | thank you for all help openstackers | 21:02 |
| *** marcoemorais has quit IRC | 21:03 | |
| anteaya | izinovik: welcome | 21:03 |
| *** markmcclain has quit IRC | 21:03 | |
| reed | anteaya, nothing should have changed, all fields have always been required | 21:03 |
| anteaya | no | 21:03 |
| *** marcoemorais has joined #openstack-infra | 21:03 | |
| *** rcarrill` has quit IRC | 21:03 | |
| anteaya | my contact information is a whitespace | 21:03 |
| reed | anteaya, if not, then that was a bug... | 21:03 |
| reed | anteaya, something should be there | 21:03 |
| anteaya | well the bug got fixed then | 21:03 |
| jogo | clarkb: so I am digging into nodepool code | 21:03 |
| anteaya | I guess | 21:03 |
| jogo | and got a question | 21:03 |
| *** marcoemorais has quit IRC | 21:03 | |
| *** marcoemorais has joined #openstack-infra | 21:04 | |
| jogo | where is the logic for 'if delete doesn't work try again in 10 minutes' | 21:04 |
| reed | anteaya, whitespace is 'something' although you may be violating the bylaws if you don't state the truth | 21:04 |
| reed | and firefox decided to drive me crazy today | 21:05 |
| *** Sukhdev has joined #openstack-infra | 21:05 | |
| mordred | jeblair, clarkb: digging into the ansible nova code this weekend, I learned a few things that would could potentially make use of ... but also, as expected, learned that we currently know a LOT about spinning up nodes :) | 21:05 |
| *** marcoemorais has quit IRC | 21:06 | |
| anteaya | reed: it is the truth, whitespace is what I give to everyone who requests my contact info | 21:06 |
| anteaya | and if ever you need me, you know where to find me | 21:06 |
| *** marcoemorais has joined #openstack-infra | 21:06 | |
| clarkb | jogo: its a cron in nodepool | 21:06 |
| *** denis_makogon has quit IRC | 21:06 | |
| reed | anteaya, :) this channel is logged: you need to follow the bylaws | 21:07 |
| jogo | clarkb: link? | 21:07 |
| *** CaptTofu_ has joined #openstack-infra | 21:08 | |
| jogo | clarkb: because I previously saw a situation where 'a node is deleted but you have to send a second delete to actually get it to go go away. trying to reproduce that now | 21:09 |
| bodepd | jesusaurus: nibalizer the example that I have in mind is: https://github.com/jenkinsci/puppet-jenkins | 21:09 |
| nibalizer | bodepd: i had suspected | 21:10 |
| nibalizer | so i think integrating that is a great idea but gonna be a ton of work | 21:10 |
| nibalizer | but i also haven't done much investigation into just how much to do | 21:10 |
| *** isviridov is now known as isviridov|away | 21:11 | |
| clarkb | jogo: https://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/nodepool.py#n1142 | 21:11 |
| nibalizer | and i would prioritize getting off of custom vcsrepo and custom apache above jenkins i think | 21:11 |
| *** dhellmann has quit IRC | 21:11 | |
| *** loquacities has quit IRC | 21:11 | |
| *** shayneburgess has quit IRC | 21:11 | |
| jogo | clarkb: ahh periodic cleanup thanks | 21:11 |
| *** marcoemorais has quit IRC | 21:12 | |
| *** dhellmann has joined #openstack-infra | 21:12 | |
| *** marcoemorais has joined #openstack-infra | 21:13 | |
| mordred | nibalizer: I _believe_ we're going to hand to land patches to upstream vcsrepo to get off of our custom one | 21:13 |
| mordred | nibalizer: if you wanted to, you know, accomplish that, I would buy you a chicken | 21:13 |
| *** loquacities has joined #openstack-infra | 21:14 | |
| nibalizer | oh are there github pulls i can review? | 21:14 |
| mordred | nibalizer: the _main_ issue, iirc, is a philosophical one - the vcsrepo seems to try to go out of its way to do the right thing for a developer laptop - that is, it tries to not break things if you did somethign weird | 21:14 |
| anteaya | reed: I do follow the bylaws | 21:14 |
| anteaya | logged or not | 21:14 |
| mordred | nibalizer: whereas we want it for production purposes, so when we way we want ref X - we WANT REF X | 21:14 |
| mordred | nibalizer: nope. never got that far | 21:14 |
| mordred | nibalizer: also, my ruby is crappy | 21:14 |
| nibalizer | hrm well maybe we can add a a froce => true | 21:14 |
| mordred | nibalizer: I _think_ that was the main issue we had | 21:15 |
| nibalizer | ya i was hoping to get bodepd to look at it since my ruby is pretty weak too | 21:15 |
| bodepd | what feature do you get by conneting zuul to swift? | 21:18 |
| *** ociuhandu has joined #openstack-infra | 21:18 | |
| *** mfer has quit IRC | 21:18 | |
| *** james_li_ has joined #openstack-infra | 21:19 | |
| bodepd | nibalizer: the alternative is just to patch the openstack one with all of the stuff that I need | 21:20 |
| bodepd | nibalizer: but given that the other jenkins module was already written as an upstream, it seems like a better path | 21:20 |
| *** matjazp has joined #openstack-infra | 21:21 | |
| *** james_li has quit IRC | 21:21 | |
| *** james_li_ is now known as james_li | 21:21 | |
| nibalizer | bodepd: do you need functionality on the master side or the slave too? | 21:23 |
| clarkb | bodepd: the intent is to haev single use upload urls per job | 21:23 |
| clarkb | bodepd: so that we can stop copying things to our rapidly shrinking disk array | 21:23 |
| clarkb | jeblair: ^ btw there is a bug fix to zuul that was merged last week. we shoul restart it once we settle in | 21:23 |
| clarkb | jeblair: it is only for the swift uploads so not super urgent | 21:24 |
| *** matjazp has quit IRC | 21:25 | |
| *** mmaglana has quit IRC | 21:26 | |
| harlowja | lifeless was there anything u wanted me to check using my rpm version spec checker tool | 21:27 |
| *** pdmars has quit IRC | 21:27 | |
| *** otherwiseguy has joined #openstack-infra | 21:29 | |
| *** shayneburgess has joined #openstack-infra | 21:29 | |
| *** matjazp has joined #openstack-infra | 21:33 | |
| *** nati_uen_ has joined #openstack-infra | 21:34 | |
| bodepd | nibalizer: I am still building reviewing everything to build out a master (I just have two patches for that) | 21:40 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/infra-specs: Multiple Data Dirs proposal https://review.openstack.org/100363 | 21:41 |
| bodepd | nibalizer: I expect to have to write my own slave | 21:41 |
| nibalizer | wooo i'm specing | 21:41 |
| bodepd | clarkb: what things were being copied? so if I omit that config, it just writes to disc? | 21:42 |
| nibalizer | bodepd: do you have a review submitted that slots out the internal jenkins module for the public one? | 21:42 |
| *** CaptTofu_ has quit IRC | 21:42 | |
| jogo | clarkb: so I don't think the cronjob is working | 21:42 |
| nibalizer | cus if you do i'll work off of that, otherwise i can start the process | 21:43 |
| *** CaptTofu_ has joined #openstack-infra | 21:43 | |
| nibalizer | also what-to-do about jjb :P | 21:43 |
| clarkb | bodepd all of the test logs. we arent using it. if you omit it then you will need a jenkins plugin or something to do it for you | 21:43 |
| clarkb | jogo its working to an extent | 21:43 |
| clarkb | jogo we arent leaking all nodes | 21:43 |
| jogo | clarkb: when I found a failed delete, I don't see any second attempts in teh logs | 21:43 |
| jogo | clarkb: I think you aren't leaking because the regular delete usually works | 21:44 |
| sweston | anteaya: hello !! | 21:44 |
| *** matjazp has quit IRC | 21:44 | |
| jogo | and I thought you doing a manual 'nodepool delete --now' implies we were leaking | 21:44 |
| clarkb | we are leaking just not every node | 21:45 |
| clarkb | I suppose its possible the first delete works | 21:45 |
| sweston | anteaya: I do believe that I have what you wanted ... https://review.openstack.org/100357 | 21:46 |
| anteaya | sweston: hi | 21:46 |
| jogo | clarkb: yeah thats what I am thinking | 21:46 |
| jogo | clarkb: so as far as I can tell (and I may be way off) | 21:46 |
| jogo | what is happening is: | 21:47 |
| jogo | we do delete a server, and wait for it to be deleted | 21:47 |
| anteaya | sweston: well for starters I think I asked you to work with at most 5 patches | 21:47 |
| anteaya | sweston: on sandbox | 21:47 |
| *** andreykurilin_ has quit IRC | 21:47 | |
| anteaya | sweston: can you abandon a bunch of those please | 21:47 |
| *** CaptTofu_ has quit IRC | 21:47 | |
| anteaya | sweston: you don't need a new patch to test a new patchset | 21:48 |
| jogo | if we hit a timeout (600 seconds), then nodepool records a stacktrace and moves on with its life, assuming the cron task will cleanup | 21:48 |
| *** james_li has quit IRC | 21:48 | |
| sweston | anteaya: yes, apologies .. I was running a script .. I created a new one, but I am so tired I must have been running the wrong one | 21:49 |
| jogo | but in the logs I cannot find a second attempt at 'cleanupServer' for nodes that failed to delete the first time | 21:49 |
| anteaya | sweston: let's address that first please | 21:49 |
| jogo | why something doesn't fully delete the first time, I'm not sure. but I thought I found cases where that happened | 21:49 |
| jogo | and a second nova delete was needed | 21:49 |
| clarkb | yes nova does that | 21:50 |
| sweston | anteaya: ok, working on it | 21:50 |
| clarkb | it is the reason the cron exists | 21:50 |
| jogo | clarkb: so /me thinks cron is broken | 21:50 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/nodepool: Add log to node cleanup task https://review.openstack.org/100368 | 21:51 |
| jogo | ^ may help us validate my theory | 21:52 |
| clarkb | jogo thank you for looking | 21:52 |
| jogo | if we never see that log I am right, if we see it I don't know | 21:52 |
| jogo | clarkb: np | 21:53 |
| *** CaptTofu_ has joined #openstack-infra | 21:53 | |
| *** smarcet has quit IRC | 21:53 | |
| sdague | krotscheck: do you have an opinion on this - https://review.openstack.org/#/c/97788/ ? | 21:54 |
| krotscheck | sdague: Urm. | 21:54 |
| krotscheck | sdague: Yes. | 21:54 |
| sdague | ok, I want that opinion :) | 21:55 |
| sdague | because it seems odd to me | 21:55 |
| sdague | but I'm not educated enough to know | 21:55 |
| krotscheck | sdague: I’m trying to out how to make this make sense. | 21:55 |
| krotscheck | If used correctly, it’s a tool. | 21:55 |
| krotscheck | It not, it’s going to make the upstream package managers reject everything that comes out of horizon. | 21:56 |
| krotscheck | Nodeenv _compiles_ node from source. | 21:56 |
| sdague | interesting.... | 21:56 |
| krotscheck | And node is one of those stupid packages which compiles in dependencies. | 21:57 |
| *** shayneburgess has quit IRC | 21:57 | |
| krotscheck | Like, for example, heartbleed. | 21:57 |
| sdague | nice | 21:57 |
| sdague | so it looks like they only want to use it for testing | 21:57 |
| sdague | do you consider that an acceptable boundary? | 21:57 |
| *** mriedem has quit IRC | 21:57 | |
| krotscheck | sdague: I’m not a package manager. | 21:57 |
| sdague | basically I'm trying to figure out if I'm -1ing or +2ing it | 21:57 |
| *** lcostantino has quit IRC | 21:58 | |
| sdague | well if it's only in test-requirements, it doesn't need to be packaged | 21:58 |
| krotscheck | sdague: Yes, but I don’t know how strict they are about package requirements for testing. Do the rquire the ability to fully test a package only from upstream repositories? | 21:58 |
| sdague | I don't know. | 21:59 |
| krotscheck | Right. | 21:59 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Add node def for puppet3 master https://review.openstack.org/83678 | 21:59 |
| sweston | anteaya: ok, I saved three. the rest are abandoned | 21:59 |
| sdague | but, being that it's redhat proposing, I'm going to assume they sort of figured that out. | 21:59 |
| sdague | at least it sounds like it's not an insane idea if done correctly | 21:59 |
| *** julim has joined #openstack-infra | 22:00 | |
| sdague | oh, they are doing it for style enforcement - https://review.openstack.org/#/c/97237/ | 22:00 |
| *** nati_uen_ has quit IRC | 22:00 | |
| *** nati_ueno has joined #openstack-infra | 22:00 | |
| krotscheck | sdague: Then why is it in global requirements? | 22:00 |
| krotscheck | Don’t they have a test-requirements? | 22:00 |
| sdague | because you can't put something in test-requirements that's not in global requirements | 22:01 |
| anteaya | sweston: thanks | 22:01 |
| *** nati_ueno has quit IRC | 22:01 | |
| anteaya | sweston: let's keep an eye on that so your script doesn't go wild again | 22:01 |
| *** homeless has quit IRC | 22:01 | |
| krotscheck | Oh, I see | 22:01 |
| *** nati_ueno has joined #openstack-infra | 22:01 | |
| sdague | also, interestingly, their tox call run_tests.sh | 22:01 |
| sweston | anteaya: it was a fairly useless script anyway, I deleted it | 22:01 |
| sdague | that might make mordred's head explode | 22:01 |
| anteaya | sweston: even better | 22:01 |
| anteaya | easy to keep track of then | 22:02 |
| anteaya | sweston: so I can open both .txt files http://logs.ci.vyatta.net/57/100357/1/check/dsvm-tempest-full/bf50077746b9445c8c79fbe27467f8ee/ | 22:02 |
| sweston | anteaya: hehe | 22:02 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/puppet-storyboard: Adding Rakefile https://review.openstack.org/100370 | 22:02 |
| anteaya | sweston: well done | 22:02 |
| *** homeless has joined #openstack-infra | 22:02 | |
| sweston | anteaya: :-) | 22:02 |
| anteaya | sweston: but the directory just contains the same txt files again | 22:02 |
| anteaya | so, why? | 22:02 |
| mtreinish | sdague: recursive venvs? | 22:02 |
| *** markmcclain1 has quit IRC | 22:02 | |
| sweston | anteaya: let me check | 22:03 |
| anteaya | kk | 22:03 |
| mtreinish | sdague: or is their run_tests kind of like pretty_tox? | 22:03 |
| sdague | mtreinish: I honestly stopped reading after that point | 22:03 |
| *** doug-fish has left #openstack-infra | 22:03 | |
| krotscheck | sdague: So, long story short, I want to have the package managers from centos, redhat, and ubuntu to +1 this, with explicit approvals from them that it’s a valid thing to use during tests. | 22:03 |
| krotscheck | sdague: And even then it makes me leery. | 22:04 |
| *** Ryan_Lane1 has joined #openstack-infra | 22:04 | |
| *** marcoemorais has quit IRC | 22:04 | |
| *** marcoemorais has joined #openstack-infra | 22:05 | |
| *** marcoemorais has quit IRC | 22:05 | |
| jogo | clarkb: yeah so it must be the cron job failing because cron job would run on nodes that are in th delete state | 22:05 |
| *** marcoemorais has joined #openstack-infra | 22:05 | |
| jogo | but no nodes in delete state are being deleted (se nodepool.nodepool:1819 | 22:06 |
| sweston | anteaya: it was a problem with the jenkins job builder file, I fixed it | 22:06 |
| jogo | jeblair: ^ | 22:06 |
| sweston | anteaya: shall I run the whole process again? | 22:06 |
| anteaya | sweston: yes please | 22:07 |
| sweston | anteaya: ok ... good thing this is super quick .. only half an hour, hehe | 22:07 |
| sweston | anteaya: what is the next step? | 22:07 |
| *** marcoemorais has quit IRC | 22:07 | |
| *** marcoemorais has joined #openstack-infra | 22:08 | |
| *** nikhil___ is now known as nikhil___|afk | 22:09 | |
| devananda | could anyone point me towards figuring out how to approve patches for stable/icehouse branch of ironic? | 22:09 |
| anteaya | sweston: well check the timestamps in the logs you have produced thus far | 22:11 |
| anteaya | are they utc timestamps? | 22:11 |
| sweston | anteaya: no | 22:13 |
| sweston | anteaya: I'll figure out how to fix that | 22:13 |
| *** matjazp has joined #openstack-infra | 22:13 | |
| *** weshay has quit IRC | 22:14 | |
| bodepd | nibalizer: I definitely don't have that. I'm still building around the current jenkins module. | 22:14 |
| anteaya | sweston: k, that's your next step | 22:15 |
| anteaya | sweston: I think it is a matter of changing the timezone of your server running your tests | 22:15 |
| sweston | anteaya: thank you | 22:16 |
| anteaya | sweston: unless that would create a problem for you | 22:16 |
| anteaya | devananda: I don't know how to do that | 22:16 |
| sweston | anteaya: shouldn't ... but the server is already in my local timezone, so I'm thinking it must be a Jenkins setting | 22:17 |
| anteaya | reed: what is this group about? https://review.openstack.org/#/admin/groups/126,members | 22:17 |
| anteaya | sweston: k, whatever works for you | 22:17 |
| * reed checking | 22:17 | |
| anteaya | but next thing is utc timestamps | 22:17 |
| sweston | anteaya: kk | 22:18 |
| reed | are auth sessions expiring faster on gerrit lately? | 22:18 |
| anteaya | reed: not that I have seen | 22:18 |
| reed | anteaya, that group is those who signed the US Government CLA | 22:18 |
| anteaya | if you have two tabs open and are signed into one and not the other | 22:18 |
| anteaya | they fight with each other | 22:18 |
| anteaya | reed: ah okay thanks | 22:18 |
| *** homeless has quit IRC | 22:19 | |
| morganfainberg | devananda, i think PTL and stable maint folks usually weigh in on any stable branch change | 22:20 |
| *** CaptTofu_ has quit IRC | 22:20 | |
| morganfainberg | devananda, and (at least in keystone) we use the same 2x+2 and then approval for it | 22:20 |
| *** _TheDodd_ has joined #openstack-infra | 22:20 | |
| *** CaptTofu_ has joined #openstack-infra | 22:20 | |
| *** ramashri has quit IRC | 22:20 | |
| devananda | morganfainberg: that's what i'd expect, but i actually don't have +2/+A powers for ironic's stable/icehouse branch | 22:21 |
| devananda | morganfainberg: OR i dont know how to do it | 22:21 |
| devananda | and am doing something wrong | 22:21 |
| *** ramashri has joined #openstack-infra | 22:21 | |
| morganfainberg | devananda, nah you should have that power, prob a fix needs to go into infra if you don't | 22:21 |
| morganfainberg | devananda, or there is a group you're not in that you should be | 22:22 |
| * morganfainberg goes and looks at how keystone works. | 22:22 | |
| devananda | morganfainberg: hmm... right. /me looks at nova | 22:22 |
| *** doddstack has quit IRC | 22:23 | |
| mtreinish | morganfainberg, devananda: it depends on the project. It's in the gerrit acls for the project. For example tempest stable branches the whole ore group has +2 on | 22:23 |
| morganfainberg | mtreinish, it's not in the keystone acl, so maybe dolph is in a special stable maintenance group? | 22:25 |
| morganfainberg | ah | 22:25 |
| mtreinish | morganfainberg: does it inherit from all-projects. Then it's https://review.openstack.org/#/admin/projects/All-Projects,access | 22:25 |
| morganfainberg | mtreinish, yeah dolph is in stable-maintenance | 22:25 |
| morganfainberg | devananda, ^ thats why dolph can do that stuff | 22:25 |
| morganfainberg | devananda, as PTL you might need to be in https://review.openstack.org/#/admin/groups/120,members | 22:26 |
| devananda | ah ha! | 22:26 |
| devananda | it wasn't jenkins config or gerrit project config | 22:26 |
| morganfainberg | or just bug those folks to do it | 22:27 |
| mtreinish | devananda: I'm on stable maint too if you want me to look at a stable thing... | 22:27 |
| devananda | adam_g: want to do stable maint for ironic? | 22:27 |
| devananda | mtreinish: ah, thx. this one: https://review.openstack.org/#/c/97475/ | 22:28 |
| adam_g | devananda, i can help with it, sure. | 22:28 |
| devananda | so secondary question - can we disable certain tempest runs on stable branch? | 22:28 |
| *** matjazp has quit IRC | 22:28 | |
| devananda | adam_g: i know we didn't enable voting in the virtual-ironic job during icehouse, but i don't know, at this point, what we'd need to backport to make that job work on it | 22:29 |
| mtreinish | devananda: not unless it's tied to a specific feature flag in the tempest config. | 22:30 |
| adam_g | devananda, IIRC it was passing when icehouse releasd but wasn't made voting till after | 22:30 |
| devananda | adam_g: ahh, never mind then. | 22:31 |
| devananda | let's see why it's failing | 22:31 |
| *** jgrimm_ has quit IRC | 22:34 | |
| *** jergerber has quit IRC | 22:35 | |
| cody-somerville | jogo: sdague: What if new hacking rules were warnings at first and then graduate to errors? or something. :) | 22:36 |
| *** flaper87 is now known as flaper87|afk | 22:37 | |
| *** UtahDave has quit IRC | 22:38 | |
| *** matjazp has joined #openstack-infra | 22:41 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Archive Ironic VM nodes console logs https://review.openstack.org/92216 | 22:42 |
| adam_g | devananda, ^ that would help us here. not sure whats up with that stable job failing. | 22:42 |
| sdague | cody-somerville: there are definitely lots of ways we could do it. :) | 22:42 |
| *** mestery has joined #openstack-infra | 22:42 | |
| cody-somerville | sdague: lintian might be a good model | 22:43 |
| *** ihrachyshka has quit IRC | 22:44 | |
| *** _TheDodd_ has quit IRC | 22:46 | |
| sdague | jogo: where's your change to get rid of large-ops on havana? | 22:47 |
| lifeless | harlowja: well that matrix I put in my comment | 22:48 |
| *** otherwiseguy has quit IRC | 22:48 | |
| harlowja | lifeless kk, forgot about that one | 22:48 |
| *** nati_ueno has quit IRC | 22:49 | |
| *** mestery has quit IRC | 22:55 | |
| *** amotoki_ has quit IRC | 22:55 | |
| mattoliverau | morning | 22:55 |
| *** thedodd has joined #openstack-infra | 22:56 | |
| *** marun has quit IRC | 22:56 | |
| anteaya | morning mattoliverau | 22:56 |
| *** bhuvan has joined #openstack-infra | 22:58 | |
| anteaya | if someone could get nodepool to delete some more suck nodes | 23:01 |
| anteaya | that could be useful | 23:01 |
| anteaya | s/suck/stuck | 23:01 |
| jhesketh | Morning | 23:01 |
| anteaya | morning jhesketh | 23:01 |
| *** lttrl has joined #openstack-infra | 23:03 | |
| *** Sukhdev has quit IRC | 23:05 | |
| *** matjazp has quit IRC | 23:06 | |
| *** zzelle_ has quit IRC | 23:07 | |
| *** rkukura_ has joined #openstack-infra | 23:08 | |
| *** rkukura has quit IRC | 23:08 | |
| *** rkukura_ is now known as rkukura | 23:08 | |
| *** nati_ueno has joined #openstack-infra | 23:10 | |
| *** mmaglana has joined #openstack-infra | 23:12 | |
| *** CaptTofu_ has quit IRC | 23:13 | |
| harlowja | lifeless comment added for matrix | 23:13 |
| *** CaptTofu_ has joined #openstack-infra | 23:13 | |
| lifeless | harlowja: thanks! | 23:16 |
| harlowja | np | 23:16 |
| lyxus | is it the proper place to talk about devstack issue ? There was a commit that broke it (or so i think) | 23:17 |
| *** CaptTofu_ has quit IRC | 23:18 | |
| *** yamahata has quit IRC | 23:18 | |
| *** thedodd has quit IRC | 23:18 | |
| *** julim has quit IRC | 23:19 | |
| lyxus | On the truc, the commit (3723814bf27fb4d78c6c3ad80d77882f75ad07c4) introduced the file tools/outfilter.py that doesn't exist | 23:19 |
| jhesketh | clarkb, fungi: Are either of you guys able to have a look at a job that's been stuck in queue for nearly 100hrs? | 23:21 |
| *** jamielennox|away is now known as jamielennox | 23:21 | |
| sdague | lyxus: -qa is the right place for that | 23:21 |
| lyxus | sdague, thanks ! | 23:21 |
| *** lttrl has quit IRC | 23:24 | |
| *** lttrl has joined #openstack-infra | 23:25 | |
| clarkb | jhesketh sorry I am not super useful today. fighting a cold and generally blah | 23:25 |
| jhesketh | no worries, get better soon! | 23:25 |
| clarkb | jhesketh is it one of the swift uploads? | 23:25 |
| sdague | clarkb: if you are still kicking - https://review.openstack.org/#/c/99750/ | 23:25 |
| clarkb | those dont work right now | 23:25 |
| jhesketh | clarkb: yes | 23:25 |
| sdague | as our ability to land havana patches is basically pretty hit or miss until we disable largeops on them | 23:26 |
| clarkb | jhesketh we need to restart zuul to pivk up my bug fix | 23:26 |
| clarkb | you can push a new patchset to remove the job | 23:27 |
| jhesketh | clarkb: which bug fix? | 23:27 |
| clarkb | the one that gets the default swift url | 23:27 |
| *** blamar has joined #openstack-infra | 23:27 | |
| *** prad_ has quit IRC | 23:30 | |
| clarkb | sdague: done | 23:32 |
| sdague | clarkb: thanks | 23:33 |
| sdague | clarkb: also, is this out there for a reason still - https://review.openstack.org/#/q/status:open+project:openstack-infra/devstack-gate+branch:master+topic:do-not-merge,n,z ? | 23:36 |
| *** sarob_ has quit IRC | 23:36 | |
| *** sarob_ has joined #openstack-infra | 23:37 | |
| *** matjazp has joined #openstack-infra | 23:37 | |
| *** melwitt has quit IRC | 23:38 | |
| clarkb | it can be abandoned or WIP'd if that helps | 23:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't run large-ops test on stable/havana branches https://review.openstack.org/99750 | 23:39 |
| *** sarob_ has quit IRC | 23:41 | |
| *** dims has quit IRC | 23:41 | |
| mtreinish | sdague: do you want to start working through the v3 api patches today? | 23:41 |
| *** matjazp has quit IRC | 23:41 | |
| *** atiwari has quit IRC | 23:47 | |
| sdague | clarkb: it keeps showing up in some slice of my queries, just curious | 23:48 |
| sdague | mtreinish: actually, I'm about to walk away and get a drink. As it's properly quitting time :) | 23:48 |
| sdague | ping me in the morning though, tomorrow might work on that | 23:49 |
| mtreinish | sdague: ok | 23:49 |
| sdague | the zuul job queue is so close to having capacity again :) | 23:49 |
| *** hemna is now known as hemna_ | 23:50 | |
| mikal | Is there a known issue in the postgres unit tests for nova at the moment? | 23:52 |
| mikal | I'm seeing: | 23:52 |
| mikal | AttributeError: 'PostgreSQLOpportunisticFixture' object has no attribute '_details' | 23:52 |
| mikal | Which is new to me | 23:52 |
| mikal | http://logs.openstack.org/75/94775/6/check/gate-nova-python26/7a09354/console.html is an example | 23:53 |
| *** sarob_ has joined #openstack-infra | 23:55 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/requirements: Set version bounds for diskimage-builder https://review.openstack.org/100385 | 23:56 |
| sdague | looks like it has some hits - http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkF0dHJpYnV0ZUVycm9yOiAnUG9zdGdyZVNRTE9wcG9ydHVuaXN0aWNGaXh0dXJlJyBvYmplY3QgaGFzIG5vIGF0dHJpYnV0ZSAnX2RldGFpbHMnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDI5NjI5NTU0MTZ9 | 23:57 |
| sdague | mikal: there are definitely a bunch of races introduced by oslo.db | 23:57 |
| sdague | especially in the unit tests | 23:57 |
| mikal | Those three clustered ones are all me I think | 23:58 |
| *** sarob__ has joined #openstack-infra | 23:58 | |
| mikal | Oh, I lie | 23:58 |
| mikal | Not all me | 23:58 |
| sdague | mikal: well it might actually be your patch? | 23:58 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/devstack-gate: Make one copy of grenade log files https://review.openstack.org/100131 | 23:58 |
| sdague | or maybe your patch does a timing shift | 23:58 |
| mikal | sdague: unlikely... Its a libvirt logging patch | 23:58 |
| sdague | enough to overlap things | 23:58 |
| sdague | ok, drinking time. mikal welcome to the rodeo :) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!