| *** mrda has quit IRC | 00:01 | |
| branen_ | fungi: Hi, so I guess the requirements integration job is broken? Any idea when it might be fixed? I’ve committed a new cinder driver and it won’t even pass its unit test without https://review.openstack.org/65179. Thanks | 00:04 |
|---|---|---|
| *** openstackstatus has quit IRC | 00:04 | |
| fungi | branen_: i'm working on it in https://review.openstack.org/66364 | 00:05 |
| branen_ | fungi: cool, thanks!! | 00:05 |
| fungi | branen_: combination of changes which happened in pip 1.5 are incompatible with what we have (externally-hosted requirements, -f url entries for non-pypi-indexed requirements like oslo.messaging beta releases) | 00:06 |
| pelix | zaro: for 52080 is there a clear idea of what should be a default option and what should only appear under sections? placing 'ignore_cache' under DEFAULTS doesn't line up with the example config under etc/jenkins_jobs.ini-sample but it's not wrong either. | 00:06 |
| *** CaptTofu has quit IRC | 00:08 | |
| branen_ | fungi: sounds like you got your hand full, thanks for working on this stuff. | 00:08 |
| *** CaptTofu has joined #openstack-infra | 00:08 | |
| *** CaptTofu has quit IRC | 00:08 | |
| *** mrodden has joined #openstack-infra | 00:13 | |
| fungi | branen_: it's my pleasure. thanks for working on openstack too! | 00:16 |
| *** kraman1 has joined #openstack-infra | 00:18 | |
| *** jhesketh__ has quit IRC | 00:19 | |
| *** jhesketh__ has joined #openstack-infra | 00:20 | |
| pelix | zaro: Just would be useful before adding the doc you requested in 48783 | 00:20 |
| *** oubiwann has quit IRC | 00:21 | |
| pelix | as once 52080 is added I need to add something for the include search path option I added and also decide if it should change from using a separate '[include]' section for options relating to parsing yaml (might at least change it to '[yaml]' when adding documentation) to just using a default option and checking the jenkins section. | 00:22 |
| zaro | pelix: you introduced the [include] section, but now that i think about it i'm not sure there's a need for it. would it work if path is just under [jenkins]? | 00:24 |
| *** CaptTofu has joined #openstack-infra | 00:26 | |
| fungi | mordred: as i suspected... "[pip] no such option: --process-dependency-links" and i've confirmed that pip install in 1.5 accepts that option, so proof we must be starting from older pip... http://logs.openstack.org/64/66364/5/check/check-requirements-integration-dsvm/40b8294/console.html#_2014-01-14_23_25_59_300 | 00:27 |
| wenlock | hey, can anyone unabandon https://review.openstack.org/#/c/51425/ | 00:30 |
| wenlock | i'd like to give it another crack | 00:30 |
| fungi | wenlock: done, thanks!!! | 00:30 |
| wenlock | fungi, awesome, thanks! | 00:30 |
| *** gokrokve has quit IRC | 00:33 | |
| *** gokrokve has joined #openstack-infra | 00:33 | |
| *** yamahata has quit IRC | 00:34 | |
| *** dcramer_ has joined #openstack-infra | 00:36 | |
| *** wenlock has quit IRC | 00:37 | |
| *** gokrokve has quit IRC | 00:37 | |
| *** dkliban has joined #openstack-infra | 00:38 | |
| *** pcrews_ has joined #openstack-infra | 00:39 | |
| *** jaypipes has quit IRC | 00:42 | |
| *** ^demon has joined #openstack-infra | 00:43 | |
| *** ^d has quit IRC | 00:46 | |
| *** ^demon has quit IRC | 00:47 | |
| pelix | zaro: it would, it just wasn't clear to me what should be placed into separate sections? should options affecting modules or specific plugins go under separate sections? or should their only ever be one section called 'jenkins'? | 00:51 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/pypi-mirror: Whitelist external packages https://review.openstack.org/66364 | 00:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/pypi-mirror: Whitelist external packages https://review.openstack.org/66364 | 00:56 |
| *** thuc has joined #openstack-infra | 01:01 | |
| *** thuc has quit IRC | 01:02 | |
| *** thuc has joined #openstack-infra | 01:02 | |
| *** thuc_ has joined #openstack-infra | 01:03 | |
| *** thuc has quit IRC | 01:03 | |
| *** hogepodge has quit IRC | 01:08 | |
| *** beagles has quit IRC | 01:12 | |
| *** hogepodge has joined #openstack-infra | 01:13 | |
| *** gokrokve has joined #openstack-infra | 01:20 | |
| *** flaper87 is now known as flaper87|afk | 01:24 | |
| *** zhiwei has joined #openstack-infra | 01:27 | |
| *** yaguang has joined #openstack-infra | 01:29 | |
| *** AaronGr is now known as AaronGr_afk | 01:31 | |
| *** mozawa has joined #openstack-infra | 01:33 | |
| *** llu has left #openstack-infra | 01:33 | |
| *** CaptTofu has quit IRC | 01:36 | |
| *** CaptTofu has joined #openstack-infra | 01:37 | |
| *** CaptTofu has quit IRC | 01:37 | |
| *** nosnos has joined #openstack-infra | 01:38 | |
| *** yamahata has joined #openstack-infra | 01:41 | |
| lifeless | fungi: ok so | 01:43 |
| *** rakhmerov has quit IRC | 01:43 | |
| lifeless | fungi: can we check nodepool on this new cloud ? | 01:43 |
| lifeless | fungi: I've spun up an instance as openstack-nodepool successfully | 01:43 |
| jhesketh__ | any config cores around able to get this through please? https://review.openstack.org/#/c/64788/ | 01:47 |
| *** praneshp has quit IRC | 01:48 | |
| *** ryanpetrello has quit IRC | 01:50 | |
| *** yamahata has quit IRC | 01:52 | |
| fungi | lifeless: yes, i'll give it a go | 01:52 |
| *** prad_ has joined #openstack-infra | 01:52 | |
| fungi | jhesketh__: i'll be doing the manual parts of that rename on saturday since it requires a gerrit outage. i'll be merging it at the time i perform the maintenance | 01:53 |
| jhesketh__ | thanks fungi, good to know :-) | 01:54 |
| *** thuc_ has quit IRC | 01:55 | |
| *** thuc has joined #openstack-infra | 01:56 | |
| fungi | jhesketh__: for future reference, the process we follow to do project renames is documented at http://ci.openstack.org/gerrit.html#renaming-a-project | 01:56 |
| jhesketh__ | ah cool, that's quite helpful thanks :-) | 01:57 |
| *** harlowja is now known as harlowja_away | 01:58 | |
| *** weshay has quit IRC | 01:58 | |
| *** yamahata has joined #openstack-infra | 01:58 | |
| *** thuc has quit IRC | 02:01 | |
| *** dkliban has quit IRC | 02:01 | |
| *** dkliban has joined #openstack-infra | 02:02 | |
| *** weshay has joined #openstack-infra | 02:02 | |
| *** greghaynes has quit IRC | 02:03 | |
| jog0 | mikal: ping | 02:04 |
| lifeless | fungi: thanks | 02:05 |
| lifeless | fungi: I'm finalising some details w.r.t. access to the broker | 02:05 |
| mikal | jog0: pong | 02:06 |
| jog0 | I can get a new rev of 3rd party reqs tomorrow | 02:06 |
| jog0 | mikal: unless you want to do it | 02:07 |
| mikal | Cool, that would rock my world | 02:07 |
| mikal | anteaya and I had decided it was rude to do it for you | 02:07 |
| jog0 | mikal: by all means have a whack at it but thanks for asking first | 02:08 |
| mikal | Cool. I will see if I get the time today. | 02:09 |
| *** reed has quit IRC | 02:09 | |
| jog0 | cool, if not just let me know and I will do it in Montreal tomorrow | 02:10 |
| mikal | Ok | 02:10 |
| *** mozawa has quit IRC | 02:14 | |
| fungi | lifeless: NotFound: No Image matching {'name': 'Ubuntu Precise 12.04 LTS Server 64-bit'}. (HTTP 404) | 02:14 |
| *** mozawa has joined #openstack-infra | 02:15 | |
| lifeless | fungi: yeah, you should upload one | 02:15 |
| lifeless | fungi: glance, it's a wonderful thing | 02:15 |
| fungi | lifeless: i'm guessing the base-image parameters are wrong in https://review.openstack.org/#/c/66491/1/modules/openstack_project/templates/nodepool/nodepool.yaml.erb | 02:15 |
| fungi | lifeless: hrm... i wonder who did that for the poc cloud | 02:15 |
| lifeless | pleia2 did | 02:16 |
| fungi | anyway, i guess that means the credentials are valus | 02:16 |
| fungi | valid | 02:16 |
| lifeless | yah | 02:16 |
| *** kraman1 has quit IRC | 02:17 | |
| *** gokrokve has quit IRC | 02:17 | |
| *** gokrokve has joined #openstack-infra | 02:17 | |
| lifeless | hmm | 02:20 |
| lifeless | where to put the broker | 02:20 |
| *** gokrokve has quit IRC | 02:21 | |
| *** gokrokve has joined #openstack-infra | 02:26 | |
| *** harlowja_away is now known as harlowja | 02:27 | |
| *** shardy has quit IRC | 02:27 | |
| *** melwitt has quit IRC | 02:28 | |
| *** gokrokve has quit IRC | 02:29 | |
| *** herndon has joined #openstack-infra | 02:29 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 02:31 |
| *** herndon has quit IRC | 02:32 | |
| pleia2 | fungi, lifeless - I just added an ubuntu cloud image into the poc cloud glance for it to use, not sure if anyone did this for the new cloud yet | 02:33 |
| *** gothicmindfood has quit IRC | 02:35 | |
| fungi | pleia2: yeah, seems not. i've added a note on https://review.openstack.org/66491 to that effect. thanks! | 02:35 |
| lifeless | fungi: pleia2: someone from infra should do that | 02:37 |
| fungi | lifeless: agreed. first we need to find/make an appropriate one. all our other providers lack glance to we're too accustomed to not having the option | 02:40 |
| *** nati_ueno has quit IRC | 02:40 | |
| *** nati_ueno has joined #openstack-infra | 02:40 | |
| lifeless | fungi: http://cloud-images.ubuntu.com/precise/current/ | 02:40 |
| fungi | and hunting down disk images and acquainting myself with glanceclient isn't in the cards for this time of night | 02:41 |
| fungi | ahh, well that's half the battle there, thanks! | 02:41 |
| lifeless | the .img files are qcow2 | 02:41 |
| *** changbl has joined #openstack-infra | 02:44 | |
| *** nati_ueno has quit IRC | 02:45 | |
| *** gokrokve has joined #openstack-infra | 02:47 | |
| *** coolsvap has quit IRC | 02:48 | |
| *** yamahata has quit IRC | 02:50 | |
| *** yamahata has joined #openstack-infra | 02:51 | |
| *** krotscheck has quit IRC | 02:55 | |
| *** jerryz has quit IRC | 02:58 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Process dependency links in reqs integration job https://review.openstack.org/66749 | 03:00 |
| fungi | mordred: ^ that's to allow us to work around needing to simulpatch both pypi-mirror and pbr | 03:01 |
| fungi | since it's the pip install calls in the integration.sh in pbr which seem to need it | 03:01 |
| *** weshay has quit IRC | 03:03 | |
| *** gokrokve has quit IRC | 03:05 | |
| *** weshay has joined #openstack-infra | 03:05 | |
| *** gokrokve has joined #openstack-infra | 03:05 | |
| *** HenryG has quit IRC | 03:09 | |
| *** gokrokve has quit IRC | 03:10 | |
| *** praneshp has joined #openstack-infra | 03:11 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/pypi-mirror: Whitelist external packages https://review.openstack.org/66364 | 03:13 |
| *** vkozhukalov has joined #openstack-infra | 03:14 | |
| *** enikanorov has joined #openstack-infra | 03:17 | |
| *** gokrokve has joined #openstack-infra | 03:18 | |
| *** enikanorov_ has quit IRC | 03:20 | |
| *** weshay has quit IRC | 03:28 | |
| *** pballand has joined #openstack-infra | 03:36 | |
| *** CaptTofu has joined #openstack-infra | 03:40 | |
| *** coolsvap has joined #openstack-infra | 03:42 | |
| *** coolsvap has quit IRC | 03:49 | |
| *** nati_ueno has joined #openstack-infra | 03:57 | |
| mordred | fungi: looking | 03:57 |
| mordred | fungi: looks good to me - you want to wait for a second on that one? | 03:58 |
| *** harlowja is now known as harlowja_away | 03:59 | |
| *** greghaynes has joined #openstack-infra | 04:00 | |
| *** pballand has quit IRC | 04:02 | |
| *** coolsvap has joined #openstack-infra | 04:02 | |
| *** coolsvap has quit IRC | 04:03 | |
| *** coolsvap has joined #openstack-infra | 04:03 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Added .gitignore for IntelliJ specific configuration files https://review.openstack.org/62956 | 04:05 |
| *** dstanek has joined #openstack-infra | 04:10 | |
| *** rwsu has quit IRC | 04:12 | |
| *** prad_ has quit IRC | 04:12 | |
| *** prad_ has joined #openstack-infra | 04:13 | |
| *** pcrews_ has quit IRC | 04:13 | |
| *** dcramer_ has quit IRC | 04:15 | |
| *** dstanek is now known as dstanekafk | 04:15 | |
| openstackgerrit | Matthew Oliver proposed a change to openstack-infra/config: Adding apache redirect for the cacti url. https://review.openstack.org/66181 | 04:22 |
| *** pballand has joined #openstack-infra | 04:28 | |
| *** prad_ has quit IRC | 04:34 | |
| *** prad_ has joined #openstack-infra | 04:34 | |
| *** Ryan_Lane has joined #openstack-infra | 04:34 | |
| *** pballand has quit IRC | 04:36 | |
| *** CaptTofu has quit IRC | 04:36 | |
| *** CaptTofu has joined #openstack-infra | 04:37 | |
| *** CaptTofu has quit IRC | 04:37 | |
| *** chandankumar has joined #openstack-infra | 04:40 | |
| *** reed has joined #openstack-infra | 04:49 | |
| *** coolsvap has quit IRC | 04:50 | |
| *** prad_ has quit IRC | 04:51 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:56 | |
| *** prad_ has joined #openstack-infra | 04:57 | |
| *** sarob has joined #openstack-infra | 04:58 | |
| *** prad_ has quit IRC | 05:05 | |
| *** coolsvap has joined #openstack-infra | 05:08 | |
| *** hartsocks has quit IRC | 05:11 | |
| *** nicedice has quit IRC | 05:17 | |
| *** SergeyLukjanov has quit IRC | 05:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:44 | |
| *** SergeyLukjanov has quit IRC | 05:49 | |
| *** sarob has quit IRC | 05:50 | |
| *** nati_ueno has quit IRC | 05:50 | |
| *** nati_ueno has joined #openstack-infra | 05:51 | |
| *** gema_ has quit IRC | 05:55 | |
| *** nati_ueno has quit IRC | 05:56 | |
| *** gema has joined #openstack-infra | 05:56 | |
| *** denis_makogon has joined #openstack-infra | 06:01 | |
| *** coolsvap has quit IRC | 06:19 | |
| *** xBsd has joined #openstack-infra | 06:21 | |
| *** gokrokve has quit IRC | 06:27 | |
| *** senk has quit IRC | 06:30 | |
| enikanorov | EmilienM: i'm not a neutron core | 06:34 |
| *** NikitaKonovalov has joined #openstack-infra | 06:39 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add turbo-hipster to rtfd https://review.openstack.org/66769 | 06:44 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Update the rtfd info now you don't need an id https://review.openstack.org/66770 | 06:44 |
| *** pelix has left #openstack-infra | 06:45 | |
| *** xBsd has quit IRC | 06:50 | |
| *** sarob has joined #openstack-infra | 06:51 | |
| *** NikitaKonovalov has quit IRC | 06:53 | |
| *** kraman1 has joined #openstack-infra | 06:54 | |
| *** sarob has quit IRC | 06:55 | |
| *** vkozhukalov has quit IRC | 06:56 | |
| *** NikitaKonovalov has joined #openstack-infra | 06:57 | |
| *** flaper87|afk is now known as flaper87 | 06:57 | |
| *** boris-42 has quit IRC | 07:02 | |
| *** reed has quit IRC | 07:07 | |
| *** coolsvap has joined #openstack-infra | 07:13 | |
| *** scairbus has joined #openstack-infra | 07:15 | |
| *** yolanda has joined #openstack-infra | 07:19 | |
| *** NikitaKonovalov has quit IRC | 07:27 | |
| *** ok_delta has joined #openstack-infra | 07:28 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:32 | |
| *** dkliban has quit IRC | 07:37 | |
| *** shardy has joined #openstack-infra | 07:41 | |
| *** denis_makogon has quit IRC | 07:52 | |
| *** sarob has joined #openstack-infra | 07:52 | |
| *** pblaho has joined #openstack-infra | 07:55 | |
| *** sarob has quit IRC | 07:56 | |
| *** boris-42 has joined #openstack-infra | 08:04 | |
| *** NikitaKonovalov has quit IRC | 08:05 | |
| *** enikanorov has quit IRC | 08:06 | |
| *** NikitaKonovalov has joined #openstack-infra | 08:10 | |
| *** afazekas has joined #openstack-infra | 08:21 | |
| pasquier-s | hi, I've got an approved change that is stuck waiting for gate jobs to start since yesterday => https://review.openstack.org/#/c/65269/ | 08:22 |
| pasquier-s | oops sorry it has finally started | 08:23 |
| *** vkozhukalov has joined #openstack-infra | 08:28 | |
| *** praneshp has quit IRC | 08:31 | |
| *** enikanorov has joined #openstack-infra | 08:35 | |
| *** zhiwei has quit IRC | 08:40 | |
| *** chandankumar has quit IRC | 08:41 | |
| *** NikitaKonovalov has quit IRC | 08:42 | |
| *** hashar has joined #openstack-infra | 08:48 | |
| *** saschpe has quit IRC | 08:51 | |
| *** sarob has joined #openstack-infra | 08:51 | |
| *** saschpe has joined #openstack-infra | 08:51 | |
| *** sarob has quit IRC | 08:55 | |
| *** mancdaz_away is now known as mancdaz | 08:58 | |
| *** scairbus has left #openstack-infra | 09:03 | |
| *** coolsvap has quit IRC | 09:04 | |
| *** chandankumar has joined #openstack-infra | 09:06 | |
| *** zhiwei has joined #openstack-infra | 09:06 | |
| *** dizquierdo has joined #openstack-infra | 09:06 | |
| *** DinaBelova has joined #openstack-infra | 09:07 | |
| *** jcoufal has joined #openstack-infra | 09:08 | |
| *** coolsvap has joined #openstack-infra | 09:09 | |
| *** coolsvap has quit IRC | 09:10 | |
| *** derekh has joined #openstack-infra | 09:12 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:13 | |
| *** markmc has joined #openstack-infra | 09:15 | |
| *** yassine has joined #openstack-infra | 09:15 | |
| *** dizquierdo has quit IRC | 09:16 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Sort tracked projects list https://review.openstack.org/63770 | 09:21 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Add rally to tracked projects list https://review.openstack.org/63741 | 09:21 |
| *** dpyzhov has joined #openstack-infra | 09:23 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Fix up `zuul --version` https://review.openstack.org/66629 | 09:32 |
| *** zhiwei has quit IRC | 09:32 | |
| hashar | :-) | 09:36 |
| *** hashar has quit IRC | 09:41 | |
| *** hashar_ has joined #openstack-infra | 09:41 | |
| *** hashar_ is now known as hashar | 09:44 | |
| *** _ruhe is now known as ruhe | 09:46 | |
| *** luisg has quit IRC | 09:49 | |
| *** luis_ has joined #openstack-infra | 09:49 | |
| *** sarob has joined #openstack-infra | 09:51 | |
| *** sarob has quit IRC | 09:56 | |
| *** mozawa has quit IRC | 10:04 | |
| *** chandankumar has quit IRC | 10:09 | |
| hashar | SergeyLukjanov: hello :-) | 10:12 |
| hashar | SergeyLukjanov: two of your Zuul changes should probably be squashed together, the ones related to exposing Zuul version in status pages | 10:12 |
| hashar | ie https://review.openstack.org/#/c/66043/ and https://review.openstack.org/#/c/66052/ | 10:12 |
| *** dpyzhov has quit IRC | 10:14 | |
| *** dizquierdo has joined #openstack-infra | 10:16 | |
| hashar | SergeyLukjanov: same for last reconfigured, https://review.openstack.org/#/c/66053/ update the app.js, it should be integrated in the change that expose the value ( https://review.openstack.org/#/c/63849/ ) | 10:16 |
| hashar | commented on all of them | 10:16 |
| SergeyLukjanov | hashar, hi | 10:17 |
| *** NikitaKonovalov has joined #openstack-infra | 10:18 | |
| SergeyLukjanov | hashar, the reason was I don't really know guys would like to change the zuul status page | 10:18 |
| hashar | SergeyLukjanov: well openstack does not use those samples iirc | 10:19 |
| SergeyLukjanov | hashar, I don't realy think that it's good idea to squash different changes | 10:19 |
| hashar | SergeyLukjanov: their Zuul status page HTML and JS is in their puppet repository | 10:19 |
| SergeyLukjanov | hashar, having them granular will be useful for reviewing and merging if ok | 10:19 |
| SergeyLukjanov | hashar, I've already added changes to them too | 10:20 |
| SergeyLukjanov | https://review.openstack.org/#/c/63850/ https://review.openstack.org/#/c/64211/ | 10:20 |
| *** matrohon has joined #openstack-infra | 10:20 | |
| *** chandankumar has joined #openstack-infra | 10:25 | |
| hashar | SergeyLukjanov: i find it a bit hard to review all those tiny changes :-) Might just be me though hehe | 10:27 |
| SergeyLukjanov | hashar, the one more reason is that version has been already merged and, so, version-related changes could be landed before | 10:28 |
| SergeyLukjanov | hashar, heh, agreed, it's often related to personal preferences | 10:29 |
| *** jhesketh__ has quit IRC | 10:30 | |
| *** pblaho has quit IRC | 10:32 | |
| *** NikitaKonovalov has quit IRC | 10:35 | |
| *** ruhe is now known as ruhe_away | 10:37 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:40 | |
| *** jhesketh_ has quit IRC | 10:42 | |
| *** zhiwei has joined #openstack-infra | 10:43 | |
| matrohon | Hi | 10:45 |
| *** max_lobur_afk is now known as max_lobur | 10:45 | |
| matrohon | does anyone knows if multi-node/multi-host environnement is targeted for continuous integration? | 10:46 |
| *** ruhe_away is now known as _ruhe | 10:46 | |
| *** sarob has joined #openstack-infra | 10:51 | |
| max_lobur | Hi Everyone. Please, could someone help me to review https://review.openstack.org/#/c/66349/ (Add futures library to global requirements) | 10:52 |
| *** sarob has quit IRC | 10:56 | |
| *** mfink has quit IRC | 10:59 | |
| *** rfolco has joined #openstack-infra | 11:02 | |
| *** lcestari has joined #openstack-infra | 11:07 | |
| *** dpyzhov has joined #openstack-infra | 11:08 | |
| *** vogxn has joined #openstack-infra | 11:10 | |
| *** hashar is now known as hasharAway | 11:11 | |
| *** zhiwei has left #openstack-infra | 11:12 | |
| *** jcoufal has quit IRC | 11:15 | |
| *** katyafervent has quit IRC | 11:17 | |
| *** dpyzhov_ has joined #openstack-infra | 11:17 | |
| *** dpyzhov has quit IRC | 11:17 | |
| *** dpyzhov_ is now known as dpyzhov | 11:17 | |
| *** katyafervent has joined #openstack-infra | 11:17 | |
| *** boris-42_ has joined #openstack-infra | 11:18 | |
| *** branen_ has quit IRC | 11:18 | |
| *** boris-42 has quit IRC | 11:19 | |
| *** vkozhukalov has quit IRC | 11:19 | |
| *** boris-42_ is now known as boris-42 | 11:19 | |
| *** vkozhukalov has joined #openstack-infra | 11:19 | |
| *** ArxCruz has joined #openstack-infra | 11:21 | |
| *** dpyzhov has quit IRC | 11:38 | |
| *** jishaom has joined #openstack-infra | 11:42 | |
| *** ok_delta has quit IRC | 11:46 | |
| *** dpyzhov has joined #openstack-infra | 11:50 | |
| *** sarob has joined #openstack-infra | 11:51 | |
| *** sarob_ has joined #openstack-infra | 11:53 | |
| *** sarob has quit IRC | 11:56 | |
| *** sarob_ has quit IRC | 11:58 | |
| *** mozawa has joined #openstack-infra | 11:58 | |
| *** yaguang has quit IRC | 12:00 | |
| *** jcoufal has joined #openstack-infra | 12:03 | |
| *** yamahata has quit IRC | 12:06 | |
| *** jamielennox is now known as jamielennox|away | 12:08 | |
| *** hasharAway is now known as hashar | 12:14 | |
| *** max_lobur has quit IRC | 12:14 | |
| *** yolanda has quit IRC | 12:14 | |
| *** yolanda has joined #openstack-infra | 12:15 | |
| *** max_lobur has joined #openstack-infra | 12:18 | |
| *** pblaho has joined #openstack-infra | 12:30 | |
| *** smarcet has joined #openstack-infra | 12:32 | |
| *** esker has joined #openstack-infra | 12:34 | |
| *** esker has quit IRC | 12:34 | |
| *** che-arne_ has joined #openstack-infra | 12:41 | |
| *** che-arne_ has quit IRC | 12:43 | |
| *** che-arne has joined #openstack-infra | 12:43 | |
| *** jishaom has quit IRC | 12:43 | |
| *** NikitaKonovalov has quit IRC | 12:44 | |
| openstackgerrit | Vladimir Kozhukalov proposed a change to openstack-infra/devstack-gate: Tiny improvement in paths https://review.openstack.org/63902 | 12:46 |
| *** julim has joined #openstack-infra | 12:49 | |
| *** sarob has joined #openstack-infra | 12:51 | |
| *** nati_ueno has joined #openstack-infra | 12:52 | |
| *** prad has joined #openstack-infra | 12:52 | |
| *** sarob has quit IRC | 12:56 | |
| *** nati_ueno has quit IRC | 12:57 | |
| *** nosnos has quit IRC | 12:59 | |
| *** heyongli has joined #openstack-infra | 12:59 | |
| *** afrittoli has joined #openstack-infra | 13:03 | |
| *** afrittoli has quit IRC | 13:04 | |
| *** andreaf has joined #openstack-infra | 13:04 | |
| *** weshay has joined #openstack-infra | 13:10 | |
| *** pasquier-s has quit IRC | 13:11 | |
| *** pasquier-s has joined #openstack-infra | 13:11 | |
| *** sandywalsh has quit IRC | 13:19 | |
| openstackgerrit | A change was merged to openstack-infra/config: Process dependency links in reqs integration job https://review.openstack.org/66749 | 13:20 |
| *** _ruhe is now known as ruhe | 13:21 | |
| *** dprince has joined #openstack-infra | 13:25 | |
| *** eharney has joined #openstack-infra | 13:32 | |
| *** sandywalsh has joined #openstack-infra | 13:33 | |
| *** boris-42_ has joined #openstack-infra | 13:38 | |
| *** enikanorov_ has joined #openstack-infra | 13:38 | |
| *** mfink has joined #openstack-infra | 13:38 | |
| *** boris-42 has quit IRC | 13:39 | |
| *** boris-42_ is now known as boris-42 | 13:40 | |
| *** salv-orlando has joined #openstack-infra | 13:40 | |
| *** enikanorov has quit IRC | 13:41 | |
| *** salv-orlando has quit IRC | 13:41 | |
| *** mfink has quit IRC | 13:43 | |
| *** mfink has joined #openstack-infra | 13:43 | |
| *** markmcclain has joined #openstack-infra | 13:44 | |
| *** yamahata has joined #openstack-infra | 13:46 | |
| *** nati_ueno has joined #openstack-infra | 13:46 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add tests for Alembic migrations https://review.openstack.org/66414 | 13:49 |
| *** ok_delta has joined #openstack-infra | 13:49 | |
| *** emagana has joined #openstack-infra | 13:50 | |
| *** thomasem has joined #openstack-infra | 13:51 | |
| *** nati_ueno has quit IRC | 13:51 | |
| *** sarob has joined #openstack-infra | 13:51 | |
| *** nati_ueno has joined #openstack-infra | 13:52 | |
| *** sarob has quit IRC | 13:56 | |
| *** thuc has joined #openstack-infra | 13:57 | |
| *** rossella_s has joined #openstack-infra | 13:58 | |
| *** rcleere has joined #openstack-infra | 14:01 | |
| *** dkranz has joined #openstack-infra | 14:02 | |
| *** ruhe is now known as ruhe_away | 14:02 | |
| *** yamahata has quit IRC | 14:05 | |
| *** tma996 has joined #openstack-infra | 14:06 | |
| *** dkliban has joined #openstack-infra | 14:08 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 14:08 |
| *** heyongli has quit IRC | 14:09 | |
| *** ruhe_away is now known as ruhe | 14:10 | |
| *** dims_ has quit IRC | 14:10 | |
| *** dims_ has joined #openstack-infra | 14:12 | |
| sdague | fungi: so how are things this morning. Nodepool seems to be deleting a lot | 14:13 |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 14:14 |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 14:15 |
| *** prad has quit IRC | 14:16 | |
| *** bknudson has joined #openstack-infra | 14:17 | |
| sdague | fungi: can we get a bounce of this to top of queue - 66656 when you get a chance? | 14:18 |
| *** rcleere has quit IRC | 14:18 | |
| *** dkranz has quit IRC | 14:23 | |
| *** jasondotstar has joined #openstack-infra | 14:26 | |
| *** thuc has quit IRC | 14:29 | |
| *** thuc has joined #openstack-infra | 14:30 | |
| fungi | sdague: yeah, the rackspace nodes we added account for most of the deletes. rackspace seems to like to timeout on delete calls far more often than hpcloud | 14:33 |
| fungi | i may try to make a manual cleanup pass just to help it out | 14:34 |
| *** blamar has quit IRC | 14:34 | |
| *** dstanek has joined #openstack-infra | 14:34 | |
| *** thuc has quit IRC | 14:34 | |
| *** SergeyLukjanov_ has joined #openstack-infra | 14:34 | |
| fungi | sdague: also, did you ever get a chance to look at the q-fwaas and q-metering services being enabled when neutron=1 in d-g? EmilienM wanted them on in time for the meetup, but i didn't know whether there were any concerns from the qa side | 14:36 |
| * fungi is just paranoid about destabilizing the gate any more than it already is right now | 14:36 | |
| *** dkranz has joined #openstack-infra | 14:38 | |
| fungi | sdague: is 66656 fixing a cause of intermittent test failures, or just making them easier to debug? it's unclear to me from the commit message and bug description | 14:39 |
| *** SergeyLukjanov has quit IRC | 14:39 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 14:39 | |
| sdague | fungi: it's required for testing further grenade patches in neutron | 14:43 |
| *** burt has joined #openstack-infra | 14:43 | |
| sdague | given that we're sprinting here this week, we won't be able to do check runs | 14:43 |
| *** ruhe is now known as ruhe_away | 14:44 | |
| *** SergeyLukjanov has quit IRC | 14:44 | |
| *** ruhe_away is now known as ruhe | 14:44 | |
| fungi | k | 14:45 |
| *** ruhe is now known as ruhe_away | 14:45 | |
| fungi | i'll try to catch it at the next gate head reset | 14:45 |
| *** SergeyLukjanov_ has joined #openstack-infra | 14:45 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 14:45 | |
| *** nati_uen_ has joined #openstack-infra | 14:46 | |
| *** ruhe_away is now known as ruhe | 14:46 | |
| *** b3nt_pin has joined #openstack-infra | 14:47 | |
| *** b3nt_pin is now known as beagles | 14:48 | |
| *** nati_ueno has quit IRC | 14:49 | |
| sdague | fungi: thanks | 14:50 |
| sdague | yeh, the issue is that in order to not be blocked at the sprint we're going to need to ability to hop some patches up, because we need d-g or devstack patches needed by tempest or grenade | 14:50 |
| fungi | sdague: is that the case for the q-fwaas and q-metering enabling patches as well? | 14:51 |
| *** sarob has joined #openstack-infra | 14:51 | |
| sdague | fungi: no, those can wait | 14:52 |
| sdague | as far as I know | 14:52 |
| fungi | okay, it was implied those were needed for the beginning of the neutron spring | 14:52 |
| *** nati_uen_ has quit IRC | 14:53 | |
| *** nati_ueno has joined #openstack-infra | 14:53 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 14:53 | |
| *** boris-42_ has joined #openstack-infra | 14:54 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 14:54 | |
| *** prad has joined #openstack-infra | 14:54 | |
| sdague | fungi: https://etherpad.openstack.org/p/montreal-code-sprint | 14:54 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Temporarily remove neutron full job from experimental queue https://review.openstack.org/66863 | 14:55 |
| sdague | I'm going to put a bunch of patches at the top of queue | 14:55 |
| *** sarob has quit IRC | 14:55 | |
| *** boris-42_ has quit IRC | 14:56 | |
| *** senk has joined #openstack-infra | 14:57 | |
| *** boris-42 has quit IRC | 14:57 | |
| sdague | fungi: 65774 also needs promotion | 14:58 |
| sdague | it's already in the gate | 14:58 |
| *** ryanpetrello has joined #openstack-infra | 14:58 | |
| fungi | that's fine. it would have to be. zuul promote only works if the change has already been +2'd and approved | 14:59 |
| sdague | yep | 14:59 |
| sdague | well aware, just heads upping :) | 14:59 |
| fungi | thanks | 15:00 |
| fungi | might not be any benefit to promoting 65774. it could go in within 25 minutes anyway | 15:01 |
| *** coolsvap has joined #openstack-infra | 15:01 | |
| fungi | there are only 3 changes ahead of it | 15:01 |
| fungi | all currently passing | 15:01 |
| sdague | fungi: ok, that's fine | 15:02 |
| *** salv-orlando has joined #openstack-infra | 15:04 | |
| *** gokrokve has joined #openstack-infra | 15:07 | |
| *** SergeyLukjanov has quit IRC | 15:09 | |
| *** SergeyLukjanov_ has joined #openstack-infra | 15:10 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 15:10 | |
| sdague | oof, the fact that the queue is 24hrs deep is no fun | 15:10 |
| *** herndon has joined #openstack-infra | 15:10 | |
| *** yamahata has joined #openstack-infra | 15:11 | |
| *** salv-orlando has quit IRC | 15:11 | |
| *** pcrews_ has joined #openstack-infra | 15:11 | |
| *** salv-orlando has joined #openstack-infra | 15:11 | |
| *** afazekas has quit IRC | 15:11 | |
| openstackgerrit | Dmitry Kulishenko proposed a change to openstack/requirements: glance requires pyOpenSSL>=0.11 https://review.openstack.org/66867 | 15:12 |
| fungi | yep. it's also not been helping that a ton of changes got approved with stale samples and are repeatedly failing their unit tests as they bubble up through the gate (though those have mostly cleared out finally) | 15:12 |
| *** dcramer_ has joined #openstack-infra | 15:13 | |
| *** marun has joined #openstack-infra | 15:18 | |
| *** SergeyLukjanov has quit IRC | 15:19 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:20 | |
| *** dpyzhov has quit IRC | 15:20 | |
| *** dstanek has quit IRC | 15:22 | |
| *** SergeyLukjanov has quit IRC | 15:22 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:23 | |
| *** ruhe is now known as ruhe_away | 15:25 | |
| *** blamar has joined #openstack-infra | 15:25 | |
| sdague | fungi: yeh, I'm going to start pulling all those things out | 15:25 |
| *** dpyzhov has joined #openstack-infra | 15:26 | |
| fungi | crap. gate head -1 just failed, setting 65774 back a round | 15:26 |
| fungi | but it'll be the new head (or in the -1 slot when i promote the other one in a sec) | 15:27 |
| sdague | fungi: yeh, rest | 15:27 |
| sdague | I'm going to start pulling the cinder patches out | 15:28 |
| *** dstanek has joined #openstack-infra | 15:29 | |
| fungi | being more aggressive with node delete retries seems to have increased our capacity a bit at least, looking at the graph | 15:30 |
| *** rcleere has joined #openstack-infra | 15:30 | |
| *** ruhe_away is now known as _ruhe | 15:34 | |
| *** chandankumar has quit IRC | 15:35 | |
| *** mfer has joined #openstack-infra | 15:38 | |
| hemanth_ | hey, can anyone throw some light on why this isn't getting merged? https://review.openstack.org/#/c/65496/ | 15:38 |
| mordred | morning all | 15:39 |
| *** chandankumar has joined #openstack-infra | 15:39 | |
| *** thedodd has joined #openstack-infra | 15:39 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_a | 15:39 | |
| fungi | sdague: 66656,1 is on top of the pops now | 15:40 |
| *** SergeyLukjanov_a is now known as SergeyLukjanov_ | 15:40 | |
| fungi | hemanth_: 65496,4 has only two other changes ahead of it in the gate at this point. see http://status.openstack.org/zuul/ | 15:41 |
| *** wenlock has joined #openstack-infra | 15:41 | |
| fungi | morning, mordred | 15:41 |
| *** nati_uen_ has joined #openstack-infra | 15:41 | |
| *** nati_uen_ has quit IRC | 15:41 | |
| hemanth_ | fungi: ah, ok. this was one taking uncharacteristically long to get merged. not sure why | 15:42 |
| *** nati_uen_ has joined #openstack-infra | 15:42 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 15:43 | |
| fungi | mordred: i merged that envvar addition for the requirements integration job and am rerunning my whitelist change for pypi-mirror now. looks promising but i've tripped over a nondeterministic requests failure in pip a couple times (bug 1268725) | 15:43 |
| uvirtbot | Launchpad bug 1268725 in openstack-ci "ConnectionError: HTTPConnectionPool(host='pypi.openstack.org', port=80): Max retries exceeded" [Medium,Triaged] https://launchpad.net/bugs/1268725 | 15:43 |
| *** DinaBelova has quit IRC | 15:43 | |
| *** senk has quit IRC | 15:43 | |
| anteaya | mordred: morning | 15:43 |
| fungi | hemanth_: it's currently taking around 24 hours for a change to merge given the depth of the integrated gate queue and frequency of failing changes | 15:44 |
| *** vkozhukalov has quit IRC | 15:44 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 15:44 | |
| fungi | we made up some lost ground while i was asleep, but it's still a lot longer than it needs to be | 15:44 |
| sdague | yeh | 15:45 |
| dstufft | it took me way too long to realize that pypi.openstack.org was not pypi.python.org in that launchpad bug | 15:45 |
| *** nati_ueno has quit IRC | 15:45 | |
| fungi | sorry about that dstufft. i tripped your trigger again :/ | 15:45 |
| dstufft | fungi: heh it's no worries :) I should probably actually read hostnames instead of just applying pattern recognition to them | 15:46 |
| fungi | mordred: anyway, so far it's been hosts in hpcloud az2 both times, and that error suggests there could be an overloaded transparent proxy someone at your esteemed organization should know about (though i'm almost certain they already do if it is) | 15:47 |
| *** nicedice has joined #openstack-infra | 15:50 | |
| *** senk has joined #openstack-infra | 15:51 | |
| *** ^d has joined #openstack-infra | 15:51 | |
| *** ^d has joined #openstack-infra | 15:51 | |
| *** asadoughi has joined #openstack-infra | 15:51 | |
| *** sarob has joined #openstack-infra | 15:51 | |
| asadoughi | hi all, trying to debug why check-tempest-dsvm-neutron-isolated failed on https://review.openstack.org/#/c/62129/ - looks related to floating ips, but i don't see a log for q-l3 (neutron's l3 agent) | 15:52 |
| fungi | asadoughi: without having looked yet, i'd wager that could be caused by the l3 agent failing to start | 15:53 |
| *** senk1 has joined #openstack-infra | 15:53 | |
| asadoughi | fungi: i'm not sure, relevant request from neutron's logs: http://paste.openstack.org/show/61277/ shows agent is running | 15:54 |
| fungi | asadoughi: ahh, yeah comparing to other runs of the same job, it looks like maybe we don't create (or don't collect) a screen log for the l3 agent | 15:55 |
| fungi | i see one for what seems to be the l2 agent | 15:55 |
| fungi | no help, i know | 15:55 |
| fungi | taking a look in devstack to see how it launches it | 15:55 |
| *** senk has quit IRC | 15:56 | |
| asadoughi | right, devstacklog doesn't show a command for `screen_it q-l3...` | 15:56 |
| *** sarob has quit IRC | 15:56 | |
| mordred | fungi: yeah - I should really get around to getting us per-region pypi mirrors... | 16:02 |
| *** dcramer_ has quit IRC | 16:03 | |
| mrodden | requests 2.x broke a lot of stuff in my local jobs as well... | 16:03 |
| *** mfer has quit IRC | 16:03 | |
| *** oubiwann has joined #openstack-infra | 16:04 | |
| *** reed has joined #openstack-infra | 16:04 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack-infra/config: Install Hbase on OpenStack CI slaves https://review.openstack.org/66884 | 16:04 |
| *** kraman1 has quit IRC | 16:05 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Stop running manage-projects https://review.openstack.org/66692 | 16:06 |
| sdague | asadoughi: where are the logs for a sample job? | 16:06 |
| mordred | fungi: I'm going to try landing some new project patches and running manage-projects by hand for each one today | 16:07 |
| *** kraman has joined #openstack-infra | 16:07 | |
| markmcclain | asadoughi: do you have a q-vpn log? | 16:08 |
| kraman | mordred: ping | 16:08 |
| mordred | fungi: partially because i'm hoping I can catch some debug output that shows what's going wrong | 16:08 |
| mordred | kraman: morning! | 16:08 |
| kraman | mordred: morning | 16:08 |
| kraman | mordred: sould use some help if you can spare a few minutes | 16:08 |
| asadoughi | sdague: i don't know what you mean by sample job | 16:08 |
| asadoughi | markmcclain: http://logs.openstack.org/29/62129/4/check/check-tempest-dsvm-neutron-isolated/ce122cf/logs/screen-q-vpn.txt.gz | 16:09 |
| *** yamahata has quit IRC | 16:09 | |
| markmcclain | ok.. when the vpn service is active q-vpn replaces the q-l3 agent | 16:09 |
| markmcclain | (vpn runs inside of the same namespace as l3) | 16:10 |
| asadoughi | oh, i see | 16:10 |
| mordred | kraman: I looked at your zuul change briefly yesterday | 16:10 |
| *** dpyzhov has quit IRC | 16:10 | |
| *** yamahata has joined #openstack-infra | 16:10 | |
| *** mozawa has quit IRC | 16:11 | |
| markmcclain | asadoughi: yeah it can be a little confusing at first | 16:11 |
| markmcclain | might make sense for us to change the name | 16:12 |
| fungi | mordred: awesome. i welcome your manage-projects testing/troubleshooting | 16:12 |
| *** emagana has quit IRC | 16:12 | |
| *** esker has joined #openstack-infra | 16:12 | |
| mordred | fungi: I'd like to land https://review.openstack.org/#/c/66692/ first, if its ok with you | 16:13 |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 16:13 | |
| fungi | mordred: that's were i was basically at as well. next obvious step was to disable autorun and then test it with -v on multiple changes (and boy do we have a few) | 16:13 |
| *** dstanek has quit IRC | 16:14 | |
| *** _ruhe is now known as ruhe | 16:14 | |
| fungi | waiting to see if the syntax check passes on 66692 and then i'm good approving | 16:14 |
| sdague | fungi: 66863 is another candidate for promotion, so we can get experimental results back faster for neutron parallel tests | 16:16 |
| mordred | fungi: ++ | 16:16 |
| *** CaptTofu has joined #openstack-infra | 16:16 | |
| *** branen has joined #openstack-infra | 16:16 | |
| *** jgillies has joined #openstack-infra | 16:20 | |
| fungi | sdague: noted | 16:21 |
| *** asadoughi has quit IRC | 16:21 | |
| sdague | though, that's a config change, so shouldn't be impacted by the deep gate | 16:21 |
| sdague | just needs review | 16:21 |
| sdague | fungi: thnks | 16:22 |
| mordred | deep gate | 16:23 |
| fungi | yes | 16:23 |
| *** dpyzhov has joined #openstack-infra | 16:23 | |
| *** dstanek has joined #openstack-infra | 16:24 | |
| *** dpyzhov has quit IRC | 16:24 | |
| *** emagana has joined #openstack-infra | 16:24 | |
| fungi | 450 nodepool nodes churning nonstop | 16:24 |
| mordred | nice | 16:24 |
| mordred | hp cloud loves us | 16:24 |
| mordred | fungi: we never moved to the new 1.1 cloud did we? | 16:25 |
| fungi | well, a good chunk of those are also rackspace performance nodes now | 16:25 |
| mordred | oh good | 16:25 |
| fungi | checking to see what our node count is in the new hp region | 16:25 |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Upgrade manila's devstack-job to tempest-job https://review.openstack.org/66846 | 16:26 |
| fungi | mordred: yeah, hpcloud-region-b is only 5 nodes at the moment... http://paste.openstack.org/show/61279/ | 16:28 |
| mordred | fungi: did we ever get graphs of relative performance of our regions? | 16:28 |
| fungi | mordred: i don't recall | 16:29 |
| mordred | k | 16:29 |
| * mordred trying to re-ramp up on that situation | 16:29 | |
| fungi | i do remember we scaled back off hpcloud-region-b after it showed something like 2x duration average for jobs run there, but i don't remember getting more detailed analysis | 16:29 |
| mordred | yah. well, the feedback I got from the hp cloud guys was to try larger images, as they do get larger cpu rate limits | 16:30 |
| mordred | but before I tried that, I wanted to make sure we had a baseline against which to evaluate | 16:30 |
| fungi | good idea | 16:30 |
| *** hashar has quit IRC | 16:31 | |
| *** gyee_ has joined #openstack-infra | 16:32 | |
| *** thuc has joined #openstack-infra | 16:34 | |
| *** dstanek has quit IRC | 16:35 | |
| *** nati_ueno has joined #openstack-infra | 16:36 | |
| fungi | syntax passed on 66692 so it's going in now | 16:37 |
| *** sparkycollier has joined #openstack-infra | 16:37 | |
| fungi | https://jenkins02.openstack.org/job/check-requirements-integration-dsvm/81/console is still going strong after nearly 45 minutes, so we may have a winner | 16:38 |
| *** smarcet has quit IRC | 16:38 | |
| *** nati_uen_ has quit IRC | 16:39 | |
| *** jcoufal has quit IRC | 16:41 | |
| jog0 | fungi: so libvirt 1.x | 16:41 |
| *** mfer has joined #openstack-infra | 16:41 | |
| fungi | jog0: good news?!? | 16:41 |
| jog0 | I was going to ask you | 16:41 |
| fungi | d'oh! | 16:41 |
| fungi | my weekend sucked of fixing other stuff and i totally spaced on that | 16:41 |
| fungi | sorry :/ | 16:42 |
| jog0 | so I talked to sdague about it just now and he said the libvirt issue was intermittent | 16:42 |
| fungi | lemme fire one up now | 16:42 |
| jog0 | fungi: sweet | 16:42 |
| fungi | got one building with uca added now | 16:43 |
| jog0 | fungi: so if things work, we may still have a intermittent issue | 16:44 |
| fungi | noted | 16:45 |
| fungi | i may loop it a few times over in the background while working on other stuff, in that case | 16:45 |
| jog0 | fungi: awesome thanks | 16:47 |
| *** dims_ is now known as dims | 16:47 | |
| fungi | i've definitely got more fires than irons now | 16:47 |
| *** DinaBelova has joined #openstack-infra | 16:47 | |
| *** coolsvap is now known as coolsvap_away | 16:49 | |
| *** boris-42 has joined #openstack-infra | 16:49 | |
| *** blamar has quit IRC | 16:50 | |
| SpamapS | fungi: hey so did you ever figure out what is going on with that missing commit? | 16:50 |
| fungi | SpamapS: yeah, you had a corrupt local clone i think | 16:51 |
| SpamapS | fungi: if I switch remotes to github I can git pull | 16:51 |
| fungi | weird. if i git gc'd it, git fsck kept reporting the same one dangling commit and would never remove it | 16:51 |
| SpamapS | fungi: so that is simply not true. | 16:51 |
| *** sarob has joined #openstack-infra | 16:51 | |
| fungi | and git show complained that it was unreadable | 16:51 |
| SpamapS | fungi: my theory: we have found a bug in cgit | 16:52 |
| fungi | and i couldn't find any record of that commit in any of the remotes | 16:52 |
| SpamapS | but because I can work around it by git remote changing.. we're not fixing said bug | 16:52 |
| fungi | lemme refresh my memory on it | 16:52 |
| SpamapS | because this is maybe the 5th time I have had this problem | 16:52 |
| fungi | i'm sensing a commonality ;) | 16:53 |
| SpamapS | right but why would github fix the problem is my question | 16:53 |
| fungi | git fsck on it claims 0c83ac25a912843cfdee3ecd8bfb70921d326974 is dangling | 16:53 |
| *** yassine has quit IRC | 16:54 | |
| fungi | git show on that tells me fatal: unable to read source tree (cba33172c368914ef38f1a97f4bcf2195601a599) | 16:54 |
| *** yassine has joined #openstack-infra | 16:54 | |
| SpamapS | https://github.com/openstack/heat/commit/0c83ac25a912843cfdee3ecd8bfb70921d326974 | 16:54 |
| fungi | and git show on that tells me fatal: bad object cba33172c368914ef38f1a97f4bcf2195601a599 | 16:54 |
| fungi | i'll go back on a hunt for 0c83ac25a912843cfdee3ecd8bfb70921d326974 and cba33172c368914ef38f1a97f4bcf2195601a599 on the remotes | 16:54 |
| SpamapS | me too | 16:54 |
| SpamapS | cba33172c368914ef38f1a97f4bcf2195601a599 does not exist on github | 16:54 |
| *** AaronGr_afk is now known as AaronGr | 16:55 | |
| *** thedodd has quit IRC | 16:55 | |
| SpamapS | fungi: so I see that 0c83ac25a912843cfdee3ecd8bfb70921d326974 is borked in my repo.. but my question is, how is git resolving that with github, but not with cgit | 16:56 |
| *** sarob has quit IRC | 16:56 | |
| SpamapS | Is it possible one is using "smart server" and one is falling back to "dumb" ? | 16:57 |
| *** thedodd has joined #openstack-infra | 16:57 | |
| sdague | fungi: any idea who at foundation I should harass about the fact that I didn't get an ballot for the election? | 16:57 |
| fungi | maybe. have you tried switching from https:// to git:// and seeing if that changes it? | 16:57 |
| *** burt has quit IRC | 16:57 | |
| fungi | sdague: jbryce might have insight into how the ballot e-mails went out. did you confirm your primary e-mail address on your foundation profile is updated to be one you still receive? | 16:58 |
| SpamapS | fungi: that does resolve the issue.. | 16:59 |
| SpamapS | fungi: however git:// == mitm evil | 16:59 |
| SpamapS | fungi: but, as a data point.. that is useful no? | 16:59 |
| fungi | SpamapS: yes, i know, just testing your theory about the smart git server | 16:59 |
| *** coolsvap_away is now known as coolsvap | 17:00 | |
| jbryce | sdague: just resent it to you | 17:00 |
| *** marun has quit IRC | 17:00 | |
| *** dkliban is now known as dkliban_afk | 17:01 | |
| *** hogepodge_ has joined #openstack-infra | 17:01 | |
| *** chandankumar has quit IRC | 17:01 | |
| *** hogepodge has quit IRC | 17:01 | |
| *** hogepodge_ is now known as hogepodge | 17:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Temporarily remove neutron full job from experimental queue https://review.openstack.org/66863 | 17:03 |
| SpamapS | fungi: are we sure cgit speaks "smart http" ? | 17:04 |
| fungi | mordred: IT LIVES! https://review.openstack.org/66364 | 17:04 |
| *** mrodden has quit IRC | 17:04 | |
| *** mfer has quit IRC | 17:05 | |
| fungi | SpamapS: i'm unsure. i think the url may be going apache->git cgi since there's no /cgit/ component in the remote url | 17:05 |
| * fungi checks the vhost config | 17:05 | |
| SpamapS | fungi: there is | 17:05 |
| SpamapS | fungi: originhttps://git.openstack.org/cgit/openstack/heat (push) | 17:05 |
| *** pblaho has quit IRC | 17:05 | |
| fungi | oh, wait... | 17:05 |
| SpamapS | (same for fetch) | 17:05 |
| fungi | why did i not notice that before? | 17:06 |
| fungi | SpamapS: what happens if you change your remote to https://git.openstack.org/openstack/heat.git instead? | 17:06 |
| fungi | THAT should be skipping cgit and going straight through apache | 17:06 |
| *** mfer has joined #openstack-infra | 17:06 | |
| fungi | i didn't even realize cgit had an option of acting as a git server | 17:07 |
| SpamapS | trying that | 17:07 |
| *** afazekas has joined #openstack-infra | 17:08 | |
| SpamapS | fatal: https://git.openstack.org/cgit/openstack/heat.git/info/refs not valid: is this a git repository? | 17:08 |
| SpamapS | oh | 17:08 |
| SpamapS | no cgit | 17:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Stop running manage-projects https://review.openstack.org/66692 | 17:08 |
| SpamapS | fungi: aha | 17:08 |
| SpamapS | fungi: that does in fact fix it | 17:08 |
| SpamapS | fungi: ok so cgit is bong as a fetch url :) | 17:08 |
| sdague | jbryce: thanks | 17:09 |
| *** tma996 has quit IRC | 17:09 | |
| fungi | SpamapS: seems that way. also explains why i'd never encountered it | 17:09 |
| *** max_lobur is now known as max_lobur_afk | 17:09 | |
| *** pballand has joined #openstack-infra | 17:09 | |
| *** rakhmerov has joined #openstack-infra | 17:09 | |
| *** marun has joined #openstack-infra | 17:10 | |
| SpamapS | fungi: well at least now I know I can just fix my remotes | 17:10 |
| SpamapS | fungi: but perhaps a 301 is in order. :) | 17:10 |
| fungi | SpamapS: not a bad idea at all | 17:10 |
| *** dcramer_ has joined #openstack-infra | 17:10 | |
| SpamapS | fungi: probably not the simplest thing though, you only want to 301 _git_ not Firefox :) | 17:11 |
| fungi | though that might take a bit of doing since it'd be shadowing subpaths of the webui, yeah | 17:11 |
| SpamapS | fungi: want me to open an openstack-ci bug? | 17:11 |
| fungi | SpamapS: please, please do | 17:11 |
| SpamapS | I think it is worth documenting as a Low-Importance-But-Hard thing | 17:11 |
| * SpamapS feels the warm flow of understanding trickling down his head into the cockles of his heart | 17:11 | |
| fungi | i'll try to get around to catching up on bug triage sometime soon. i started in on it last night and only got about 3 done before i passed out | 17:11 |
| *** krotscheck has joined #openstack-infra | 17:14 | |
| *** branen has quit IRC | 17:15 | |
| SpamapS | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1269510 filed | 17:16 |
| uvirtbot | Launchpad bug 1269510 in openstack-ci "git.openstack.org/cgit should not be used to serve git directly" [Undecided,New] | 17:16 |
| *** jasondotstar has quit IRC | 17:20 | |
| *** mrodden has joined #openstack-infra | 17:21 | |
| fungi | SpamapS: a quick bughunt for cgit doesn't turn up obvious candidates | 17:21 |
| *** che-arne has quit IRC | 17:23 | |
| *** branen has joined #openstack-infra | 17:24 | |
| *** senk1 has quit IRC | 17:25 | |
| SpamapS | fungi: ty for working on it. I've now gone through and fixed all my remotes. | 17:25 |
| *** mancdaz is now known as mancdaz_away | 17:26 | |
| fungi | SpamapS: you're welcome. sorry for the inconvenience | 17:26 |
| SpamapS | fungi: np .. as I alluded to earlier.. relief is had.. knowing is beautiful. :) | 17:27 |
| *** moted has joined #openstack-infra | 17:27 | |
| fungi | indeed | 17:27 |
| *** dizquierdo has quit IRC | 17:29 | |
| *** markmc has quit IRC | 17:30 | |
| *** senk has joined #openstack-infra | 17:30 | |
| *** afazekas has quit IRC | 17:31 | |
| mordred | fungi: woot! I would like to land that | 17:36 |
| fungi | mordred: so if you're good with https://review.openstack.org/66364 i think i want to try to cram it through today so all the pending requirements/pbr/pypi-mirror changes can be rechecked | 17:36 |
| *** sparkycollier has quit IRC | 17:36 | |
| mordred | fungi: yes please | 17:36 |
| mordred | fungi: in a perfect world, I think that we should re-write that to consume the whitelist froma config file | 17:37 |
| mordred | BUT | 17:37 |
| mordred | I dont' care | 17:37 |
| fungi | see the notes i just added to the last comment on the review | 17:37 |
| mordred | so ha | 17:37 |
| mordred | yeah. actually, I agree with your note | 17:37 |
| fungi | the uglier it is, the more incentive to remove it soon ;) | 17:37 |
| mordred | fungi: I've +2'd it - land at will | 17:37 |
| fungi | thanks | 17:38 |
| *** sparkycollier has joined #openstack-infra | 17:38 | |
| *** marun has quit IRC | 17:39 | |
| notmyname | the patch queue was reset about 2 hours ago? (implied from the times on http://f.slukjanov.name/w/review.o.o/66056/1/status.html) | 17:39 |
| *** sparkycollier has quit IRC | 17:39 | |
| fungi | notmyname: yes, the way the prioritization works, any new patchsets in the check pipeline which landed before the gate reset get dibs on next available slaves, so those are still in the process of being serviced | 17:40 |
| *** marun has joined #openstack-infra | 17:40 | |
| fungi | i'm manually retrying deletes on all discarded slaves now to try and get us a little extra capacity so we can get through them more quickly | 17:41 |
| notmyname | fungi: no, there are patches that currently have a time of about 2 hours that have actually been in the gate queue for closer to 20 hours. | 17:41 |
| fungi | notmyname: oh, i see what you're talking about. that's addressed by https://review.openstack.org/66095 but we need a zuul restart for that to take effect and the volume's been too high to justify the extra impact | 17:43 |
| fungi | ~2 hours ago we reordered the queue simultaneous with a gate reset which was taking place so that we could bump another stability fix to the front | 17:44 |
| notmyname | ok | 17:44 |
| mordred | I feel like we're cooking in a high-volume kitchen | 17:44 |
| *** burt has joined #openstack-infra | 17:45 | |
| *** emagana has quit IRC | 17:45 | |
| notmyname | so any time a manual reorder happens, the time is reset? ie there isn't a way to automatically reorder gate bugs like there is with CVE bugs? | 17:45 |
| *** marun has quit IRC | 17:45 | |
| *** emagana has joined #openstack-infra | 17:45 | |
| *** nati_ueno has quit IRC | 17:45 | |
| *** salv-orlando has quit IRC | 17:45 | |
| *** salv-orlando has joined #openstack-infra | 17:46 | |
| *** nati_ueno has joined #openstack-infra | 17:46 | |
| fungi | notmyname: right. the "zuul promote" command was implemented as a dequeue-and-requeue operation, so the enqueued time on those changes was being reset. that fix saves and reapplies the timestamp in that event | 17:46 |
| notmyname | fungi: thanks for the info | 17:47 |
| fungi | you're welcome | 17:47 |
| fungi | mordred: yes, and i keep setting my apron on fire | 17:47 |
| fungi | order up! | 17:48 |
| *** ruhe is now known as ruhe_away | 17:49 | |
| mordred | ok. landing storyboard-webclient addition - will be guinea pig #1 | 17:50 |
| fungi | mordred: i'm thrilled at all the progress on that | 17:51 |
| fungi | kudos to all the awesome people working to make it happen | 17:51 |
| krotscheck | Hooookay. Let's do this. | 17:52 |
| mordred | fungi: fun part is - I think horizon is interested in also doing some similar things with tooling, so we can test-run them on storyboard before they start to actually affect opensatck itself | 17:53 |
| krotscheck | david-lyle was commenting on that during the meeting yesterday. There's still a lot of worries about there not being an official recent package for node on rhel/ubuntu yet though :/ | 17:54 |
| *** emagana has quit IRC | 17:54 | |
| *** salv-orlando has quit IRC | 17:54 | |
| david-lyle | there was promise though | 17:55 |
| mordred | krotscheck: yeah - but there is recent enough to support the toolchain build-side though, yeah? | 17:55 |
| *** salv-orlando has joined #openstack-infra | 17:55 | |
| *** herndon has quit IRC | 17:55 | |
| david-lyle | where we had none before | 17:55 |
| *** emagana has joined #openstack-infra | 17:55 | |
| krotscheck | mordred: Let me check. I don't think so. | 17:55 |
| mordred | 0.10.24 is what's in debian | 17:55 |
| fungi | yeah, and you're targeting trusty for the deployment platform anyway, right? | 17:56 |
| mordred | yup | 17:57 |
| fungi | it's got 0.10.24 right now as well | 17:57 |
| krotscheck | precise has 0.6.12 | 17:57 |
| mordred | if precise doesn't have a nodejs package - I expect that's simple enough to backport | 17:57 |
| mordred | and put in UCA | 17:57 |
| krotscheck | UCA? Universal cheerleader association? | 17:57 |
| mordred | I consider that sane enough | 17:57 |
| mordred | krotscheck: "Ubuntu Cloud Archive" | 17:58 |
| fungi | ubuntu cloud awesomeness | 17:58 |
| mordred | it's the archive ubuntu produces with backports of needed software for older ubuntu releases like precise | 17:58 |
| *** ruhe_away is now known as _ruhe | 17:58 | |
| krotscheck | Got it. | 17:58 |
| *** rossella_s has quit IRC | 17:59 | |
| zaro | krotscheck: great work! | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| zaro | fungi, mordred: in order to build gerrit 3rd party plugins we need to add them as submodules to gerrit core. how do you feel about that? | 18:01 |
| mordred | zaro: wow, really/ | 18:01 |
| mordred | ? | 18:01 |
| mordred | that's a terrible design | 18:01 |
| mordred | zaro: I feel very poorly about it | 18:02 |
| mordred | zaro: is there actually no other way to do it? | 18:02 |
| *** jerryz has joined #openstack-infra | 18:02 | |
| zaro | mordred: it's becuase upstream hasn't setup a repsoitory for their binaries. so plugins can reference pre-built core. | 18:02 |
| fungi | mordred: zaro: the problem is that the submodules list in the gerrit repo contains relative filesystem paths, not urls | 18:02 |
| mordred | hang on ... | 18:02 |
| mordred | what if | 18:02 |
| mordred | we don't actualy add and COMMIT the submodules | 18:03 |
| mordred | but we treat "git submodule add" as a build step | 18:03 |
| mordred | and do it every time | 18:03 |
| zaro | mordred: that will work. | 18:03 |
| fungi | how do we feel about patching that list, maybe convincing upstream it's even desirable? | 18:03 |
| zaro | fungi: patching what list? | 18:04 |
| mordred | that way, the build job for "gerrit-plugin-foo" would clone gerrit, then cd to the right place, then git submodule add gerrit-plugin-foo, then build it | 18:04 |
| mordred | fungi: ^^ | 18:04 |
| fungi | zaro: http://git.openstack.org/cgit/openstack-infra/gerrit/tree/.gitmodules?h=openstack/2.8 | 18:04 |
| *** smarcet has joined #openstack-infra | 18:05 | |
| zaro | fungi: uhh unlikely. because that this is only core plugins. | 18:05 |
| mordred | fungi: I don't think we need to | 18:05 |
| zaro | *this/list | 18:05 |
| fungi | zaro: ohhh.... i get what you're saying. third-party modules for gerrit, not the core modules | 18:05 |
| *** gothicmindfood has joined #openstack-infra | 18:05 | |
| zaro | fungi: yep. | 18:05 |
| krotscheck | gothicmindfood: mordred just landed storyboard-webclient | 18:05 |
| mordred | krotscheck: hasn't landed yet | 18:06 |
| zaro | i think mordred is on to something.. | 18:06 |
| mordred | still waiting on jenkins bandwidth | 18:06 |
| krotscheck | mordred: Ah, my bad | 18:06 |
| *** dkliban_afk is now known as dkliban | 18:06 | |
| * gothicmindfood would like to break out the champagne anyways | 18:06 | |
| mordred | krotscheck: it's been approved - when it lands, I'm going to manually run manage-projects so that if the bug happens I can watch it and debug | 18:06 |
| fungi | krotscheck: are we ready for me to try and launch you a dev server with your puppetry patch? | 18:06 |
| fungi | oh, i guess it needs that repo to exist first | 18:07 |
| mordred | ysh | 18:07 |
| mordred | ya | 18:07 |
| zaro | mordred: so would we rebuid gerrit core in every gerrit plugin job? | 18:07 |
| mordred | zaro: why not? | 18:07 |
| mordred | rebuilding gerrit core should result in the same thing, yeah? | 18:07 |
| *** ok_delta has quit IRC | 18:07 | |
| fungi | we have clouds. i'm not too worried about the inefficiency of rebuilding gerrit every time we rebuild a plugin | 18:07 |
| zaro | mordred: that will work. fungi answered my question. | 18:08 |
| mordred | also, I don't expect to rebuild any of them all that frequently | 18:08 |
| markmcclain | fungi: 66872 needs promotion | 18:08 |
| fungi | markmcclain: thanks having a look | 18:09 |
| markmcclain | fungi: thanks | 18:09 |
| fungi | markmcclain: yep, it's going in on the next reset | 18:10 |
| markmcclain | awesome | 18:10 |
| zaro | mgagne: ping | 18:12 |
| mgagne | zaro: sup! | 18:12 |
| *** obondarev_ has joined #openstack-infra | 18:13 | |
| pballand | I am a setuptools/distutils/pbr newbie, and have some commands (elg. make) I need to run before packaging. Is there better way to do this than adding logic before setuptools.setup() in setup.py? | 18:13 |
| *** sarob has joined #openstack-infra | 18:13 | |
| *** nati_ueno has quit IRC | 18:14 | |
| zaro | mgagne: pelix had a good question.. it's regarding review 52080 | 18:14 |
| mgagne | zaro: regarding ini sections? | 18:14 |
| *** nati_ueno has joined #openstack-infra | 18:14 | |
| mordred | hi pballand | 18:14 |
| zaro | mgagne: yes, what should be placed into separate sections? | 18:14 |
| zaro | mgagne: what is your thought? | 18:15 |
| mgagne | zaro: I'm on the fence regarding this feature tbh | 18:15 |
| mordred | pballand: can you expand a little bit on what you're trying to do? as in, why are you wanting to run make? (I'm wanting to help to find the right answer for you) | 18:15 |
| mgagne | zaro: I mean, what prevents someone from adding an other do not overwrite X | 18:15 |
| mordred | pballand: in general, in pbr land, you never want to add code to setup.py - but there are other ways | 18:16 |
| pballand | mordred: sure: the project has a file that needs to be "compiled" (its actually a grammar which is parsed to create a parser) | 18:16 |
| zaro | mgagne: yeah, i haven't seen a use case that requires another section. | 18:16 |
| *** thedodd has quit IRC | 18:17 | |
| mordred | pballand: cool. for which things does that need to be done? sdist/install? just sdist? just install? | 18:17 |
| mgagne | zaro: lets put it in DEFAULT until we find an other use case | 18:17 |
| zaro | mgagne: also as you pointed out it requires more validation. | 18:17 |
| mgagne | zaro: IMO, jenkins is for jenkins credentials | 18:17 |
| pballand | mordred: probably just install - for sdist I think just including the grammar is sufficient | 18:18 |
| mordred | ok | 18:18 |
| mordred | pballand: let me find a link to point you at... | 18:18 |
| *** rossella_s has joined #openstack-infra | 18:18 | |
| *** mrmartin has joined #openstack-infra | 18:18 | |
| pballand | mordred: awesome, thanks! | 18:18 |
| mgagne | zaro: with openstack projects, people tends to put everything in DEFAULT until someone comes and suggests moving all related configs to a new section/group. | 18:18 |
| zaro | mgagne: i'm ok with 52080, would you please give it a re-review? let's push that thru because it's been around for a long time. | 18:19 |
| mrmartin | re | 18:19 |
| zaro | mgagne: yeah, i think that's the way to do it. | 18:19 |
| mgagne | zaro: you tested the change? | 18:19 |
| mordred | pballand: do you have a repo anywhere? | 18:19 |
| mgagne | zaro: my setup is dead, it got killed during christmas =( | 18:19 |
| pballand | mordred: https://github.com/stackforge/congress | 18:20 |
| *** gothicmindfood_ has joined #openstack-infra | 18:20 | |
| mordred | pballand: cool. I'll make you a non-functional patch | 18:20 |
| mordred | pballand: that will show you where it should go | 18:20 |
| pballand | mordred: prefect - if you are so inclined, the actual command I need to run is sitting there in Makefile | 18:21 |
| zaro | mgagne: dead? what happened. | 18:21 |
| zaro | mgagne: don't remember now, since it's been so long. but i'm pretty sure i did. | 18:22 |
| mgagne | zaro: someone had the great idea to reinstall the compute nodes and live migration wasn't setup yet =) | 18:22 |
| *** emagana has quit IRC | 18:22 | |
| *** gothicmindfood has quit IRC | 18:22 | |
| *** praneshp has joined #openstack-infra | 18:23 | |
| *** emagana has joined #openstack-infra | 18:23 | |
| zaro | mgagne: yikes! no backup? | 18:23 |
| mgagne | zaro: nothing =) I should be able to set it back up in a couple of minutes, didn't have time to do it yet | 18:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Add storyboard-webclient to gerrit https://review.openstack.org/64761 | 18:25 |
| fungi | krotscheck: gothicmindfood_: ^ | 18:26 |
| krotscheck | Oh mhyyyyyy | 18:26 |
| fungi | mordred: still has to actually run manage-projects on it though | 18:26 |
| zaro | fungi: got a quick minute to review 1st pass at an upgrade script? https://etherpad.openstack.org/p/gerrit_upgrade_script | 18:26 |
| fungi | krotscheck: you sounded like sulu just then | 18:26 |
| fungi | s/sulu/takei/ | 18:27 |
| krotscheck | fungi: I'm so takey | 18:27 |
| krotscheck | fungi: I'm so takei | 18:27 |
| zaro | mgagne: fantastic! | 18:27 |
| fungi | zaro: sure. i mean, not really but i will anyway | 18:27 |
| krotscheck | Well, maybe not _that_ takei | 18:27 |
| *** openstackgerrit has quit IRC | 18:27 | |
| *** openstackgerrit has joined #openstack-infra | 18:27 | |
| *** nicedice has quit IRC | 18:28 | |
| mordred | fungi: m-p seems to have run successfully - where shoudl I look for failure? | 18:29 |
| fungi | zaro: i had never followed up, but i guess the schema upgrades were running into a mismatch between account_group_by_id and account_groups tables then? | 18:29 |
| *** nicedice has joined #openstack-infra | 18:29 | |
| fungi | mordred: git branch -a in /home/gerrit2/review_site/git/openstack-infra/storyboard-web-client.git | 18:30 |
| fungi | or i guess pass that as --git-dir= | 18:30 |
| fungi | same for the one in /var/lib/git | 18:31 |
| mordred | fungi: yup. both worked | 18:31 |
| fungi | also, you may have simply gotten (un)lucky | 18:31 |
| mordred | yah. I just want to make sure I'm looking for the right thing | 18:31 |
| fungi | new groups, if any, were created and acl looks correct in gerrit webui? | 18:31 |
| fungi | that covers the suite of obvious errors i've seen manifest when it's run from puppet | 18:32 |
| mrmartin | fungi: hi | 18:33 |
| zaro | fungi: i'm not sure what the root of the problem is but the truncate workaround seems to fix the problem. it's explained in the gerrit issue. | 18:33 |
| *** nati_ueno has quit IRC | 18:34 | |
| zaro | fungi: all i know is that 'ICLA' group doesn't get added to any of the db tables. | 18:34 |
| *** nati_ueno has joined #openstack-infra | 18:34 | |
| mrmartin | fungi: I made the script templates required for community portal testing and tarball release | 18:35 |
| fungi | mrmartin: oh, awesome! | 18:36 |
| *** harlowja has joined #openstack-infra | 18:36 | |
| mrmartin | fungi: http://paste.openstack.org/show/61287/ and http://paste.openstack.org/show/61288/ | 18:36 |
| fungi | zaro: right, i'm guessing that's unrelated to the account_group_by_id problem though | 18:36 |
| fungi | though i guess it could be related | 18:36 |
| *** fbo is now known as fbo_away | 18:37 | |
| mrmartin | fungi: the first one is a simple build script, and the second one will be the unittest, currently it just makes a packages, and tests the installation process | 18:37 |
| *** marun has joined #openstack-infra | 18:37 | |
| fungi | mrmartin: we can probably do this safely on single-use slaves. i'm not keen on sudo'ing pear to install php packages on long-running slaves | 18:38 |
| *** harlowja_away has quit IRC | 18:39 | |
| mrmartin | fungi: yeap, of course. if we can publish snapshot tarballs, it is a half success, because I can update the puppet scripts for groups-dev.openstack.org and groups.openstack.org. | 18:39 |
| fungi | mrmartin: do these work sanely on plain cloud images for ubuntu 12.04 lts? | 18:39 |
| mrmartin | I've tested those with vagrant | 18:40 |
| fungi | if so, it'll probably be fairly easy to implement | 18:40 |
| jgriffith | fungi: any pointers on how to add local gating tests/checks here in my lab similar to what some of the neutron vendors are doing? | 18:41 |
| zaro | fungi: no, that's unreleated. | 18:41 |
| zaro | fungi: account_group_by_id needs to be truncated to allow upgrade script to run thru without failing. | 18:41 |
| mrmartin | fungi: I'll open a blueprint on launchpad for this integration, so we can track the progress there. | 18:41 |
| fungi | mrmartin: the scripts themselves can go into a tools or similar subdirectory in the groups project, and then we can just call them from fairly simple jenkins jobs on single-use slaves and collect up the scraps left behind by the build or the exit code of the test script | 18:41 |
| zaro | fungi: after it completes there's another problem with signing CLA because of the missing 'ICLA' group in the db. | 18:42 |
| mrmartin | ok, we have a scripts directory there. | 18:42 |
| fungi | jgriffith: http://ci.openstack.org/third_party.html | 18:42 |
| jgriffith | fungi: thanks! | 18:42 |
| fungi | zaro: yep, i remember that one more clearly since i spent some time hunting it down | 18:42 |
| fungi | mrmartin: that should be plenty fine i would think | 18:43 |
| mrmartin | what I currently need to understand, how we could integrate this with zuul/jenkins | 18:43 |
| mrmartin | fungi: ok, I'll do it now, and come back when it is ready. | 18:43 |
| zaro | i could not find ICLA group info in any of the db tables, pre or post migration. | 18:43 |
| zaro | fungi: so i had to find the UUID in the All-Projects group file | 18:44 |
| fungi | mrmartin: the jobs are configured in here... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config | 18:44 |
| *** nicedice has quit IRC | 18:44 | |
| *** nicedice has joined #openstack-infra | 18:44 | |
| fungi | zaro: right, i saw it had generated a uuid so i suspected you'd have to fish it out of the file | 18:44 |
| fungi | mrmartin: those are templated/abstracted yaml descriptions of jenkins jobs which run things | 18:45 |
| mrmartin | Yep, I previously checked the storyboard jobs, but currenlty not everything is clear for me | 18:45 |
| fungi | mrmartin: yeah, the storyboard.yaml file is a pretty good example of the sorts of things you'd be doing to run those scripts and act on/collect the results | 18:46 |
| fungi | mrmartin: and then zuul has a list of what jobs to run when on changes for which projects... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 18:46 |
| mordred | fungi: I'm on a weird network segment - can you ssh -p 29418 review.openstack.org replicate openstack-infra/storyboard-webclient for me? | 18:47 |
| mrmartin | in this review: https://review.openstack.org/#/c/64761/7/modules/openstack_project/files/jenkins_job_builder/config/storyboard.yaml where the bootstrap.sh and build.sh files are coming from? | 18:47 |
| fungi | mordred: _gerrit_ replicate but sure, done just now | 18:48 |
| mordred | fungi: thanks | 18:48 |
| fungi | np | 18:48 |
| openstackgerrit | Edward Raigosa proposed a change to openstack-infra/config: Make pip install from upstream better https://review.openstack.org/51425 | 18:48 |
| *** dprince has quit IRC | 18:49 | |
| wenlock | mordred, hope that patch is still desirable | 18:49 |
| fungi | mrmartin: ./bin/bootstrap.sh is relative to the project's git repository which gets checked out by the gerrit-git-prep builder in that job | 18:49 |
| fungi | mrmartin: so you might have ./scripts/build.sh or similar | 18:50 |
| mordred | wenlock: always. also - have you met AaronGr yet? | 18:50 |
| wenlock | indeed i did, had a good chat with him last week | 18:50 |
| mrmartin | fungi: ok, thanks, Now I need to manage the evening tale tasks for children, I'll come back soon. | 18:50 |
| fungi | mrmartin: enjoy. talk to you later | 18:51 |
| mordred | awesome | 18:51 |
| *** mrmartin has quit IRC | 18:51 | |
| wenlock | mordred, we're really interested in some of the refactoring he has in mind... hope he gets traction on that | 18:51 |
| mgagne | zaro: DEFAULT section it should be right? | 18:52 |
| AaronGr | wenlock: a few patches in flight at the moment, a massive one pending. | 18:53 |
| wenlock | AaronGr, nice! :D | 18:53 |
| zaro | mgagne: huh? should be right? | 18:53 |
| mgagne | zaro: keep_descriptions config | 18:53 |
| mgagne | zaro: DEFAULT_CONF puts it in DEFAULT, actual code expects it to be in jenkins | 18:54 |
| mordred | I think fungi will like most of AaronGr's patches | 18:54 |
| *** elasticio has joined #openstack-infra | 18:55 | |
| mgagne | zaro: and doc mentions keep_description, not keep_descriptionS... | 18:55 |
| mgagne | zaro: consistency! :O | 18:55 |
| *** marun has quit IRC | 18:55 | |
| *** marun has joined #openstack-infra | 18:55 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Cleanup HACKING.rst https://review.openstack.org/62133 | 18:56 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Move hacking guide to root directory https://review.openstack.org/62132 | 18:56 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Turn Python3 section into a list https://review.openstack.org/62135 | 18:56 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Re-Add section on assertRaises(Exception https://review.openstack.org/62134 | 18:56 |
| openstackgerrit | Joe Gordon proposed a change to openstack-dev/hacking: Add Python3 deprecated assert* to HACKING.rst https://review.openstack.org/62136 | 18:56 |
| zaro | mgagne: line 97, loads all of DEFAULT_CONF | 18:57 |
| fungi | mordred: if they're fixing things i don't need to fix, i'm very much in favor | 18:57 |
| mgagne | zaro: sure. go see the value of DEFAULT_CONF =) | 18:57 |
| *** sarob has quit IRC | 18:58 | |
| mordred | fungi: well - for now they're refactoring out things that are hardcoded, and also squashing the inheritance heirarchy a bit for simplification | 18:58 |
| *** sarob has joined #openstack-infra | 18:58 | |
| mordred | fungi: but yeah | 18:59 |
| *** eharney has quit IRC | 18:59 | |
| zaro | mgagne: ohh you mean the mismatch name 'keep_description'? | 18:59 |
| *** emagana has quit IRC | 18:59 | |
| mgagne | zaro: well, everything | 18:59 |
| mgagne | zaro: there is no consistency between code, doc and default value | 18:59 |
| *** emagana has joined #openstack-infra | 18:59 | |
| fungi | mordred: sounds great | 19:00 |
| zaro | mgagne: ohh i see, glad i got you to take a look. i didn't noticed mismatches. | 19:01 |
| mgagne | zaro: well, I tried the patch and it didn't work. =) | 19:01 |
| fungi | zaro: your scripted upgrade plan looks great. i think you missed populating the contents of the new group (so that users don't need to re-accept the icla) so i added a placeholder there | 19:01 |
| zaro | mgagne: my mind's playing tricks on me then. i might have only tested one of the earlier patches. | 19:02 |
| mgagne | zaro: yha, np | 19:02 |
| *** sarob has quit IRC | 19:03 | |
| *** emagana_ has joined #openstack-infra | 19:04 | |
| *** mrmartin has joined #openstack-infra | 19:06 | |
| *** jcoufal has joined #openstack-infra | 19:06 | |
| *** harlowja_ has joined #openstack-infra | 19:07 | |
| *** emagana has quit IRC | 19:07 | |
| *** sarob has joined #openstack-infra | 19:07 | |
| *** dprince has joined #openstack-infra | 19:09 | |
| *** pballand has quit IRC | 19:10 | |
| *** harlowja has quit IRC | 19:10 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Added route tests for authentication https://review.openstack.org/66911 | 19:11 |
| zaro | fungi: great thanks. is there a place in gerrit for this script or is best place in etherpad? | 19:11 |
| sdague | fungi: https://review.openstack.org/#/c/66872/ is a promote that we need for grenade | 19:12 |
| sdague | to get working code on havana branch for neutron | 19:13 |
| *** _david_ has joined #openstack-infra | 19:14 | |
| jog0 | fungi: how is libvirt 1.x test going? | 19:17 |
| *** emagana_ has quit IRC | 19:17 | |
| *** pballand has joined #openstack-infra | 19:17 | |
| fungi | sdague: it's already in my hopper waiting for me to catch the next major gate reset | 19:17 |
| sdague | fungi: also 66656 needs a promote still as well | 19:17 |
| *** emagana has joined #openstack-infra | 19:17 | |
| *** marun has quit IRC | 19:18 | |
| *** marun has joined #openstack-infra | 19:18 | |
| fungi | jog0: for some reason my tempest runs are aborting in safe-devstack-vm-gate-wrap.sh with no output after the ip configuration output. i haven't had time to dig into why yet | 19:18 |
| *** gokrokve has quit IRC | 19:18 | |
| jog0 | fungi: kk | 19:19 |
| fungi | sdague: yep, that's back in my list as well. i saw it failed so i reverified and have it ready to repromote | 19:19 |
| *** gokrokve has joined #openstack-infra | 19:19 | |
| fungi | sdague: and 66364 which gets the requirements integration jobs passing again | 19:19 |
| fungi | jog0: and it's repeatable, several times in a row. trying one without uca as a quick sanity check now | 19:20 |
| jog0 | cool | 19:21 |
| *** emagana has quit IRC | 19:22 | |
| *** gokrokve has quit IRC | 19:23 | |
| *** miqui has joined #openstack-infra | 19:23 | |
| *** herndon has joined #openstack-infra | 19:23 | |
| mordred | fungi: we should implement a "reorder-on-next-reset" command :) | 19:24 |
| *** _ruhe is now known as ruhe | 19:24 | |
| fungi | mordred: agreed, because i missed the last one while i was busy trying to fix other stuff | 19:24 |
| *** sarob has quit IRC | 19:25 | |
| *** sarob has joined #openstack-infra | 19:25 | |
| jog0 | fungi: can you figure out what version of libvirt your using? | 19:26 |
| fungi | jog0: sure, just a sec and i'll let you know what it's pulling in | 19:28 |
| fungi | waiting on this next vm to finish launching so i can check iy | 19:28 |
| fungi | it | 19:28 |
| *** marun has quit IRC | 19:28 | |
| *** nati_uen_ has joined #openstack-infra | 19:28 | |
| *** marun has joined #openstack-infra | 19:28 | |
| *** dstanek has joined #openstack-infra | 19:28 | |
| jog0 | sweet | 19:29 |
| *** sarob has quit IRC | 19:30 | |
| *** gokrokve has joined #openstack-infra | 19:31 | |
| *** nati_uen_ has quit IRC | 19:31 | |
| *** nati_ueno has quit IRC | 19:31 | |
| *** nati_ueno has joined #openstack-infra | 19:32 | |
| *** gokrokve has quit IRC | 19:34 | |
| *** gokrokve has joined #openstack-infra | 19:34 | |
| fungi | jog0: oh, wait, right i was building that last one without uca to confirm i'm not doing something stupid in my launch script. i'll have to check on the next uca one | 19:34 |
| *** mgagne has quit IRC | 19:35 | |
| *** mgagne has joined #openstack-infra | 19:35 | |
| *** sarob has joined #openstack-infra | 19:38 | |
| *** eharney has joined #openstack-infra | 19:39 | |
| *** gokrokve has quit IRC | 19:39 | |
| *** coolsvap is now known as coolsvap_away | 19:40 | |
| *** jaypipes has joined #openstack-infra | 19:41 | |
| mgagne | how many vcpus are there per bare-precise nodes? | 19:42 |
| fungi | mgagne: 4 i believe (i'd have to check the flavors across all providers/regions though) | 19:43 |
| mgagne | fungi: ok, more than 1 =) | 19:43 |
| fungi | yes | 19:43 |
| mgagne | fungi: enough for me :P | 19:43 |
| *** thuc has quit IRC | 19:43 | |
| *** sarob has quit IRC | 19:48 | |
| *** marun has quit IRC | 19:48 | |
| *** marun has joined #openstack-infra | 19:48 | |
| *** sarob has joined #openstack-infra | 19:48 | |
| *** thuc has joined #openstack-infra | 19:49 | |
| fungi | reset! | 19:49 |
| *** thuc has quit IRC | 19:50 | |
| *** thuc has joined #openstack-infra | 19:50 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Fixed scp reference to unit test output. https://review.openstack.org/66918 | 19:51 |
| *** thuc has quit IRC | 19:51 | |
| *** thuc has joined #openstack-infra | 19:51 | |
| *** thuc has quit IRC | 19:52 | |
| *** sdake has quit IRC | 19:52 | |
| *** thuc has joined #openstack-infra | 19:52 | |
| *** sarob has quit IRC | 19:52 | |
| *** thuc has quit IRC | 19:53 | |
| *** thuc has joined #openstack-infra | 19:54 | |
| sdague | fungi: another reset | 19:55 |
| fungi | sdague: yep, the promote's been running for a few minutes now waiting on zuul to finish recalculating everything | 19:56 |
| fungi | but thanks for the heads up | 19:56 |
| openstackgerrit | Julien Danjou proposed a change to openstack-infra/devstack-gate: Remove configure_tempest.sh https://review.openstack.org/66922 | 19:56 |
| *** vipul is now known as vipul-away | 19:57 | |
| *** vipul-away is now known as vipul | 19:57 | |
| *** thedodd has joined #openstack-infra | 20:00 | |
| *** blamar has joined #openstack-infra | 20:02 | |
| *** sarob has joined #openstack-infra | 20:03 | |
| *** krotscheck is now known as _krotscheck_lunc | 20:04 | |
| *** gokrokve has joined #openstack-infra | 20:05 | |
| fungi | sdague: markmcclain: the requested stability/debugging fixes are showing at the top now | 20:05 |
| markmcclain | fungi: thanks | 20:06 |
| *** gothicmindfood_ has quit IRC | 20:06 | |
| sdague | fungi: thanks | 20:06 |
| *** smarcet has left #openstack-infra | 20:07 | |
| *** hashar has joined #openstack-infra | 20:07 | |
| *** jasondotstar has joined #openstack-infra | 20:10 | |
| *** gothicmindfood has joined #openstack-infra | 20:11 | |
| jog0 | fungi: http://paste.openstack.org/show/61300/ | 20:11 |
| jog0 | is that our network or gerrit | 20:11 |
| *** vipul is now known as vipul-away | 20:13 | |
| *** denis_makogon_ has joined #openstack-infra | 20:13 | |
| lifeless | jog0: your network I think | 20:13 |
| lifeless | jog0: I just pushed ok | 20:13 |
| anteaya | for reference salv-orlando is hitting the same Write failed: Broken pipe issue | 20:13 |
| *** denis_makogon_ is now known as denis_makogon | 20:14 | |
| fungi | jog0: i found my issue. i'd "optimized" my test script in a not very great way. rerunning now to test with uca again | 20:14 |
| fungi | jog0: i have a suspicion that's your network, but i'll test | 20:15 |
| lifeless | and SpamapS just saw a gerrit error too | 20:15 |
| SpamapS | The following command failed with exit code 128 "git fetch gerrit refs/changes/96/58596/6" | 20:15 |
| SpamapS | from gerritssh://clint-fewbar@review.openstack.org:29418/openstack/tripleo-heat-templates.git (fetch) | 20:15 |
| *** salv-orlando has quit IRC | 20:16 | |
| *** salv-orlando has joined #openstack-infra | 20:16 | |
| jog0 | fungi: those ar the best kind of optimizations | 20:17 |
| fungi | if it is a recurrent problem it may be something intermittent. i'm able to do ssh api things to gerrit's 29418/tcp interface just fine right now | 20:17 |
| fungi | i'll see if the extremely useless gerrit logs can shed any light | 20:17 |
| *** thomasem has quit IRC | 20:18 | |
| *** prad has quit IRC | 20:19 | |
| *** derekh has joined #openstack-infra | 20:19 | |
| jog0 | its re-occuring | 20:21 |
| jog0 | I can't get through | 20:21 |
| salv-orlando | Meh I have tried to change my key as well but to no avail | 20:21 |
| fungi | jog0: can you 'ssh -p 29418 review.openstack.org gerrit ls-projects' for example? | 20:21 |
| fungi | i'm going to guess something between you and the gerrit server us fiddling with your socket | 20:22 |
| fungi | er, is | 20:22 |
| jog0 | fungi: that command works for me | 20:23 |
| fungi | huh | 20:23 |
| *** marun has quit IRC | 20:23 | |
| *** marun has joined #openstack-infra | 20:23 | |
| openstackgerrit | Ilya Kharin proposed a change to openstack-infra/config: Additional jobs for python-rallyclient https://review.openstack.org/66929 | 20:23 |
| *** prad has joined #openstack-infra | 20:23 | |
| fungi | jog0: what's the "gerrit" fetch remote listed by 'git remote -v'? | 20:24 |
| jog0 | ssh://jogo@review.openstack.org:29418/openstack-dev/devstack.git | 20:24 |
| jog0 | FWIW this worked 5 minutes ago | 20:24 |
| fungi | very weird | 20:24 |
| salv-orlando | I have this error: "Agent admitted failure to sign using the key." | 20:24 |
| salv-orlando | both with git-review and ssh -p 29418 review.openstack.org | 20:24 |
| *** nati_uen_ has joined #openstack-infra | 20:26 | |
| *** nati_uen_ has quit IRC | 20:26 | |
| sdague | yeh, git review definitely boned from here | 20:27 |
| *** nati_uen_ has joined #openstack-infra | 20:27 | |
| fungi | so weird. i just updated from a similarly-formed gerrit remote url, pulled changes from the git service on it, et cetera | 20:28 |
| *** ruhe is now known as _ruhe | 20:28 | |
| sdague | fungi: did you try a push? | 20:29 |
| anteaya | fwiw lifeless and SpamapS are not in the same room as jog0 and salv-orlando, jog0 & salv-orlando: at code sprint, lifeless & SpamapS: not at code sprint | 20:29 |
| fungi | not yet. i need something to push | 20:29 |
| *** nati_ueno has quit IRC | 20:29 | |
| fungi | all my works in progress which i would push keep getting interrupted by researching random broken stuff | 20:29 |
| sdague | I'm pushing my tree to my linode | 20:29 |
| fungi | i'll just rebase something | 20:29 |
| SpamapS | right I'm at home | 20:29 |
| sdague | it worked from my linode | 20:30 |
| sdague | hmmmm | 20:30 |
| fungi | though the error SpamapS reported also looked different (possibly related, but maybe not) | 20:30 |
| sdague | fungi: from my linode I'm going on ipv6, let me find an external ipv4 link | 20:31 |
| *** sarob_ has joined #openstack-infra | 20:32 | |
| SpamapS | fatal: Couldn't find remote ref refs/changes/96/58596/7 | 20:32 |
| fungi | sdague: yeah, i'm ipv6 through to review.o.o as well | 20:32 |
| *** sarob_ has quit IRC | 20:32 | |
| *** sarob_ has joined #openstack-infra | 20:33 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Simplify devstack-logs publisher https://review.openstack.org/64938 | 20:33 |
| fungi | that ^ seemed to work | 20:34 |
| fungi | SpamapS: was your error while trying to push or fetch? | 20:34 |
| *** yassine has quit IRC | 20:35 | |
| *** sarob has quit IRC | 20:35 | |
| SpamapS | fungi: git review -d 58596 on tripleo-heat-templates | 20:35 |
| *** sarob_ has quit IRC | 20:35 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wip: test don't merge https://review.openstack.org/66932 | 20:35 |
| *** sarob has joined #openstack-infra | 20:35 | |
| sdague | yeh, so it's something about here | 20:35 |
| *** yassine has joined #openstack-infra | 20:35 | |
| *** yassine has quit IRC | 20:36 | |
| fungi | SpamapS: that's definitely repeatable for me | 20:37 |
| SpamapS | fungi: ah so something else maybe? | 20:39 |
| * SpamapS wants that breaking stuff trophy back ;) | 20:39 | |
| sdague | so my current suspicion here is that we triggered the university firewall | 20:39 |
| *** coolsvap_away has quit IRC | 20:39 | |
| sdague | and they are just killing the big gerrit streams because they just think it's bittorrent | 20:40 |
| *** chrisstreeter_ has joined #openstack-infra | 20:40 | |
| fungi | yeah, seems like gerrit lacks that ref for some reason. i'll hunt it down on the server... possible something related to connectivity caused lifeless's upload of the new patchset on 58596 to get into gerrit's webui but fail to add the corresponding ref to its git repo... i'll have to go digging there | 20:40 |
| *** sarob has quit IRC | 20:40 | |
| *** mrda has joined #openstack-infra | 20:40 | |
| fungi | but also possible something else unrelated broke on gerrit | 20:40 |
| * fungi sighs | 20:41 | |
| sdague | fungi: so from us, there is probably nothign you can do to help, it's all our side | 20:41 |
| fungi | sdague: that's sounding likely | 20:41 |
| hashar | poor zuul is driving me mad from time to time :D | 20:42 |
| hashar | luckily I got patch for you guys! | 20:42 |
| *** DinaBelova has quit IRC | 20:42 | |
| fungi | sdague: you're probably stuck forwarding gerrit ssh api connections through a tunnel/proxy on your linode or something | 20:42 |
| fungi | hashar: great! what was the bug? | 20:43 |
| sdague | fungi: yeh | 20:43 |
| fungi | (i mean, not great that you lost time dealing with a bug, but great that you have a patch) | 20:43 |
| hashar | fungi: zuul-server would keep trying to run when a wrong layout is given and it is in non daemon mode ( -d ) :D | 20:43 |
| *** _krotscheck_lunc is now known as krotscheck | 20:43 | |
| *** salv-orlando has quit IRC | 20:43 | |
| hashar | fungi: the configuration exception ends up being ignored for some reason. That is a basic change really | 20:43 |
| hashar | I got a nastier one which prevent me from running tests because it can't bind a socket on port 0 | 20:44 |
| fungi | hashar: ooh, yeah we don't often run it with -d except when testing things, afaik, so would likely not have spotted that | 20:44 |
| hashar | fungi: and that has driven me mad for a couple hours this afternoon. Just found out it was because of an invalid layout.yaml hehe | 20:44 |
| fungi | hashar: binding on port 0 on the other hand is usually socket library magic for picking an available ephemeral port | 20:45 |
| hashar | fungi: I guess I have an issue on mac os x. After all, I was missing select.poll() | 20:45 |
| fungi | hashar: what os are you trying to run the tests on | 20:45 |
| fungi | ? | 20:45 |
| fungi | aha! | 20:45 |
| hashar | Mac OS X 1.7 | 20:45 |
| fungi | yes, i believe that very well may not work on bsd/macos | 20:45 |
| hashar | had to install python 2.7.6 using homebrew which luckily provide a configuration switch to enable poll() ( brew install python --with-poll ) | 20:46 |
| hashar | if I was smart enough, I would write a wrapper to support both linux poll and BSD kqueue or something | 20:46 |
| fungi | it has nothing to do with being smart, just having an infinite amount of available time | 20:47 |
| hashar | lol | 20:47 |
| hashar | also lost time trying to figure out how to set the name of Zuul process when using ps / top | 20:47 |
| *** yolanda has quit IRC | 20:48 | |
| hashar | short answer: it is a linux only system call ( pcrtl() or something) | 20:48 |
| hashar | I eventually wanted to name the threads :D | 20:48 |
| *** _david_ has quit IRC | 20:48 | |
| fungi | hashar: renaming python processes is voodoo. there's no good way, and the available ways are not very portable | 20:48 |
| hashar | yeah figured that out the hard way :( | 20:48 |
| hashar | such a pity | 20:48 |
| *** vipul-away is now known as vipul | 20:49 | |
| fungi | it's a good lesson though. i tried it ages ago with one of my projects and eventually had it mostly working on the right platforms under the right circumstances, but got frustrated and ripped it all back out in the end | 20:49 |
| fungi | because, while it's convenient, it's still cosmetic and non-portable | 20:49 |
| *** salv-orlando has joined #openstack-infra | 20:50 | |
| *** salv-orlando has quit IRC | 20:50 | |
| *** salv-orlando has joined #openstack-infra | 20:50 | |
| fungi | pretty much any script running under an interpreter will exhibit the same behavior in the process list | 20:50 |
| *** mrmartin has quit IRC | 20:51 | |
| *** HenryG has joined #openstack-infra | 20:52 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Export a GEARDSERVER to tripleo-gate. https://review.openstack.org/66262 | 20:53 |
| lifeless | fungi: pleia2: derekh: ^ | 20:53 |
| lifeless | that eneds to land along with the new cloud definition. | 20:53 |
| lifeless | then our experimental queue should be usable | 20:53 |
| *** yolanda has joined #openstack-infra | 20:55 | |
| *** sandywalsh has quit IRC | 20:55 | |
| fungi | SpamapS: your error was pebcak. i'm taking back the trophy | 20:59 |
| *** harlowja_ is now known as harlowja_away | 20:59 | |
| fungi | SpamapS: openstack-infra/tripleo-ci != openstack/tripleo-heat-templates | 20:59 |
| derekh | lifeless: wrong IP I think https://review.openstack.org/#/c/66262/ | 20:59 |
| SpamapS | fungi: _ooohhhhhhhhh_ | 21:01 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Let us configure Gearman logging https://review.openstack.org/66938 | 21:01 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Log components starts in Zuul.Server https://review.openstack.org/66939 | 21:01 |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Die out on invalid layout https://review.openstack.org/66940 | 21:01 |
| fungi | SpamapS: if you're in the correct repository, 'git review -d 58596' should work just fine | 21:01 |
| fungi | (does for me anyway) | 21:01 |
| SpamapS | yeah | 21:02 |
| sdague | fungi: is there a way to find out who added an etherpad change? | 21:02 |
| fungi | sdague: depends on how you define "who" | 21:02 |
| sdague | someone in the room has a client that's adding one line at a time periodically to the etherpad | 21:02 |
| lifeless | derekh: thanks | 21:02 |
| sdague | which makes it super slow | 21:02 |
| lifeless | derekh: fixed (though you could yourself :P) | 21:03 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Export a GEARDSERVER to tripleo-gate. https://review.openstack.org/66262 | 21:03 |
| *** sandywalsh has joined #openstack-infra | 21:03 | |
| sdague | I want to know which client added this change - https://etherpad.openstack.org/p/montreal-code-sprint/timeslider#4856 | 21:03 |
| *** yolanda has quit IRC | 21:03 | |
| fungi | sdague: i can probably get it from the admin api in epl | 21:03 |
| sdague | cool | 21:04 |
| fungi | sdague: alternatively, bisect the crowd for the necessary interval, or get each of them to join a specific etherpad and see whose gets extra lines | 21:04 |
| fungi | but i'll see what i can find for you | 21:04 |
| sdague | thank you | 21:05 |
| *** alexpilotti has joined #openstack-infra | 21:07 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 21:08 | |
| fungi | https://jenkins03.openstack.org/job/gate-tempest-dsvm-full/2141/console had better succeed. there's lots of pending merges riding on it | 21:09 |
| fungi | we're already up to at least 8 changes lined up to merge if it does | 21:10 |
| *** CaptTofu has quit IRC | 21:10 | |
| *** jasondotstar has quit IRC | 21:10 | |
| *** dprince has quit IRC | 21:10 | |
| fungi | jog0: libvirt0 1.1.1-0ubuntu8~cloud2 | 21:11 |
| hashar | being curious, does anyone know how long it takes to build a Jenkins VM ? | 21:13 |
| *** nati_ueno has joined #openstack-infra | 21:15 | |
| *** salv-orlando has quit IRC | 21:15 | |
| *** salv-orlando_ has joined #openstack-infra | 21:15 | |
| *** sdake_ has joined #openstack-infra | 21:15 | |
| *** nati_ueno has quit IRC | 21:16 | |
| fungi | hashar: a vm configured to run as a jenkins master, or a vm acting as a slave? | 21:16 |
| hashar | the "one use only" VM | 21:16 |
| *** nati_ueno has joined #openstack-infra | 21:16 | |
| hashar | cause the node pool graph shows a lot of VM being in build | 21:16 |
| fungi | hashar: it varies by cloud provider. roughly 15 minutes on average i think | 21:17 |
| hashar | so I was wondering if that is because VM takes a long time to build or if it is simply that you consume a lot of them | 21:17 |
| fungi | but yes, it's both | 21:17 |
| hashar | ohh | 21:17 |
| hashar | isn't it all about copying and unpausing a preprovisionned VM ? | 21:17 |
| fungi | that's how we do it, but there's still post-clone things which have to be done to prep and add it to jenkins | 21:18 |
| * fungi looks at the timers in the nodepool list for ready nodes to get a more accurate guess | 21:18 | |
| *** derekh has quit IRC | 21:18 | |
| hashar | would it make sense to clone the repositories in the image? this way the preparation "just" have to fetch updates | 21:18 |
| *** nati_uen_ has quit IRC | 21:19 | |
| anteaya | the copying is the image we create every 24 hours with the dependencies, but we still need to clone the devstack and build | 21:19 |
| hashar | fungi: where is that node pool list ? :) | 21:20 |
| *** thomasem has joined #openstack-infra | 21:20 | |
| hashar | anteaya: can't you get devstack cloned as well ? (not sure what it is though) | 21:20 |
| fungi | hashar: i have to run 'sudo nodepool list' in a shell on nodepool.openstack.org | 21:20 |
| hashar | fungi: skip it :) probably not worth your time :D | 21:21 |
| fungi | hashar: anteaya: we do pre-clone all the repositories we use regularly before snapshotting it, including everything devstack wants | 21:21 |
| hashar | so probably not much to speed up :/ | 21:22 |
| hashar | and https://jenkins03.openstack.org/job/gate-tempest-dsvm-full/2141/console is probably going to fail | 21:22 |
| hashar | 2014-01-15 21:21:19.873 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 349: [[: stable/grizzly: division by 0 (error token is "grizzly") | 21:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Fixed scp reference to unit test output. https://review.openstack.org/66918 | 21:23 |
| *** senk has quit IRC | 21:23 | |
| fungi | hashar: actually, looks like the majority tend to stay in a building state for less than 10 minutes | 21:24 |
| *** chrisstreeter_ has quit IRC | 21:24 | |
| fungi | i think most of it has to do with how the providers shuffle virtual block devices on their respective backend implementations | 21:25 |
| openstackgerrit | A change was merged to openstack-infra/pypi-mirror: Whitelist external packages https://review.openstack.org/66364 | 21:25 |
| fungi | let the openstack/requirements floodgates swing open! ^ | 21:25 |
| fungi | also, that was part of a string of 9 changes which all merged in one shot out of the main integrated gate queue, so things are definitely moving through | 21:26 |
| sdague | fungi: sweet | 21:27 |
| dkranz | fungi: Is the gate ok now? Should we be pushing stuff through +A now? | 21:27 |
| fungi | dkranz: not sure what you mean | 21:28 |
| dkranz | fungi: We have a bunch ready to go to the gate queue if it is ok | 21:28 |
| fungi | dkranz: it's still 65 deep | 21:28 |
| dkranz | fungi: sdague said at the last qa meeting we should hold off | 21:28 |
| dkranz | fungi: I'm just looking for guidance | 21:29 |
| fungi | we've never quite recovered from the setback monday morning when nodepool stopped adding slaves for about 6-8 hours | 21:29 |
| sdague | dkranz: so it cleared out over the weekend, but we had nodepool issue that got us back | 21:29 |
| dkranz | fungi: ok, the problem is just that I don't know if 65 is ok or we should stop approving until it gets lower | 21:29 |
| fungi | but i don't see any reason not to approve more changes if you think they're safe ones and fit to be merged | 21:29 |
| dkranz | sdague: So should I stop +A for today? | 21:30 |
| dkranz | sdague: That's usually my first stop when reviewing | 21:30 |
| fungi | i mean, the other projects certainly don't seem to be holding back | 21:30 |
| dkranz | fungi: :) | 21:30 |
| sdague | dkranz: yeh, I wouldn't bother holding back. The gate is deep enough that we're having to promote the neutron fixes anyway | 21:30 |
| dkranz | sdague: ok, thanks. So be it. | 21:31 |
| dkranz | sdague: Glad to hear there are neutron fixes! | 21:31 |
| fungi | hashar: yeah, watching the stats i'm collecting out of the nodepool list results, most nodes seem to be transitioning from building to ready at around 10-15 minutes | 21:31 |
| fungi | which matches my recollection | 21:32 |
| fungi | we also probably have numbers in graphite.openstack.org for that, but don't quote me | 21:32 |
| openstackgerrit | David Caro proposed a change to openstack-infra/jenkins-job-builder: Added config options to not overwrite jobs desc https://review.openstack.org/52080 | 21:33 |
| hashar | fungi: gate-tempest-dsvm-full still has some nasty shell error :/ | 21:33 |
| sdague | hashar: you have link ? | 21:34 |
| hashar | fungi: and about vm building, I wonder whether it could be speeded up somehow. | 21:34 |
| hashar | sdague: https://jenkins03.openstack.org/job/gate-tempest-dsvm-full/2141/console | 21:34 |
| fungi | hashar: yes, https://review.openstack.org/64938 fixes some i think | 21:34 |
| hashar | 2014-01-15 21:21:19.873 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 349: [[: stable/grizzly: division by 0 (error token is "grizzly") | 21:34 |
| fungi | hashar: oh, there's another already in the gate for that one | 21:34 |
| *** gothicmindfood has quit IRC | 21:34 | |
| * fungi finds | 21:34 | |
| hashar | looking at the shell, someone is comparing strings using -ne which is arithmetic comparaison | 21:34 |
| sdague | yeh, that bounced off the gate | 21:35 |
| hashar | did not prevent that job from being successful though | 21:35 |
| fungi | hashar: that's https://review.openstack.org/63934 | 21:35 |
| sdague | oh, yeh, dkranz we need to update the whitelist | 21:36 |
| dkranz | sdague: For ? | 21:36 |
| *** boris-42 has quit IRC | 21:36 | |
| sdague | dkranz: to start enforcing again | 21:36 |
| sdague | http://logs.openstack.org/34/63934/3/gate/gate-tempest-dsvm-full/28c2e9a/console.html | 21:36 |
| dkranz | sdague: Oh, the ceilo team is working on fixing the bug now | 21:36 |
| fungi | right, i remember now. the hope was that the projects would fix their new non-whitelisted errors so that would be able to merge | 21:36 |
| fungi | that | 21:36 |
| dkranz | sdague: If they don't fix it by tomorrow I will change the whitelist | 21:37 |
| dkranz | sdague: This was just discussed in their meeting 20min ago | 21:37 |
| sdague | there are nova issues as well | 21:37 |
| dkranz | sdague: Ugh | 21:37 |
| dkranz | sdague: Do you have a list? | 21:37 |
| *** denis_makogon has quit IRC | 21:37 | |
| sdague | dkranz: ttp://logs.openstack.org/34/63934/3/gate/gate-tempest-dsvm-full/28c2e9a/console.html | 21:37 |
| sdague | look at the fails on - https://review.openstack.org/#/c/63934/ | 21:38 |
| dkranz | sdague: ok | 21:39 |
| dkranz | sdague: I will put them all in the whitelist. | 21:39 |
| dkranz | sdague: Do we want to fast-track throught the gate? | 21:39 |
| *** gothicmindfood has joined #openstack-infra | 21:40 | |
| *** markwash has joined #openstack-infra | 21:40 | |
| sdague | dkranz: once it passes, sure | 21:40 |
| dkranz | sdague: ok, I'll get on that now | 21:40 |
| sdague | thanks | 21:40 |
| *** boris-42 has joined #openstack-infra | 21:40 | |
| markwash | hi folks, curious if anyone is seeing issues with psutils not available on pypi in the versions we want? | 21:41 |
| hashar | off, have a good day! | 21:42 |
| markwash | nm actually | 21:44 |
| mordred | markwash: I blame jaypipes | 21:44 |
| fungi | markwash: yes, you're probably using pip 1.5.x or virtualenv 1.11.x | 21:44 |
| *** jhesketh_ has joined #openstack-infra | 21:44 | |
| fungi | markwash: psutil is not hosted on pypi, merely linked from it | 21:44 |
| markwash | yeah, I saw that on -dev from while back | 21:44 |
| mordred | we should get psutil fixed - or stop using it | 21:44 |
| *** miqui has quit IRC | 21:45 | |
| fungi | it's one of the four requirements we have external to pypi at the moment (along with netifaces and a couple others) | 21:45 |
| *** dcramer_ has quit IRC | 21:45 | |
| fungi | SpamapS was able to convince the netaddr devs to fix themselves, so proof that asking them helps at least some of the time | 21:46 |
| *** nati_ueno has quit IRC | 21:46 | |
| *** sarob has joined #openstack-infra | 21:46 | |
| fungi | have a good night, hashar! | 21:46 |
| fungi | sdague: bad news on the eplite api... the only calls which take a pad revno are for retrieving full (unannotated) content | 21:47 |
| fungi | might be possible to hunt it down from the database, but that would take some doing | 21:47 |
| *** salv-orlando_ has quit IRC | 21:49 | |
| *** salv-orlando has joined #openstack-infra | 21:49 | |
| *** nati_ueno has joined #openstack-infra | 21:50 | |
| *** sarob has quit IRC | 21:50 | |
| *** senk has joined #openstack-infra | 21:52 | |
| *** luqas has joined #openstack-infra | 21:54 | |
| *** salv-orlando has quit IRC | 21:54 | |
| *** salv-orlando has joined #openstack-infra | 21:55 | |
| *** nati_ueno has quit IRC | 21:55 | |
| *** nati_ueno has joined #openstack-infra | 21:56 | |
| *** prad has quit IRC | 21:59 | |
| *** thuc has quit IRC | 22:00 | |
| *** thuc has joined #openstack-infra | 22:01 | |
| *** thuc has quit IRC | 22:01 | |
| sdague | fungi: don't worry about it | 22:01 |
| *** thuc has joined #openstack-infra | 22:01 | |
| *** nati_uen_ has joined #openstack-infra | 22:03 | |
| *** senk has quit IRC | 22:04 | |
| sdague | fungi: are we at full quota on the providers/ | 22:04 |
| sdague | ? | 22:04 |
| fungi | sdague: i believe so, yes | 22:05 |
| sdague | ok, bummer | 22:05 |
| dkranz | sdague: I pushed this ten minutes ago https://review.openstack.org/#/c/66954/ | 22:05 |
| anteaya | fungi: thanks | 22:05 |
| fungi | sdague: we're churning ~450 nodes up and down already, as it is | 22:05 |
| sdague | fungi: yeh, it just means the check queue is taking 2hrs | 22:05 |
| *** slong has joined #openstack-infra | 22:06 | |
| dstufft | mordred: what would it take to convince pbr to switch to defining dependencies inside of setup.cfg instead of piggybacking off of requirements.txt | 22:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Catch exceptions from nova flavor-list calls https://review.openstack.org/66958 | 22:06 |
| dkranz | sdague: Huh. So I will not be able to babysit the whitelist change. I can approve it later if no one else does. | 22:06 |
| dkranz | sdague: Who can push it to the top of the gate queue? | 22:07 |
| *** nati_ueno has quit IRC | 22:07 | |
| jgriffith | fungi: mordred sdague wanted to bounce some ideas on third party testing if any of you have time or are interested? | 22:07 |
| *** sdake_ is now known as sdake | 22:08 | |
| fungi | jgriffith: have you seen https://review.openstack.org/63478 ? | 22:09 |
| fungi | dkranz: i can | 22:09 |
| jgriffith | I had not | 22:09 |
| fungi | jgriffith: we're trying to get a little more policy around the existing set of recommendations | 22:09 |
| fungi | jgriffith: but your bouncy ideas may be in no way related to that. what ideas? | 22:09 |
| *** gothicmindfood has quit IRC | 22:10 | |
| jgriffith | fungi: well.. so my idea was that I just want to have periodic checks against my companies storage/cinder backend | 22:10 |
| *** EmilienM has quit IRC | 22:10 | |
| *** pabelanger has quit IRC | 22:10 | |
| jgriffith | fungi: so something like nightly maybe? Pull a random cinder commit and run the tempest test suite with my backend configured | 22:10 |
| *** pabelanger has joined #openstack-infra | 22:10 | |
| jgriffith | non-voting obviously | 22:10 |
| jgriffith | but to be able to auto test and publicly record results | 22:11 |
| fungi | jgriffith: right, i've thought about that in the past as well. problem is we currently don't have great reporting mechanisms for periodic jobs. we send e-mail to interested parties | 22:11 |
| jgriffith | fungi: hmm | 22:11 |
| fungi | but there's almost certainly a better way | 22:11 |
| fungi | most of our testing feedback is centered around the code review system | 22:11 |
| jgriffith | fungi: Understood | 22:11 |
| *** hashar has quit IRC | 22:12 | |
| jgriffith | fungi: What about a "3'rd party tests" dashboard/website? | 22:12 |
| fungi | what we probably need is a portal/dashboard where periodic tasks can report for a quick overview and then descend from into detailed results | 22:12 |
| fungi | heh | 22:12 |
| jgriffith | fungi: :) | 22:12 |
| *** EmilienM has joined #openstack-infra | 22:12 | |
| jgriffith | at least I'm not talking crazy :) | 22:12 |
| fungi | i'm not sure it's just a third-party thing either. we could really use that for stable bitrot checks, coverage reports, anything | 22:12 |
| jgriffith | fungi: interesting | 22:13 |
| jgriffith | fungi: so I don't have a huge number of resources, but I'd be VERY interested in contributing them to help with other things | 22:13 |
| *** mfer has quit IRC | 22:14 | |
| jgriffith | fungi: so do you think it's sane to start down the path of duplicating the devstack/tempest setup off of a gerrit trigger, run tempest and dump results to a web-server somewhere. Then later we can work together to do something like the dashboard we just talked about? | 22:14 |
| jgriffith | fungi: I guess from openstack/ci to start it's irrelevant | 22:14 |
| dkranz | fungi: Thanks. How late will you be around? | 22:15 |
| fungi | jgriffith: i hate to say "bring it up on the infra/dev ml" mainly because i've been blind to e-mail for nearly a week. it's been too busy to keep up. but i really think it's of sufficient utility we should get some discussion going with more people | 22:15 |
| jgriffith | fungi: haha.. that may be best | 22:15 |
| jgriffith | fungi: just wanted to make sure I was even on the right track and not missing an obvious effort that was already taking place :) | 22:15 |
| *** jasondotstar has joined #openstack-infra | 22:16 | |
| fungi | dkranz: probably until 0300 utc, depending on whether my girlfriend makes me put down the laptop | 22:16 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: DO NOT MERGE: test out eatmydata https://review.openstack.org/66960 | 22:16 |
| jog0 | sdague: ^ | 22:16 |
| dkranz | fungi: :) Hopefully that is enough time to get it through the check queue | 22:16 |
| jog0 | jaypipes: ^^ | 22:16 |
| *** jasondotstar has quit IRC | 22:16 | |
| fungi | jgriffith: yeah i think it's a great idea. we could potentially get a lot of benefit from some place to publish (or at least aggregate) that sort of stuff | 22:18 |
| *** branen has quit IRC | 22:18 | |
| *** asalkeld has joined #openstack-infra | 22:18 | |
| fungi | jog0: we did some experiments in the past with eatmydata and the results weren't especially exciting, but it's worth trying again for sure | 22:19 |
| asalkeld | hi, any advice on updating my foundation email address? | 22:19 |
| fungi | i think we determined that we were more cpu bound than i/o bound | 22:19 |
| jog0 | fungi: o_O well we will know in an hour or two | 22:19 |
| mordred | jgriffith: ++ to dashboard | 22:19 |
| fungi | asalkeld: log in at https://www.openstack.org/profile/ | 22:19 |
| *** thomasem has quit IRC | 22:19 | |
| asalkeld | cool thanks fungi | 22:20 |
| fungi | asalkeld: you're welcome | 22:20 |
| mordred | jgriffith: also, perhaps (don't kill me krotscheck) that's a thing that shoudl get reported to storyboard | 22:20 |
| *** gothicmindfood has joined #openstack-infra | 22:20 | |
| * mordred runs away before he's asked to explain how | 22:20 | |
| fungi | mordred: ooh! | 22:20 |
| fungi | mordred: step 3... profit! | 22:20 |
| mordred | fungi: yah. right? | 22:20 |
| mordred | fungi: but seriously - it's probably a terrible idea | 22:21 |
| mordred | we should potentially either have a separate thing - or maybe use statsd | 22:21 |
| mordred | or I'll just let jgriffith figure it out | 22:21 |
| fungi | since jeblair's not here to say it, i'm going to channel him... "let's reserve speculation on new storyboard features until there are basic storyboard features working" ;) | 22:21 |
| *** dkranz has quit IRC | 22:21 | |
| mordred | fungi: nicely done | 22:21 |
| anteaya | I don't think having to deal with the group of folk deploying the 3rd party testing systems right now is a good group to sent to storyboard atm | 22:22 |
| marun | fungi: ping | 22:22 |
| anteaya | thinking mostly of the neutron hoard | 22:22 |
| *** branen has joined #openstack-infra | 22:23 | |
| krotscheck | _shrug_. Once the API is up you can report anything to storyboard that you want to. How about helping out? ;) | 22:23 |
| mordred | fungi: the storyboard-webclient unittest work | 22:24 |
| marun | fungi: I'm working on adding a new functional testing job for neutron to the config repo, and am confused about how to ensure the job will run with the necessary sudo privileges. | 22:24 |
| marun | fungi: I also want to run unstack.sh in post_test_hook and am not sure of the path on a jenkins slave. | 22:25 |
| *** jaypipes has quit IRC | 22:25 | |
| fungi | marun: if it runs on a devstack-precise node type, the jenkins, stack and tempest users all have full sudo privs, i believe | 22:26 |
| anteaya | krotscheck: are you aware of what I am doing right now? | 22:26 |
| fungi | marun: devstack ends up installed in /opt/stack/new | 22:26 |
| krotscheck | anteaya: Helping? :) | 22:27 |
| marun | fungi: great on the second, still confused on the first | 22:27 |
| marun | fungi: how do I ensure the job runs as a specific user? | 22:27 |
| fungi | marun: the job starts as the jenkins user | 22:27 |
| fungi | marun: if it's being run from within the context of devstack-gate (as hook functions) then that sets up the stack and tempest accounts as needed | 22:28 |
| marun | fungi: ah, ok. and are there restrictions on what can be executed as sudo? | 22:28 |
| fungi | marun: none | 22:28 |
| marun | fungi: perfect. | 22:28 |
| marun | fungi: danke! | 22:28 |
| jgriffith | mordred: ok... great... so you're going to imlement story-board for this and we're all set | 22:29 |
| jgriffith | mordred: :) | 22:29 |
| fungi | marun: that's if the node type is devstack-precise. if it's bare-precise or precise then jenkins cannot sudo (the "node" parameter to the job definition) | 22:29 |
| fungi | i'm very bandwidth-constrained at the moment... in the kitchen trying not to burn dinner | 22:29 |
| marun | fungi: ah, sorry. thanks for the help! | 22:30 |
| jgriffith | mordred: fungi Ok.. so I sent a brief ramble on the topic to the ML I'll see what comes back and start diggin in to how to proceed here | 22:30 |
| jgriffith | mordred: fungi thanks for the help/tips | 22:30 |
| *** CaptTofu has joined #openstack-infra | 22:30 | |
| *** salv-orlando has quit IRC | 22:30 | |
| *** salv-orlando has joined #openstack-infra | 22:30 | |
| fungi | jgriffith: expect some delay on infra ml conversations as we're still running on skeleton crew for core reviewers/root admins | 22:31 |
| *** jcoufal has quit IRC | 22:31 | |
| jgriffith | fungi: ha! I sent it to all of dev, so that should keep me busy for a month | 22:31 |
| anteaya | fungi: skeleton crew meaning yourself, eh? | 22:32 |
| fungi | anteaya: we have mordred! | 22:32 |
| anteaya | he he he | 22:32 |
| anteaya | how you making out there, boss? | 22:32 |
| fungi | jgriffith: great idea | 22:32 |
| pballand | mordred: I dropped offline for a bit - any update on a pointer to how to extend setup.py? | 22:33 |
| *** fbo_away has quit IRC | 22:33 | |
| *** nati_uen_ has quit IRC | 22:33 | |
| *** obondarev_ has quit IRC | 22:33 | |
| openstackgerrit | Maru Newby proposed a change to openstack-infra/config: Add an experimental functional job for neutron. https://review.openstack.org/66967 | 22:33 |
| *** nati_ueno has joined #openstack-infra | 22:34 | |
| *** nati_ueno has quit IRC | 22:35 | |
| *** nati_ueno has joined #openstack-infra | 22:36 | |
| *** marun has quit IRC | 22:36 | |
| *** asalkeld has left #openstack-infra | 22:36 | |
| *** HenryG has quit IRC | 22:42 | |
| zaro | fungi: hey.. just noticed something. the account agreement tables pre and post upgrade look identical. | 22:42 |
| fungi | zaro: yes, they're not used any longer | 22:43 |
| zaro | fungi: which leads me to think that we don't need to insert users to that table at all. | 22:43 |
| fungi | zaro: we don't. we need to take the list of users from that table and add them to the new group | 22:43 |
| zaro | fungi: didn't you make a suggestion to populate them? | 22:43 |
| *** derekh has joined #openstack-infra | 22:43 | |
| zaro | fungi: ohh i see. | 22:44 |
| fungi | we need a select query on the table to find all users who have already agreed to the icla, so we can put them in the group you're creating for the icla | 22:44 |
| *** salv-orlando has quit IRC | 22:44 | |
| zaro | fungi: yah, got it. | 22:44 |
| fungi | otherwise they'll all have to agree again before they can push new changes, and that would be very disruptive | 22:44 |
| *** salv-orlando has joined #openstack-infra | 22:44 | |
| *** marun has joined #openstack-infra | 22:45 | |
| fungi | zaro: i believe the icla is in there as cla_id=2 and we want the account_id fields where status='V' | 22:46 |
| fungi | select account_id from account_agreements where cla_id=2 and status='V'; | 22:47 |
| *** mfink has quit IRC | 22:47 | |
| fungi | or something along those lines | 22:47 |
| *** luqas has quit IRC | 22:49 | |
| *** salv-orlando has quit IRC | 22:50 | |
| *** salv-orlando has joined #openstack-infra | 22:50 | |
| *** salv-orlando has quit IRC | 22:50 | |
| *** hogepodge has quit IRC | 22:52 | |
| *** nati_ueno has quit IRC | 22:56 | |
| *** yamahata has quit IRC | 22:57 | |
| *** jamielennox|away is now known as jamielennox | 22:57 | |
| *** ^d is now known as ^demon|away | 22:58 | |
| *** flaper87 is now known as flaper87|afk | 22:59 | |
| *** herndon has quit IRC | 22:59 | |
| *** rossella_s has quit IRC | 23:01 | |
| notmyname | what's the current jenkins job timeout value? still 120 minutes? | 23:02 |
| mordred | pballand: sorry - no, I got distracted - I'll get it back to you | 23:02 |
| *** harlowja_away is now known as harlowja_ | 23:02 | |
| mordred | fungi: wait - we have me? | 23:02 |
| mordred | that's terrifying | 23:02 |
| *** jhesketh__ has joined #openstack-infra | 23:03 | |
| *** marun has quit IRC | 23:04 | |
| *** nati_ueno has joined #openstack-infra | 23:05 | |
| *** thedodd has quit IRC | 23:05 | |
| *** nati_ueno has quit IRC | 23:06 | |
| *** CaptTofu has quit IRC | 23:06 | |
| fungi | mordred: seems that way | 23:07 |
| *** jgillies has quit IRC | 23:07 | |
| fungi | notmyname: it varies widely by job. which specific job? | 23:07 |
| notmyname | the one at the top of the queue. looks like it just finished successfully (1h21min) | 23:08 |
| *** jgillies has joined #openstack-infra | 23:08 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Enable Firewall as a Service plugin https://review.openstack.org/65822 | 23:08 |
| *** jhesketh_ has quit IRC | 23:08 | |
| fungi | notmyname: i believe the timeout on the tempest-postgres job was temporarily upped to 90 minutes while qa people try to whittle it back down | 23:09 |
| notmyname | the -fll job | 23:09 |
| notmyname | *full | 23:09 |
| * fungi checks | 23:09 | |
| *** herndon has joined #openstack-infra | 23:10 | |
| *** burt has quit IRC | 23:11 | |
| notmyname | no worries, it's cleared out now. I was looking for a timeframe to send my reverify bug | 23:11 |
| *** portante_ has joined #openstack-infra | 23:12 | |
| *** mtaylor has joined #openstack-infra | 23:12 | |
| *** katyafervent_ has joined #openstack-infra | 23:12 | |
| *** gyee__ has joined #openstack-infra | 23:13 | |
| *** jgriffit2 has joined #openstack-infra | 23:14 | |
| *** wayneseguin has joined #openstack-infra | 23:14 | |
| notmyname | unless there is a better way, you can use logstash to see how many times something gets rechecked in the gate queue. search for the patch, the changeset, a log mesage only printed once, and a job only run once. | 23:14 |
| notmyname | eg http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9jaGFuZ2U6XCI2MTU4NVwiIEFORCBidWlsZF9wYXRjaHNldDpcIjE1XCIgQU5EIG1lc3NhZ2U6XCJbU0NQXSBDb3B5aW5nIGNvbnNvbGUgbG9nLlwiIEFORCBidWlsZF9uYW1lOlwiZ2F0ZS1zd2lmdC1wZXA4XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjE3MjgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIiLCJzdGFtcCI6MTM4OTgyNzA3MDA3MX0= | 23:14 |
| notmyname | so you can see that that patch was approved about 24 hours ago and has been rechecked 6 times (so far). it was just hit by bug 1259003 and will now enter the queue again at the bottom | 23:15 |
| uvirtbot | Launchpad bug 1259003 in grenade "Failed to create a volume during volume exercise" [Undecided,New] https://launchpad.net/bugs/1259003 | 23:15 |
| *** pballand has quit IRC | 23:16 | |
| *** pasquier-s_ has joined #openstack-infra | 23:16 | |
| *** mrda has quit IRC | 23:16 | |
| notmyname | (frustration note, this makes it very hard to get stuff done during a sprint--such as what's happening yesterday and today) | 23:16 |
| *** SlickN1k has joined #openstack-infra | 23:16 | |
| *** portante has quit IRC | 23:17 | |
| *** thuc has quit IRC | 23:17 | |
| *** mordred has quit IRC | 23:17 | |
| *** katyafervent has quit IRC | 23:17 | |
| *** gyee_ has quit IRC | 23:17 | |
| *** jgriffith has quit IRC | 23:17 | |
| *** wayneeseguin has quit IRC | 23:17 | |
| *** jlk has quit IRC | 23:17 | |
| *** jlk has joined #openstack-infra | 23:17 | |
| *** weshay has quit IRC | 23:17 | |
| *** pasquier-s has quit IRC | 23:17 | |
| *** pvo has quit IRC | 23:17 | |
| *** SlickNik has quit IRC | 23:17 | |
| *** pabelanger has quit IRC | 23:17 | |
| *** kpepple has quit IRC | 23:17 | |
| *** weshay has joined #openstack-infra | 23:17 | |
| *** pvo has joined #openstack-infra | 23:17 | |
| *** pabelanger has joined #openstack-infra | 23:17 | |
| *** dragondm has quit IRC | 23:17 | |
| *** jlk has quit IRC | 23:17 | |
| *** jlk has joined #openstack-infra | 23:17 | |
| *** wayneseguin is now known as wayneeseguin | 23:17 | |
| *** SlickN1k is now known as SlickNik | 23:17 | |
| *** kpepple has joined #openstack-infra | 23:17 | |
| *** pabelanger has quit IRC | 23:17 | |
| *** pabelanger has joined #openstack-infra | 23:17 | |
| *** dragondm has joined #openstack-infra | 23:20 | |
| *** dstanek has quit IRC | 23:21 | |
| fungi | i completely sympathize. i went through much of the same getting the fixes to the requirements integration jobs merged. keeping it all running can be equally frustrating at times ;) | 23:21 |
| notmyname | how long does it take for jenkins to pick up a reverify? https://review.openstack.org/#/c/61585/ has been waiting over 10 minutes | 23:22 |
| *** obondarev_ has joined #openstack-infra | 23:22 | |
| *** senk has joined #openstack-infra | 23:22 | |
| fungi | if it's in the middle of recalculating the current state of the gate merge sequence, it currently queues up new events and results until that's complete. jeblair has some ideas on how to decentralize that and spread it over a cluster of workers so that it's near-instantaneoius | 23:23 |
| fungi | near-instantaneous | 23:23 |
| notmyname | ah ok. I guess waiting an extra 15 minutes or so doesn't really matter when it's going to take another day or two to land ;-/ | 23:24 |
| notmyname | just making sure it didn't get skipped or that I did something wrong | 23:24 |
| *** derekh has quit IRC | 23:24 | |
| fungi | looking at the shape of the gate change diagram on the status page and the queued events/results numbers at the top, i expect it to appear there in a few more minutes | 23:24 |
| *** salv-orlando has joined #openstack-infra | 23:25 | |
| notmyname | another question along those lines...this slow gate seems like it's going to cause some issues for the next mid-cycle releases (next week, right?). I'm somewhat concerned about getting stuff in for the swift RC by the end of this week | 23:25 |
| notmyname | errr..and actual question: are we going to have a major problem next week? | 23:26 |
| fungi | i'm *hoping* that the slowness (which seems to be due primarily to so many changes being approved each day) means that we'll see less of a scramble to merge more stuff then | 23:26 |
| notmyname | well, I'd argue that running through a gate queue 5+ times for a single patch also contributes to slowness ;-) | 23:27 |
| fungi | yes, definitely. though we saw much slower merge rates back when the failure rate was higher | 23:28 |
| fungi | i watched 9 changes in a shot all merge earlier this afternoon | 23:28 |
| *** dims has quit IRC | 23:28 | |
| notmyname | I'll completely agree that people are working on getting gate-blocking bugs fixed. and that's good. but since we can't write code without bugs, it seems like the underlying assumptions don't really fit with the current scale of the zuul queue | 23:30 |
| *** jgrimm has quit IRC | 23:30 | |
| *** torgomatic has joined #openstack-infra | 23:32 | |
| notmyname | ah, there they go. re-enqueued | 23:33 |
| *** markmcclain has quit IRC | 23:34 | |
| *** obondarev_ has quit IRC | 23:34 | |
| *** markmcclain has joined #openstack-infra | 23:34 | |
| jhesketh__ | Howdy | 23:34 |
| *** obondarev_ has joined #openstack-infra | 23:35 | |
| fungi | howdy jhesketh__ | 23:35 |
| notmyname | fungi: (just thinking "aloud" here) would it be possible to have a separate zuul check queue for each repo/project/program? what problems would that cause? | 23:36 |
| notmyname | as long as we have more test nodes than queues, then it should mean faster merges for each repo | 23:36 |
| fungi | notmyname: it's definitely possible (we already do that). it's the integrated release projects which have common integration tests that gate on each other's functionality (test that a change to project a doesn't make project b unusable) | 23:37 |
| fungi | oh, check not gate | 23:37 |
| notmyname | the zuul queue seems orthogonal to the tests run | 23:37 |
| notmyname | nono, the gate queue is what I mean | 23:37 |
| *** gothicmindfood has quit IRC | 23:37 | |
| notmyname | (the zuul status page has "change queue") | 23:38 |
| notmyname | sorry, I meant change queue, not check queue. specifically I care about gates, not checks | 23:38 |
| fungi | well, we have things like upgrade jobs too, so for example are you adding a nova to stable/havana which works with glance but after upgrading to master it doesn't... things like that | 23:38 |
| fungi | er, adding a nova CHANGE to stable/havana | 23:39 |
| *** obondarev_ has quit IRC | 23:39 | |
| *** markmcclain has quit IRC | 23:39 | |
| notmyname | how often does that happen? it's "stable", right there in the name. it shouldn't be the integration that dictates the day-to-day process for all the other repos | 23:40 |
| fungi | i think maybe what you're getting at is, changes in the gate are tested against other changes ahead of them also in the gate, so does a change to nova allow a later change to glance to work (whereas if they merged in the other order they would not), or does a change to glance work with master nova but there's a nova change ahead of it which changes functionality it depends upon | 23:41 |
| clarkb | hello | 23:41 |
| fungi | clarkb! | 23:41 |
| *** thuc has joined #openstack-infra | 23:41 | |
| notmyname | fungi: ya, sortof. I'm saying that nova changes queue up with other nova changes and glance changes queue up with other glance changes, etc | 23:41 |
| notmyname | guessing at close-to-realistic numbers, if we average about 8 patches landing per hour, then if we have enough contributors or projects to submit more than 8 patches per hour, then the gate will perpetually get longer | 23:42 |
| fungi | notmyname: effectively the hole it's attempting to close is the "my patch passed all tests and merged. if it broke something because another patch was merging at the same time to a related project, it's not my fault and someone else can go hunt it down" | 23:42 |
| *** sarob has joined #openstack-infra | 23:42 | |
| clarkb | fungi: hi there. Anything you want me to catch up on really quickly? | 23:42 |
| clarkb | fungi: I am severely jet lagged, but am happy to poke at things if there is an urgent need somewhere | 23:43 |
| fungi | clarkb: i have puppet disabled on the nodepool server and a really, really awful patch proposed for catching novaclient exceptions in it | 23:43 |
| clarkb | otehrwise I will take my time catching up on mail | 23:43 |
| notmyname | fungi: if you merge a patch in one project that breaks and API that another project is depending on, then wouldn't other clients be affected too? point being, it's a big problem to break APIs | 23:43 |
| *** praneshp has quit IRC | 23:43 | |
| *** dims has joined #openstack-infra | 23:44 | |
| clarkb | fungi: so nodepool did die about 30 hours ago? | 23:44 |
| clarkb | sorry I was just getting on planes iirc as that happened | 23:44 |
| fungi | notmyname: the current gate design, at least the theory behind it, is to force troubleshooting of broken changes back on the developers proposing them, rather than hunt them down after they've merged. i agree in practice it only seems to work reliably for deterministic errors | 23:45 |
| *** rcleere has quit IRC | 23:45 | |
| fungi | ideal api abstraction design principles aside | 23:46 |
| clarkb | fungi: notmyname: fwiw it seems to catc hthe non deterministic ones pretty well too, but we have a bad habit of forcing those changes through anyways | 23:46 |
| clarkb | then bad stuff happens | 23:46 |
| fungi | i also agree that some software design principles would make the occurrence of those sorts of interaction issues less frequent. we lack ideal software in that regard | 23:47 |
| *** sarob has quit IRC | 23:47 | |
| notmyname | well, of course we do :-) we can't assume any of us are writing perfect code. but we can't simply "try harder" to reduce bugs. | 23:48 |
| fungi | and it's possible that we're creating a large rube-goldberg mechanism which enables developers to not properly abstract interfaces by testing whether it works anyway. i won't deny that | 23:48 |
| *** sarob has joined #openstack-infra | 23:48 | |
| *** sarob has quit IRC | 23:49 | |
| *** gothicmindfood has joined #openstack-infra | 23:49 | |
| fungi | quite often, if you set a bar, people will just aim for that as "good enough" | 23:50 |
| *** thuc has quit IRC | 23:50 | |
| *** sarob has joined #openstack-infra | 23:50 | |
| fungi | i'm hoping our core reviewers are focusing on design questions to help mitigate much of that | 23:51 |
| fungi | but it does pretty much boil down to "gating is slow because of all the ways we've implemented to try and avoid merging more bugs" | 23:53 |
| *** yamahata has joined #openstack-infra | 23:53 | |
| fungi | the complexity grows to the point where it takes less and less to gum up the works | 23:53 |
| clarkb | fungi: are the client exception catching changes for nodepool there to aid in debugging whatever caused it to fail earlier? | 23:54 |
| fungi | clarkb: no, to avoid it getting in the way of nodepool continuing to do its job | 23:55 |
| clarkb | gotcha | 23:55 |
| fungi | clarkb: basically, the poc cloud for tripleo was taken offline. it came time for new images to get built, so nodepoold started asking all the providers for image lists | 23:55 |
| notmyname | fungi: what I'm very concerned about is how people are or will be working around the openstack gate because of frustration with it | 23:56 |
| clarkb | notmyname: you mean fixing bugs? (sorry) | 23:56 |
| fungi | clarkb: at that point, nodepoold went into a short-circuit in its main loop, and didn't launch any new nodes | 23:56 |
| fungi | just kept trying over and over to build a flavor list | 23:57 |
| *** mrodden has quit IRC | 23:57 | |
| clarkb | silly nodepool | 23:57 |
| *** jcoufal has joined #openstack-infra | 23:57 | |
| notmyname | clarkb: ya, bugs need to be fixed. nobody is arguing against that | 23:58 |
| *** oubiwann has quit IRC | 23:58 | |
| fungi | notmyname: i do think that we may find there's an upper-bound on complexity beyond which we can't scale to merge new changes faster than a particular rate. i'm not sure we've reached it yet since we still have some ideas for continuing to speed it up and clean it up | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!