| *** banix has quit IRC | 00:01 | |
| *** mrodden has quit IRC | 00:03 | |
| *** mrodden has joined #openstack-infra | 00:04 | |
| *** mrodden has quit IRC | 00:04 | |
| fungi | jeblair: that's what i'm wondering. in 2.4 self-owned is when owner_group_id=group_id | 00:04 |
|---|---|---|
| *** rwsu has joined #openstack-infra | 00:07 | |
| *** dcramer_ has joined #openstack-infra | 00:07 | |
| *** oubiwann has quit IRC | 00:08 | |
| *** sarob has joined #openstack-infra | 00:09 | |
| *** gokrokve has quit IRC | 00:13 | |
| *** mdenny has quit IRC | 00:13 | |
| *** gokrokve has joined #openstack-infra | 00:13 | |
| *** fallenpegasus2 has joined #openstack-infra | 00:16 | |
| *** fallenpegasus has quit IRC | 00:18 | |
| *** gokrokve has quit IRC | 00:18 | |
| *** yamahata has joined #openstack-infra | 00:20 | |
| *** dcramer_ has quit IRC | 00:20 | |
| bknudson | is "No distributions at all found for netifaces>=0.5" ? I was having this problem a few days ago with netaddr. | 00:21 |
| bknudson | is ... a known problem | 00:21 |
| bknudson | I'm running into it with keystone now, but gate doesn't seem to have it. | 00:22 |
| bknudson | doesn't seem to have a problem with it | 00:22 |
| fungi | bknudson: i have patches up to fix that in swift... don't recall if they've merged yet | 00:24 |
| bknudson | fungi: https://review.openstack.org/#/c/65023/ is merged in swift | 00:24 |
| fungi | though i guess my patches just fix it for running tests under tox | 00:24 |
| *** AaronGr is now known as AaronGr_afk | 00:25 | |
| bknudson | is the gate using older virtualenv or whatever the prob was? | 00:25 |
| fungi | bknudson: yes, we have the gate set to use virtualenv 1.10.1 which bundles pip 1.4.1 | 00:25 |
| bknudson | ok, that makes sense. | 00:26 |
| bknudson | thanks! | 00:26 |
| fungi | bknudson: what you're seeing crops up with virtualenv 1.11 which bundles pip 1.5 | 00:26 |
| fungi | bknudson: the *fix* for that is to badger the netifaces devs to start uploading releases to pypi | 00:26 |
| fungi | (or, more work probably, to stop using netifaces and replace it with something else) | 00:26 |
| bknudson | I don't know how keystone picked this up. | 00:27 |
| bknudson | picked up the requirement on netifaces | 00:27 |
| bknudson | ok, test-requirements.txt has it. | 00:27 |
| fungi | bknudson: i saw someone updated the shared bug and etherpad with a note about proposed patches to keystone for it | 00:27 |
| fungi | there may already be an outstanding change which wants your review ;) | 00:28 |
| *** fallenpegasus2 has quit IRC | 00:28 | |
| *** dkliban is now known as dkliban_afk | 00:28 | |
| morganfainberg | bknudson, https://github.com/openstack/keystone/commit/f3af6918639013a5219a6192d5777e9437e20afa | 00:29 |
| bknudson | morganfainberg: that was only 2 years ago. | 00:29 |
| morganfainberg | well, jenkins uploaded a fix because global requirements changed irt | 00:30 |
| bknudson | maybe I should run tests more often than every 2 years. | 00:30 |
| morganfainberg | https://github.com/openstack/keystone/commit/7ae43f53911e010df11b8ee557fef2b5bff2e77a is the last time that line changed | 00:30 |
| morganfainberg | ~2 months ago | 00:30 |
| morganfainberg | pypi seems to claim 0.8 is "state of the art" | 00:30 |
| morganfainberg | https://pypi.python.org/pypi/netifaces | 00:31 |
| * morganfainberg is confused. | 00:31 | |
| morganfainberg | fwiw, test-requirements.txt:netifaces>=0.5 is the only place i see netifaces in the code | 00:31 |
| morganfainberg | might be safe to remove it? | 00:31 |
| bknudson | morganfainberg: that would be nice! | 00:31 |
| *** rfolco has joined #openstack-infra | 00:32 | |
| *** vipuls is now known as vipuls-away | 00:33 | |
| *** vipuls-away is now known as vipuls | 00:33 | |
| *** dstanek has joined #openstack-infra | 00:33 | |
| fungi | yes, so sounds like the "stop using it" solution might actually be the easiest path for keystone ;) | 00:33 |
| * morganfainberg ducks back over to -dev. | 00:33 | |
| morganfainberg | fungi, hopefully! | 00:33 |
| fungi | bknudson: propose a change which just removes that line, and see what breaks | 00:33 |
| bknudson | fungi: I'll try it. | 00:33 |
| *** dpyzhov has quit IRC | 00:34 | |
| *** dpyzhov has joined #openstack-infra | 00:36 | |
| *** ^d is now known as ^demon|away | 00:37 | |
| *** ^demon|away has quit IRC | 00:39 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Capture libvirtd logs https://review.openstack.org/65833 | 00:47 |
| *** praneshp has quit IRC | 00:48 | |
| *** yamahata has quit IRC | 00:50 | |
| *** praneshp has joined #openstack-infra | 00:51 | |
| *** fallenpegasus has joined #openstack-infra | 00:54 | |
| *** rnirmal has quit IRC | 00:59 | |
| *** dcramer_ has joined #openstack-infra | 01:03 | |
| *** dpyzhov has quit IRC | 01:04 | |
| *** sarob_ has joined #openstack-infra | 01:06 | |
| *** sarob has quit IRC | 01:09 | |
| *** sarob_ has quit IRC | 01:11 | |
| *** jasondotstar has joined #openstack-infra | 01:16 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move devstack hooks from infra config to solum repo https://review.openstack.org/65414 | 01:20 |
| *** rfolco has quit IRC | 01:20 | |
| *** jasondotstar has quit IRC | 01:22 | |
| *** jasondotstar has joined #openstack-infra | 01:23 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1261182 https://review.openstack.org/65768 | 01:23 |
| *** dstanek has quit IRC | 01:23 | |
| *** fallenpegasus has quit IRC | 01:24 | |
| *** mozawa has joined #openstack-infra | 01:24 | |
| *** fifieldt has joined #openstack-infra | 01:24 | |
| *** fallenpegasus has joined #openstack-infra | 01:25 | |
| *** thuc has quit IRC | 01:26 | |
| *** thuc has joined #openstack-infra | 01:26 | |
| *** thuc has quit IRC | 01:30 | |
| *** mriedem has joined #openstack-infra | 01:32 | |
| *** nosnos has joined #openstack-infra | 01:33 | |
| *** yaguang has joined #openstack-infra | 01:33 | |
| jog0 | 2014-01-10 01:32:52.982 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 349: [[: stable/grizzly: division by 0 (error token is "grizzly") | 01:36 |
| *** alexpilotti has quit IRC | 01:38 | |
| *** alexpilotti has joined #openstack-infra | 01:39 | |
| jog0 | looks like a bash error | 01:39 |
| lifeless | [[ is eval mode | 01:39 |
| lifeless | probably wants [ stable/grizzly ] | 01:39 |
| lifeless | or better [ -z "stable/grizzly" ] | 01:40 |
| jog0 | if [[ "$LOCALRC_BRANCH" -ne "stable/grizzly" ]] && \ | 01:41 |
| jog0 | is the current logic | 01:41 |
| fungi | jog0: sdague had a change to fix it, but the fix was incorrect. i need to see if there was a new patchset | 01:43 |
| dims | google search says -ne is for arithmetic only.... | 01:43 |
| lifeless | dims: correct | 01:43 |
| jog0 | fungi: ahh | 01:43 |
| lifeless | != for strings | 01:43 |
| fungi | jog0: should still be in review | 01:43 |
| fungi | turns out there is more there than just the trivial -ne vs != operator error | 01:43 |
| jog0 | https://review.openstack.org/#/c/63934/ | 01:43 |
| sdague | yeh so we should push that through | 01:45 |
| fungi | sdague: how can i tell whether it's working? with -x off and check_logs.py not echoing when it runs, i'm having trouble confirming from the console logs | 01:48 |
| sdague | so probably ping dkranz tomorrow | 01:50 |
| sdague | honestly, my brain is sort of liquid at the moment | 01:50 |
| fungi | oh, wait, i should have been looking at grenade logs anyway i think | 01:50 |
| jog0 | so the gate is moving again btw | 01:50 |
| fungi | sdague: makes sense | 01:50 |
| sdague | jog0: it's still at 71 changes | 01:51 |
| jog0 | sdague: yup | 01:51 |
| jog0 | but we just merged a bunch | 01:51 |
| jog0 | sdague: hopefully tomorrow AM things will be better | 01:53 |
| sdague | jog0: we can always hope | 01:53 |
| jog0 | I'me hopeful because the gate failure rates graph on http://status.openstack.org/elastic-recheck/ looks good | 01:53 |
| jog0 | err better | 01:53 |
| morganfainberg | lifeless, if you have a handy link, is there a way to know the worker-id in-side the test framework ... or a way to identify ... say a worker-specific temp dir? | 01:54 |
| morganfainberg | lifeless, if so, i think i can fix keystone parallel testing in a couple lines of code | 01:54 |
| lifeless | morganfainberg: you could use your pid | 01:54 |
| morganfainberg | lifeless, ah, valid idea | 01:54 |
| morganfainberg | lifeless, cool. | 01:54 |
| lifeless | wouldn't scale to multiple machines sharing e.g. one ldap server, but single machine parallelism should be sufficient for | 01:55 |
| *** reed has quit IRC | 01:55 | |
| sdague | lifeless: is there a more correct way to handle this - https://github.com/openstack-infra/elastic-recheck/blob/master/tox.ini#L16 | 01:56 |
| sdague | because the problem with that declaration is that I can't run tox -epy27 one_test | 01:57 |
| sdague | because it runs them all regardless | 01:57 |
| lifeless | right, because it's unionin it with tests.unit as a filter | 02:00 |
| lifeless | just run testr by hand ? | 02:00 |
| lifeless | .tox/py27/bin/testr run one_test | 02:00 |
| lifeless | or .tox/py27/bin/activate && testr run one_test ? | 02:01 |
| sdague | hmmm... ok, no way to fix it so the passthrough works as expected? | 02:03 |
| *** fallenpegasus has quit IRC | 02:03 | |
| *** sandywalsh has quit IRC | 02:03 | |
| *** fallenpegasus has joined #openstack-infra | 02:04 | |
| *** fallenpegasus2 has joined #openstack-infra | 02:07 | |
| *** rockyg has quit IRC | 02:07 | |
| *** fallenpegasus has quit IRC | 02:07 | |
| sdague | well, it's a new year, so I'll harass you once more for moving testrepository to git somewhere :) | 02:10 |
| sdague | ok, time to really call it a night | 02:10 |
| *** coolsvap has quit IRC | 02:14 | |
| *** sarob has joined #openstack-infra | 02:17 | |
| *** fallenpegasus2 has quit IRC | 02:18 | |
| *** melwitt has quit IRC | 02:18 | |
| *** fallenpegasus has joined #openstack-infra | 02:18 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move nova/savanna to only use bare-precise nodes https://review.openstack.org/65807 | 02:18 |
| *** sandywalsh has joined #openstack-infra | 02:19 | |
| *** fallenpegasus has quit IRC | 02:21 | |
| *** fallenpegasus2 has joined #openstack-infra | 02:21 | |
| *** sarob has quit IRC | 02:21 | |
| *** julim has joined #openstack-infra | 02:23 | |
| *** fallenpegasus2 has quit IRC | 02:23 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Enable tempest/ironic gate tests https://review.openstack.org/65845 | 02:25 |
| *** fallenpegasus has joined #openstack-infra | 02:27 | |
| *** gokrokve has joined #openstack-infra | 02:27 | |
| *** praneshp has quit IRC | 02:28 | |
| morganfainberg | is there a way to run a final cleanup when a testr worker exits? | 02:31 |
| *** dstanek has joined #openstack-infra | 02:31 | |
| morganfainberg | i guess i could just do the cleanup ahead of time. | 02:32 |
| morganfainberg | e.g. at the start of a test run | 02:32 |
| morganfainberg | hm. | 02:32 |
| *** dstanek has quit IRC | 02:36 | |
| *** dcramer_ has quit IRC | 02:42 | |
| *** jasondotstar has quit IRC | 02:49 | |
| *** jerryz has quit IRC | 02:51 | |
| *** dkranz has joined #openstack-infra | 02:53 | |
| *** ^d has joined #openstack-infra | 02:55 | |
| *** fallenpegasus has quit IRC | 02:56 | |
| *** dcramer_ has joined #openstack-infra | 02:56 | |
| *** banix has joined #openstack-infra | 02:57 | |
| *** fallenpegasus has joined #openstack-infra | 02:57 | |
| lifeless | jeblair: so - thought - calculate the expected % chance of the next job passing along each branch, then schedule the next highest % job. | 03:02 |
| lifeless | jeblair: would need a chance metric that isn't just constant, or you fill the tree equally | 03:03 |
| lifeless | jeblair: stats from different project success rates might do that | 03:04 |
| *** sarob has joined #openstack-infra | 03:04 | |
| *** gokrokve has quit IRC | 03:04 | |
| lifeless | jeblair: or a bias towards success; or an offset to probability based on branch forkign | 03:04 |
| *** gokrokve has joined #openstack-infra | 03:05 | |
| *** gokrokve has quit IRC | 03:09 | |
| *** mestery has quit IRC | 03:16 | |
| *** nati_ueno has quit IRC | 03:17 | |
| *** ^d has quit IRC | 03:18 | |
| *** miqui has quit IRC | 03:22 | |
| *** CaptTofu has quit IRC | 03:23 | |
| *** CaptTofu has joined #openstack-infra | 03:24 | |
| *** fallenpegasus has quit IRC | 03:26 | |
| *** pcrews has quit IRC | 03:27 | |
| *** CaptTofu has quit IRC | 03:28 | |
| *** harlowja is now known as harlowja_away | 03:31 | |
| *** Shrews has quit IRC | 03:39 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/publications: Update periodic pipeline slide and url https://review.openstack.org/65854 | 03:39 |
| *** jishaom has joined #openstack-infra | 03:40 | |
| clarkb | my laptop doesnt want to charge now :/ gonna need to sort that out during luncj | 03:42 |
| jeblair | clarkb: you can use mine if needed. also, it's possible the computer at the lectern may be usable since yours is on the net. | 03:43 |
| clarkb | cool thanks | 03:44 |
| jeblair | clarkb: fungi: i think we should go ahead and revoke mordred's gerrit access too, just in case | 03:44 |
| *** gokrokve has joined #openstack-infra | 03:45 | |
| *** chandankumar has joined #openstack-infra | 03:45 | |
| fungi | jeblair: you mean in addition to what i mentioned in the e-mail thread? (i already removed his ssh keys) remove him from the administrators group too? | 03:45 |
| clarkb | ++ | 03:46 |
| fungi | and core infra groups, et cetera i guess | 03:46 |
| clarkb | until he can invalidate lp auth stuff | 03:46 |
| jeblair | fungi: yes; in case his lp creds are compromised | 03:46 |
| fungi | in case of cached lp openid | 03:46 |
| fungi | doing now | 03:46 |
| jeblair | fungi: i have a copy of his group membership list; i can remove it with a sql command | 03:47 |
| jeblair | fungi: shall i? | 03:48 |
| fungi | jeblair: oh, i guess do that. it'll be faster | 03:48 |
| jeblair | done | 03:49 |
| fungi | that'll cover corner cases | 03:49 |
| *** ryanpetrello has joined #openstack-infra | 03:49 | |
| fungi | should we worry about his access to lp resources? jenkins master webuis? wiki? | 03:50 |
| StevenK | fungi: If you're concerned, his LP account can be suspended (IE: no login for him) until he's back and fine | 03:51 |
| *** nicedice has joined #openstack-infra | 03:51 | |
| fungi | github organization controls also spring to mind, and pypi | 03:52 |
| jeblair | fungi: good point. without knowing the details, we should probably hit those too. | 03:53 |
| fungi | not authenticated through lp, but similar cached login credential risks | 03:53 |
| jeblair | fungi: i'll remove him from openstack-ci-admins and openstack-administrators on lp, which should take care of jenkins and lp itself | 03:54 |
| jeblair | fungi: do you want to take care of github? | 03:54 |
| jeblair | pypi going to be hard since no groups | 03:54 |
| fungi | jeblair: doing github now | 03:55 |
| *** CaptTofu has joined #openstack-infra | 03:55 | |
| jeblair | ugh: openstack-ci-admins is owned by mordred and not openstack-administrators... | 03:57 |
| jeblair | i guess we should ask an lp question to get that changed | 03:57 |
| *** CaptTofu has quit IRC | 03:58 | |
| *** CaptTofu has joined #openstack-infra | 03:58 | |
| *** dcramer_ has quit IRC | 04:01 | |
| fungi | i've removed emonty from openstack-infra, openstack, openstack-dev, openstack-ci and stackforge organization owner teams on github | 04:04 |
| fungi | i need a once-a-month refresher on why their interface annoys me. that was my booster shot | 04:04 |
| jeblair | ha | 04:04 |
| jeblair | oh good grief | 04:05 |
| StevenK | jeblair: You can probably ask a question, and then ping wgrant in #launchpad and point him at it | 04:05 |
| jeblair | there are pypi packages owned _only_ by him | 04:05 |
| jeblair | StevenK: thx | 04:05 |
| clarkb | jeblair: ping dstufft? | 04:05 |
| clarkb | dstufft: please to rescue us from emonty | 04:06 |
| *** ryanpetrello has quit IRC | 04:06 | |
| jeblair | clarkb: sdirectory on pypi | 04:06 |
| jeblair | gah sdirector even | 04:06 |
| fungi | sdirector | 04:07 |
| fungi | yeah | 04:07 |
| jeblair | dstufft: mordred had his laptop stolen, and we're not sure about the extend of possible compromise... | 04:10 |
| jeblair | dstufft: would it be possible for you to give us a list of pypi packages owned by 'sdirector' | 04:10 |
| jeblair | dstufft: and then we ask you to add 'openstackci' as an owner and remove sdirector for a subset of that list? | 04:11 |
| jeblair | dstufft: it would be much appreciated | 04:11 |
| clarkb | jeblair person sitting next to lana behind us may help on pypi front too | 04:15 |
| clarkb | forget his name | 04:15 |
| *** mriedem has quit IRC | 04:17 | |
| fifieldt | poor monty | 04:22 |
| *** AaronGr_afk is now known as AaronGr_Zzz | 04:22 | |
| *** praneshp has joined #openstack-infra | 04:27 | |
| *** yamahata has joined #openstack-infra | 04:39 | |
| *** CaptTofu has quit IRC | 04:47 | |
| *** CaptTofu_ has joined #openstack-infra | 04:47 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Add new javamelody-plugin project https://review.openstack.org/65162 | 04:47 |
| *** sarob has quit IRC | 04:50 | |
| *** sarob has joined #openstack-infra | 04:51 | |
| *** jishaom has quit IRC | 04:51 | |
| *** banix has quit IRC | 04:52 | |
| *** sarob has quit IRC | 04:56 | |
| openstackgerrit | Alexei Kornienko proposed a change to openstack/requirements: Added argcomplete requirement https://review.openstack.org/65863 | 04:58 |
| *** jroovers has joined #openstack-infra | 04:59 | |
| *** amotoki has joined #openstack-infra | 05:00 | |
| *** coolsvap has joined #openstack-infra | 05:02 | |
| *** praneshp_ has joined #openstack-infra | 05:04 | |
| *** praneshp has quit IRC | 05:05 | |
| *** praneshp_ is now known as praneshp | 05:05 | |
| *** masayukig has quit IRC | 05:14 | |
| *** afazekas has joined #openstack-infra | 05:17 | |
| *** rcarrillocruz has quit IRC | 05:27 | |
| *** nicedice has quit IRC | 05:27 | |
| *** nati_ueno has joined #openstack-infra | 05:28 | |
| *** SergeyLukjanov has joined #openstack-infra | 05:34 | |
| *** ^d has joined #openstack-infra | 05:35 | |
| *** kraman1 has joined #openstack-infra | 05:36 | |
| devananda | jeblair: re: ironic check/gate patch, can we enable it (move it out of experimental pipe) in a non-voting way, ,in all three projects? | 05:36 |
| devananda | jeblair: my concern is if we only add it to ironic gate, then ironic's gate may be unknowingly broken bychanges in devstack or tempest | 05:37 |
| jeblair | devananda: right, but then if keystone breaks ironic, tempest can't merge patches | 05:38 |
| devananda | and we'll have no way to unblock our own gate | 05:38 |
| jeblair | devananda: yes, you'd have to propose a change to tempest or devstack in that case | 05:39 |
| devananda | jeblair: it seems like a terrible idea for this to remain in experimental pipie for devstack/tempest but be gating for ironic | 05:40 |
| devananda | devs in those projects wouldn't know when they break us | 05:40 |
| jeblair | devananda: let's discuss this another time | 05:40 |
| devananda | k | 05:40 |
| *** SergeyLukjanov has quit IRC | 05:41 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch horizon to use bare-precise nodes https://review.openstack.org/65736 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch ceilometer to use bare-precise nodes https://review.openstack.org/65737 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch swift to use bare-precise nodes https://review.openstack.org/65738 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch cinder to use bare-precise nodes https://review.openstack.org/65732 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch glance to use bare-precise nodes https://review.openstack.org/65733 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch keystone to use bare-precise nodes https://review.openstack.org/65734 | 05:57 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Switch heat to use bare-precise nodes https://review.openstack.org/65735 | 05:57 |
| *** ^d has quit IRC | 06:06 | |
| fifieldt | nice presentation | 06:06 |
| *** nicedice has joined #openstack-infra | 06:15 | |
| anteaya | fifieldt: where are you watching from? | 06:16 |
| *** dstanek has joined #openstack-infra | 06:16 | |
| fifieldt | taipei | 06:17 |
| anteaya | fifieldt: nice | 06:17 |
| anteaya | how is the stream quality? | 06:17 |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/config: Add a Loading paragraph onto zuul's status page https://review.openstack.org/65872 | 06:19 |
| *** HenryG has quit IRC | 06:22 | |
| fifieldt | very good | 06:24 |
| anteaya | great | 06:25 |
| anteaya | glad to have you along in spirit | 06:26 |
| *** gyee_ has quit IRC | 06:34 | |
| clarkb | jeblair: I think I neglected to mention you did the crm114 work :/ sorry | 06:36 |
| jeblair | clarkb: and the bug graphs. ;) it's ok; that's the nature of backstage work. :) | 06:41 |
| *** HenryG has joined #openstack-infra | 06:41 | |
| fifieldt | can I interest anyone in reviewing a one line change to jeepyb? https://review.openstack.org/#/c/65607/ :) | 06:42 |
| *** pblaho has joined #openstack-infra | 06:44 | |
| anteaya | fifieldt: +1 | 06:46 |
| fifieldt | cheers mate :) | 06:46 |
| anteaya | a comment, I don't usually see folks voting on their own patches | 06:47 |
| fifieldt | noted | 06:47 |
| anteaya | neutron keeps getting a new patchset from Jenkins before we can get this in: https://review.openstack.org/#/c/64786/ | 06:48 |
| anteaya | suggestions? | 06:48 |
| *** jcoufal has joined #openstack-infra | 06:54 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/jeepyb: Add entrypoint for welcome_message https://review.openstack.org/65607 | 06:58 |
| *** yolanda has joined #openstack-infra | 07:13 | |
| *** jamielennox is now known as jamielennox|away | 07:13 | |
| *** yamahata has quit IRC | 07:14 | |
| *** gokrokve has quit IRC | 07:16 | |
| *** gokrokve has joined #openstack-infra | 07:17 | |
| *** gokrokve has quit IRC | 07:21 | |
| dstufft | clarkb: jeblair sorry I was asleep | 07:41 |
| *** dpyzhov has joined #openstack-infra | 07:41 | |
| *** fbo_away is now known as fbo | 07:46 | |
| *** jasondotstar has joined #openstack-infra | 07:47 | |
| dstufft | clarkb: jeblair guess y'all are asleep now yourslef :( | 07:47 |
| *** gokrokve has joined #openstack-infra | 07:47 | |
| *** gokrokve_ has joined #openstack-infra | 07:49 | |
| *** flaper87|afk is now known as flaper87 | 07:51 | |
| *** gokrokve has quit IRC | 07:52 | |
| *** katyafervent has quit IRC | 07:52 | |
| *** gokrokve_ has quit IRC | 07:53 | |
| *** dstanek has quit IRC | 07:54 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 07:57 |
| clarkb | dstufft we are awake | 07:58 |
| dstufft | clarkb: oh cool | 07:59 |
| clarkb | help us dstufft you are our only hope | 07:59 |
| dstufft | clarkb: do you still need help? | 07:59 |
| clarkb | yes | 07:59 |
| dstufft | what do you need from me? | 07:59 |
| clarkb | can you get us a list of pypi packages owned by sdirector then we can give you a subset to take ownership of using the openstackci account | 08:00 |
| clarkb | mordreds laptop was stolen and we are taking precautions | 08:00 |
| dstufft | Is sdirector monty on pypi? | 08:00 |
| clarkb | currently on perth cell service so may be slow to respond | 08:01 |
| clarkb | yes sdirector is monty | 08:01 |
| dstufft | clarkb: https://gist.github.com/dstufft/0599520096124a94044d | 08:01 |
| clarkb | dstufft thanks we are finding wifi now so that we can edit that list | 08:02 |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 08:03 |
| *** denis_makogon has joined #openstack-infra | 08:04 | |
| *** rcarrillocruz has joined #openstack-infra | 08:08 | |
| *** ilyashakhat has joined #openstack-infra | 08:09 | |
| jeblair | dstufft: http://paste.openstack.org/show/60932/ | 08:12 |
| jeblair | dstufft: if you could add 'openstackci' as an owner for all of those and remove 'sdirector' that would be swell. thanks. | 08:12 |
| dstufft | clarkb jeblair I'm going to trust you on this given the nature of the problem. If I get yelled at or sdirector ends up not being monty I'm gonna be not a happy camper :D but yea I can do that | 08:13 |
| *** katyafervent has joined #openstack-infra | 08:15 | |
| jeblair | dstufft: thanks. i could gpg sign something if you want, but i don't know if i'm in your web of trust. | 08:15 |
| *** sergmelikyan has joined #openstack-infra | 08:16 | |
| *** luqas has joined #openstack-infra | 08:17 | |
| * katyafervent is now away: See you later! | 08:17 | |
| *** katyafervent is now known as katyafervent_awa | 08:17 | |
| *** alexpilotti has quit IRC | 08:17 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 08:18 |
| jeblair | katyafervent_awa: please disable your away message; thanks! | 08:22 |
| *** mancdaz_away is now known as mancdaz | 08:23 | |
| *** dizquierdo has joined #openstack-infra | 08:24 | |
| *** dstanek has joined #openstack-infra | 08:25 | |
| *** katyafervent_awa is now known as katyafervent | 08:27 | |
| dstufft | jeblair: clarkb Ok, here's all that's left https://gist.github.com/dstufft/e1cfdb197a8b4dbd1304 | 08:28 |
| *** yamahata has joined #openstack-infra | 08:29 | |
| *** persia__ is now known as persia | 08:30 | |
| clarkb | dstufft looks good thanks. | 08:31 |
| *** persia has quit IRC | 08:31 | |
| *** persia has joined #openstack-infra | 08:31 | |
| dstufft | clarkb: smack monty for not having his laptop encrypted :D | 08:31 |
| clarkb | we will | 08:32 |
| *** hashar has joined #openstack-infra | 08:33 | |
| dstufft | clarkb: let me know if there's anything else I can help with | 08:33 |
| * dstufft sheds his admin privs | 08:33 | |
| clarkb | I think thats it for now. big help owe you beer | 08:34 |
| *** mayu has joined #openstack-infra | 08:35 | |
| mayu | ping anteaya | 08:35 |
| *** nati_ueno has quit IRC | 08:39 | |
| mayu | is there a guide for donating test resource | 08:39 |
| *** hashar has quit IRC | 08:39 | |
| mayu | anyboby know how to donate test resource | 08:40 |
| *** dizquierdo has quit IRC | 08:43 | |
| *** hashar has joined #openstack-infra | 08:43 | |
| *** chandankumar_ has joined #openstack-infra | 08:48 | |
| *** CaptTofu_ has quit IRC | 08:48 | |
| *** CaptTofu has joined #openstack-infra | 08:49 | |
| *** rcarrillocruz1 has joined #openstack-infra | 08:52 | |
| *** mayu has quit IRC | 08:52 | |
| *** rcarrillocruz has quit IRC | 08:54 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Gather horizon/apache2 logs https://review.openstack.org/64490 | 08:59 |
| *** chandankumar_ has quit IRC | 08:59 | |
| kruskakli | anteaya: u there ? | 08:59 |
| *** luqas has quit IRC | 09:00 | |
| *** jpich has joined #openstack-infra | 09:05 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 09:05 |
| *** derekh has joined #openstack-infra | 09:06 | |
| *** mozawa has quit IRC | 09:15 | |
| *** gilliard has quit IRC | 09:19 | |
| *** bauzas has joined #openstack-infra | 09:21 | |
| *** yassine has joined #openstack-infra | 09:22 | |
| *** luqas has joined #openstack-infra | 09:23 | |
| *** yassine has quit IRC | 09:29 | |
| *** markmc has joined #openstack-infra | 09:32 | |
| *** _ruhe is now known as ruhe | 09:33 | |
| *** chandankumar_ has joined #openstack-infra | 09:40 | |
| *** gokrokve has joined #openstack-infra | 09:48 | |
| *** denis_makogon has quit IRC | 09:48 | |
| *** nosnos has quit IRC | 09:49 | |
| *** hashar has quit IRC | 09:51 | |
| *** chandankumar_ has quit IRC | 09:52 | |
| *** gokrokve has quit IRC | 09:53 | |
| *** yaguang has quit IRC | 09:54 | |
| *** dizquierdo has joined #openstack-infra | 09:56 | |
| *** CaptTofu has quit IRC | 09:57 | |
| *** nicedice has quit IRC | 10:01 | |
| *** nicedice has joined #openstack-infra | 10:02 | |
| *** yamahata has quit IRC | 10:05 | |
| *** nicedice has quit IRC | 10:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:10 | |
| *** nicedice has joined #openstack-infra | 10:10 | |
| EmilienM | Hello, I wonder why https://review.openstack.org/#/c/62733 has not been merged, and what could I do ? | 10:22 |
| *** ruhe is now known as _ruhe | 10:26 | |
| *** jorisroovers has joined #openstack-infra | 10:28 | |
| *** jroovers has quit IRC | 10:30 | |
| *** michchap_ has joined #openstack-infra | 10:31 | |
| *** michchap has quit IRC | 10:32 | |
| *** jorisroovers has quit IRC | 10:32 | |
| SergeyLukjanov | EmilienM, it looks like zuul was restarted in this time | 10:33 |
| SergeyLukjanov | when it was merged | 10:33 |
| EmilienM | SergeyLukjanov: yeah, what I thought. What could we do then? | 10:33 |
| SergeyLukjanov | EmilienM, you need to reverify it | 10:34 |
| SergeyLukjanov | but due to the fact that it's not a bug but just a zuul restart, I think that you can use "reverify bug 1" this time | 10:34 |
| SergeyLukjanov | or probably ask someone to approve it again, AFAIK it should help too | 10:35 |
| EmilienM | SergeyLukjanov: ok, thank you for your help | 10:35 |
| SergeyLukjanov | EmilienM, np | 10:35 |
| *** chandankumar_ has joined #openstack-infra | 10:37 | |
| *** lcestari has joined #openstack-infra | 10:38 | |
| *** dpyzhov has quit IRC | 10:44 | |
| *** bogdando has joined #openstack-infra | 10:44 | |
| EmilienM | SergeyLukjanov: it worked, thanks :-) | 10:45 |
| SergeyLukjanov | EmilienM, great ;) | 10:45 |
| *** gokrokve has joined #openstack-infra | 10:48 | |
| sergmelikyan | Guys, why jobs on Zuul are restarting? Verify job on https://review.openstack.org/61074 already around 1 day is stuck in queue | 10:50 |
| *** nicedice has quit IRC | 10:50 | |
| *** pblaho has quit IRC | 10:51 | |
| *** gokrokve has quit IRC | 10:53 | |
| *** _ruhe is now known as ruhe | 11:00 | |
| SergeyLukjanov | sergmelikyan, it was reverified about 5 hoursago | 11:01 |
| SergeyLukjanov | sergmelikyan, dsvm jobs are queued currently | 11:02 |
| SergeyLukjanov | sergmelikyan, you find it here - http://status.openstack.org/zuul/ | 11:02 |
| *** Bada has joined #openstack-infra | 11:03 | |
| *** ruhe is now known as ruhe_away | 11:04 | |
| *** ruhe_away is now known as ruhe | 11:04 | |
| *** dstanek has quit IRC | 11:05 | |
| sergmelikyan | SergeyLukjanov, thx! missed reverify comment (thought it was previous one, sent by me yesterday). | 11:06 |
| SergeyLukjanov | sergmelikyan, np | 11:07 |
| *** fbo is now known as fbo_away | 11:07 | |
| *** johnthetubaguy has joined #openstack-infra | 11:14 | |
| *** Ajaeger has joined #openstack-infra | 11:15 | |
| *** chandankumar_ has quit IRC | 11:18 | |
| *** luqas has quit IRC | 11:18 | |
| *** rfolco has joined #openstack-infra | 11:19 | |
| *** luqas has joined #openstack-infra | 11:21 | |
| *** andreaf has quit IRC | 11:26 | |
| *** andreaf has joined #openstack-infra | 11:26 | |
| *** dpyzhov has joined #openstack-infra | 11:29 | |
| *** ruhe is now known as _ruhe | 11:33 | |
| *** yamahata has joined #openstack-infra | 11:34 | |
| *** _ruhe is now known as ruhe | 11:45 | |
| *** jasondotstar has quit IRC | 11:45 | |
| *** ymr1 has joined #openstack-infra | 11:46 | |
| *** ymr1 has quit IRC | 11:48 | |
| *** gokrokve has joined #openstack-infra | 11:48 | |
| *** ymr1 has joined #openstack-infra | 11:50 | |
| *** gokrokve_ has joined #openstack-infra | 11:50 | |
| kruskakli | when I'm using tox based on master it want to install a requirement named 'psutil' which fails, thus I can't run my testcase, any ideas on what to do about it? | 11:53 |
| *** gokrokve has quit IRC | 11:53 | |
| kruskakli | in stable/havana this problem doesn't occur | 11:53 |
| kruskakli | based on neutron/master that is | 11:54 |
| *** gokrokve_ has quit IRC | 11:55 | |
| Ajaeger | kruskakli: is psutil on pypi? | 11:55 |
| Ajaeger | Seems to be - so it's not that bug I had in mind | 11:55 |
| kruskakli | Ajaeger: sorry I don't know anything about python and its tools | 11:57 |
| *** yamahata has quit IRC | 12:00 | |
| kruskakli | I run: tox -e py27 -- neutron.tests.unit.ml2.test_mecha nism_ncs | 12:00 |
| kruskakli | and it ends with: Could not find a version that satisfies the requirement psutil>=0.6.1,<1.0 (from -r /home/vagrant/neutron/requirements.txt (line 19)) | 12:01 |
| *** BobBallAway is now known as BobBall | 12:01 | |
| *** chandankumar_ has joined #openstack-infra | 12:07 | |
| zigo | mordred: There's now Debian official images in HP cloud! :) | 12:09 |
| zigo | mordred: Do you know who I should get in touch at Rackspace to make it happen there as well? | 12:09 |
| *** dstanek has joined #openstack-infra | 12:11 | |
| *** chandankumar_ has quit IRC | 12:12 | |
| kruskakli | not knowing anything, if I do: pip install psutil I seem to get psutil-1.2.1 which seem to contradict the 'psutil>=0.6.1,<1.0' requirement | 12:22 |
| kruskakli | so how can anyone get this to work? | 12:23 |
| kruskakli | perhaps I should ask it in the neutron channel? | 12:23 |
| * kruskakli sigh... | 12:23 | |
| *** dstanek has quit IRC | 12:33 | |
| *** praneshp has quit IRC | 12:35 | |
| *** bauzas has quit IRC | 12:43 | |
| *** coolsvap has quit IRC | 12:46 | |
| *** CaptTofu has joined #openstack-infra | 12:46 | |
| *** gokrokve has joined #openstack-infra | 12:48 | |
| *** jroovers has joined #openstack-infra | 12:49 | |
| *** GheRivero is now known as GheRivero_afk | 12:49 | |
| *** hashar has joined #openstack-infra | 12:49 | |
| *** gokrokve has quit IRC | 12:53 | |
| *** mozawa has joined #openstack-infra | 12:53 | |
| *** plomakin has quit IRC | 12:53 | |
| *** plomakin has joined #openstack-infra | 12:53 | |
| *** fbo_away is now known as fbo | 12:57 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 13:05 | |
| *** matel has joined #openstack-infra | 13:08 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: parse the failed jobs in stream https://review.openstack.org/65789 | 13:08 |
| *** smarcet has joined #openstack-infra | 13:09 | |
| kruskakli | YES, an ugly workaround made my day: sed -i -e '/psutil/ s/,.*//' requirements.txt | 13:09 |
| *** fbo is now known as fbo_away | 13:15 | |
| *** dpyzhov has quit IRC | 13:17 | |
| *** sandywalsh has quit IRC | 13:18 | |
| sergmelikyan | Now I am sure that https://review.openstack.org/61074 is restarting. Around 20 minutes ago I have seen job for this change with ETA 24 minutes and almost finished gate jobs | 13:19 |
| sergmelikyan | :( | 13:19 |
| *** weshay has joined #openstack-infra | 13:20 | |
| portante | clayg: this almost merged, made it to 2nd in line, only to fail. :( https://review.openstack.org/51228 | 13:20 |
| portante | I'll reverify it, back in line it goes ... | 13:20 |
| portante | acoles: ^^^ | 13:21 |
| *** dpyzhov has joined #openstack-infra | 13:22 | |
| *** mestery has joined #openstack-infra | 13:28 | |
| *** nprivalova has joined #openstack-infra | 13:28 | |
| *** mestery has quit IRC | 13:29 | |
| *** sparkycollier has joined #openstack-infra | 13:31 | |
| *** sandywalsh has joined #openstack-infra | 13:32 | |
| *** mfink has joined #openstack-infra | 13:32 | |
| *** ryanpetrello has joined #openstack-infra | 13:32 | |
| *** eharney has quit IRC | 13:32 | |
| *** dizquierdo has quit IRC | 13:39 | |
| *** fifieldt has quit IRC | 13:39 | |
| *** sparkycollier has quit IRC | 13:39 | |
| *** thomasem has joined #openstack-infra | 13:42 | |
| *** nicedice has joined #openstack-infra | 13:43 | |
| matel | Hello, Does anyone know if we have a swap partition in the devstack nodes? | 13:43 |
| *** CaptTofu has quit IRC | 13:44 | |
| *** Shrews has joined #openstack-infra | 13:45 | |
| *** CaptTofu has joined #openstack-infra | 13:45 | |
| *** che-arne has joined #openstack-infra | 13:47 | |
| *** wenlock has joined #openstack-infra | 13:48 | |
| *** gokrokve has joined #openstack-infra | 13:48 | |
| *** dprince has joined #openstack-infra | 13:50 | |
| *** Ajaeger has quit IRC | 13:52 | |
| *** gokrokve has quit IRC | 13:53 | |
| clarkb | matel: we do there is a setup swap function in devstack-gate | 13:54 |
| *** jasondotstar has joined #openstack-infra | 13:54 | |
| matel | clarkb: thanks! | 13:54 |
| *** dkliban_afk is now known as dkliban | 13:54 | |
| fungi | anteaya: on https://review.openstack.org/64786 i think the main challenge is that translators are working steadily on neutron strings and the gate has been backed up deeper than it could clear between scheduled translation update proposal jobs. maybe just wait for translators or the gate volume to settle down slightly | 13:56 |
| fungi | mayu left :( i wanted to discuss the test resource offer! oh well | 13:56 |
| fungi | EmilienM: on 62733, as a core reviewer on that project, you can just (re)add your approval vote if there's no real bug to reverify against | 13:59 |
| fungi | SergeyLukjanov: ^ | 13:59 |
| EmilienM | fungi: thank you, I'll know for next time :-) | 13:59 |
| SergeyLukjanov | fungi, morning | 13:59 |
| hashar | hello | 14:00 |
| SergeyLukjanov | fungi, readd means that you could just post +A commend again? | 14:00 |
| fungi | kruskakli: actually, i think that *is* the bug Ajaeger was thinking of, he just didn't know where to confirm... if you look at https://pypi.python.org/simple/psutil/ there are no tarballs for <1.0 (only external links) | 14:02 |
| *** mestery has joined #openstack-infra | 14:03 | |
| fungi | SergeyLukjanov: yes, and good afternoon to you sir | 14:03 |
| kruskakli | fungi: ok, should I log it as bug somewhere, or? | 14:03 |
| hashar | SergeyLukjanov: your Zuul patch to show the reconfigured timestamp is almost good to go i think, just missing an update of the status HTML. https://review.openstack.org/#/c/63849/ :D | 14:04 |
| *** rockyg has joined #openstack-infra | 14:04 | |
| *** banix has joined #openstack-infra | 14:04 | |
| fungi | kruskakli: anyway, your options are to use pip 1.4.1 instead of 1.5 (which probably means using virtualenv 1.10.1 instead of 1.11), or adjust tox.ini to pass --allow-external psutil --allow-insecure psutil to the pip install command. bug is https://launchpad.net/bugs/1267364 | 14:04 |
| SergeyLukjanov | hashar, yup, thanks for review, it's on my backlog, hope to complete it today | 14:05 |
| hashar | SergeyLukjanov: also commented out an idea which would be to detect the Zuul layout.yaml is in a git repository and thus get the git sha1 from there. | 14:05 |
| hashar | but that is a different feature | 14:05 |
| SergeyLukjanov | hashar, that's the next I was planned to do | 14:05 |
| fungi | kruskakli: sorry, linked wrong bug. clearly i'm tracking too many open bugs in my browser right now. https://launchpad.net/bugs/1266513 | 14:05 |
| SergeyLukjanov | next step* | 14:05 |
| hashar | SergeyLukjanov: awesome :) | 14:06 |
| SergeyLukjanov | hashar, it's not really helpful due to the fact that in infra layout.yaml file is installed by puppet AFAIK and isn't in the repo I think | 14:06 |
| SergeyLukjanov | hashar, but probably we could add some git/puppet magic to store sha to the file | 14:07 |
| hashar | SergeyLukjanov: on my install the layout is in a git repository which is then used to deploy. | 14:07 |
| SergeyLukjanov | hashar, oh, cool, then I'll try to make such patch too | 14:07 |
| SergeyLukjanov | hashar, I'm only able to read puppet scripts, no good exp on running them | 14:08 |
| *** dcramer_ has joined #openstack-infra | 14:08 | |
| kruskakli | fungi: perhaps the 'psutil>=0.6,<1.0' is too restrictive? seems awkward to have to revert to an older version of pip (but what do I know... :-) | 14:09 |
| hashar | SergeyLukjanov: don't waste your time on it though :D | 14:09 |
| SergeyLukjanov | hashar :) | 14:09 |
| *** rockyg has quit IRC | 14:09 | |
| fungi | kruskakli: i agree. presumably something in openstack is broken with >=1 (you could propose a change to openstack/requirements to lift that cap and see if integration tests pass) | 14:10 |
| BobBall | Does anyone have a nodepool setup / instructions for setting up a nodepool easily that we can use to test https://review.openstack.org/#/c/61463/ and https://review.openstack.org/#/c/65261/ ? | 14:10 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Add entrypoint for welcome_message https://review.openstack.org/65607 | 14:10 |
| fungi | BobBall: jeblair is still drafting the official documentation for it, so right now is't pretty much code comments, docstrings and whatever might be in the readme | 14:11 |
| fungi | BobBall: i know he was working on it this week while he was travelling for lca, so hopefully it will be clearer soon | 14:12 |
| BobBall | and I don't suppose there is a test nodepool somewhere? | 14:12 |
| *** chandankumar has quit IRC | 14:12 | |
| *** thuc has joined #openstack-infra | 14:12 | |
| fungi | BobBall: we have a nodepool instance more or less running on our jenkins-dev.openstack.org server if that's what you're asking. or if you mean a way to test code changes, i believe it has some unit tests. i'd have to double check that though | 14:13 |
| *** thuc_ has joined #openstack-infra | 14:13 | |
| *** bauzas has joined #openstack-infra | 14:13 | |
| BobBall | I meant a way to test code changes ;) | 14:13 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wip: moving readiness checks into stream https://review.openstack.org/65961 | 14:15 |
| fungi | zigo: pvo might know who to contact about getting official debian images added to rackspace cloud... i assume these i see in the rackspace image-list are unofficial? http://paste.openstack.org/show/60974/ | 14:16 |
| SergeyLukjanov | BobBall, we're using nodepool for building slaves for savanna-ci, I don't know about any problems with installing it | 14:17 |
| *** thuc has quit IRC | 14:17 | |
| zigo | fungi: Correct. | 14:17 |
| zigo | I have no idea how they have been made. | 14:18 |
| zigo | fungi: pvo == Paul Voccio, right? | 14:19 |
| * fungi checks /whois | 14:19 | |
| BobBall | yes | 14:19 |
| ruhe | BobBall: that should be easy. just tweak all prepare_node* files to reduce number of installed packages. then configure nodepool to use your openstack cloud (it could be even devstack installation) | 14:20 |
| BobBall | that is correct zigo | 14:20 |
| BobBall | the other person who might be able to help is anthonym (Ant Messerli) | 14:20 |
| BobBall | thanks ruhe | 14:20 |
| BobBall | sorry! antonym... Erroneous h there. | 14:21 |
| SergeyLukjanov | BobBall, nodepool will work only with nova-net out of the box | 14:21 |
| anteaya | fungi: perhaps the weekend is a good time for 64786 | 14:22 |
| fungi | BobBall: looking in the nodepool source, we seem to have the beginnings of fakes in the tools dir, but tox isn't configured for running any unit tests and i don't see any written yet | 14:22 |
| *** oubiwann has joined #openstack-infra | 14:23 | |
| fungi | anteaya: perhaps... the gate's flowing much faster now that some fixes are in, but the head is still ~15 hours old | 14:23 |
| BobBall | ok, thanks to all. I'll dig deeper. | 14:23 |
| fungi | BobBall: if you end up getting time to write tests though, we'd love the contribution! i'm quite sure it's on the to-do list once people are free'd up from other urgency | 14:24 |
| *** pliszka has joined #openstack-infra | 14:24 | |
| BobBall | *grin* | 14:24 |
| fungi | two things you can never have enough of... documentation and tests | 14:25 |
| notmyname | where are the files that config what channels are logged? | 14:26 |
| anteaya | SergeyLukjanov: please don't suggest people game the reverify bug number syntax | 14:26 |
| anteaya | core can reverify no bug | 14:26 |
| anteaya | or ask in here for a bug number | 14:26 |
| ruhe | fungi: btw, is there a way to find prioritized backlog for infra so i could pick up a task? or should i contact infra team? | 14:27 |
| *** dims has quit IRC | 14:27 | |
| anteaya | fungi: thanks | 14:27 |
| * anteaya admires fungi's juggling skills | 14:28 | |
| kruskakli | anteaya: Hi, I've managed to get my test.env going again and would like to get back my voting rights (I'll settle with manual voting hereafter :-) | 14:28 |
| anteaya | kruskakli: let's start with you offering logs as a patch for a week | 14:29 |
| SergeyLukjanov | anteaya, does you mean bug about slave agent failures? | 14:29 |
| anteaya | that enables people to review your logs and offer your comments for improvement | 14:29 |
| kruskakli | anteaya: ? | 14:29 |
| SergeyLukjanov | anteaya, and I don't think that cores could reverify no bug (https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L34) | 14:29 |
| SergeyLukjanov | anteaya, only be resending approve as fungi mentioned before | 14:29 |
| anteaya | kruskakli: you have to prove your system is stable and that you can offer logs for developers | 14:29 |
| kruskakli | anteaya: but if it's not doing any voting, does it matter? | 14:30 |
| anteaya | SergeyLukjanov: my biggest concern is the suggestion that people game the syntax | 14:30 |
| *** DinaBelova has joined #openstack-infra | 14:30 | |
| *** dims has joined #openstack-infra | 14:31 | |
| anteaya | we are trying to get folks in neutron to use the current syntax as suggested, telling people to game that undoes all my effort | 14:31 |
| anteaya | kruskakli: how many patches have you offered to openstack as a contributor? | 14:31 |
| kruskakli | anteaya: none | 14:31 |
| SergeyLukjanov | anteaya, I'm probably misunderstanding you :( | 14:32 |
| *** thuc_ has quit IRC | 14:32 | |
| *** thuc has joined #openstack-infra | 14:32 | |
| *** cyeoh_ has joined #openstack-infra | 14:32 | |
| *** cyeoh_ is now known as cyeoh | 14:32 | |
| notmyname | fungi: do you know where the configs are that control what channels are logged and available in eavesdrop? | 14:32 |
| SergeyLukjanov | anteaya, are you talking about recheck syntax for savanna-ci or something else? | 14:33 |
| anteaya | kruskakli: offer at least 3 patches to any project of your choosing and then get back to me on the importance of test logs for developers | 14:33 |
| anteaya | SergeyLukjanov: you suggested EmilienM do recheck bug 1 for his patch | 14:33 |
| anteaya | please don't tell people to do that | 14:33 |
| * kruskakli scratching head... | 14:33 | |
| ruhe | notmyname: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/eavesdrop.pp | 14:33 |
| SergeyLukjanov | anteaya, oh, yup, the reason was that I forgot firstly about the approve readding | 14:34 |
| fungi | notmyname: yeah, what ruhe linked | 14:34 |
| *** mfink has quit IRC | 14:35 | |
| kruskakli | anteaya: hm...I don't understand, I'm not doingn any openstack dev. at the moment, I just want to maintain our NCS driver | 14:35 |
| *** prad has joined #openstack-infra | 14:35 | |
| SergeyLukjanov | anteaya, I don't like bug 1 too and currently know about the correct way to reverify | 14:35 |
| notmyname | fungi: ruhe: thanks | 14:35 |
| *** mfink has joined #openstack-infra | 14:35 | |
| anteaya | SergeyLukjanov: okay fine, please encourage people to follow the intended use of reverify | 14:35 |
| SergeyLukjanov | anteaya, sure | 14:36 |
| anteaya | SergeyLukjanov: the correct way, if core on the project, is reverify no bug | 14:36 |
| openstackgerrit | John Dickinson proposed a change to openstack-infra/config: start logging openstack-swift https://review.openstack.org/65969 | 14:36 |
| anteaya | SergeyLukjanov: if not core, to offer reverify bug <bug-number> | 14:36 |
| fungi | anteaya: no, reverify no bug hasn't worked for a couple months. we disabled that, remember? | 14:36 |
| SergeyLukjanov | anteaya, reverify no bug was removed | 14:36 |
| anteaya | fungi: I thought core could still use it | 14:36 |
| SergeyLukjanov | anteaya, the only way is to resend approve comment | 14:36 |
| anteaya | in the case of infra issues | 14:37 |
| fungi | anteaya: you (remove if necessary and re-)add your approval vote | 14:37 |
| anteaya | oh okay | 14:37 |
| *** thuc has quit IRC | 14:37 | |
| anteaya | so again, core can do this | 14:37 |
| SergeyLukjanov | anteaya, fungi, but on the other hand, it looks like reverify no bug :( | 14:37 |
| fungi | anteaya: and for infra issues, when the infra team is doing this en masse, we have a 'zuul enqueue' utility we can use to add changes directly to a pipeline | 14:37 |
| SergeyLukjanov | but little more uncomfortable | 14:37 |
| anteaya | kruskakli: how can you fully hope to understand what developers need from your system, if you have no perception of what a developer experiences? | 14:38 |
| *** rossella_s has joined #openstack-infra | 14:38 | |
| fungi | SergeyLukjanov: the big difference from the old "reverify no bug" is that it's now limited to core reviewers on projects, who are hopefully somewhat more accountable for and aware of the problem at hand | 14:38 |
| anteaya | fungi: so core can still use revierify no bug? | 14:39 |
| fungi | anteaya: no, i just got done explaining that | 14:39 |
| fungi | anteaya: core can use approval votes | 14:39 |
| *** dpyzhov has quit IRC | 14:39 | |
| anteaya | k, they remove and readd approval | 14:39 |
| SergeyLukjanov | fungi, oh, correct, it's in fact limited to the core now | 14:39 |
| SergeyLukjanov | and that's ok | 14:39 |
| fungi | anteaya: the "approved" column you see in the review screen | 14:39 |
| *** yamahata has joined #openstack-infra | 14:39 | |
| anteaya | right | 14:40 |
| anteaya | so non-core needs a bug number | 14:40 |
| anteaya | not a fake one | 14:40 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support to specify GitLab version. https://review.openstack.org/65745 | 14:41 |
| fungi | anteaya: i was just pointing out that by removing zuul's match on the "reverify no bug" comment syntax, effectively the only ways to reenqueue a change are for a core reviewer to readd an approval vote, for a contributor to leave a "recheck bug xxxxx" comment (which i've also seen going horribly wrong lately, but that's another issue), or for an infra root admin to directly add the change to the gate | 14:41 |
| SergeyLukjanov | anteaya, the problem is that the fake one could be used ;) | 14:43 |
| fungi | ruhe: prioritized infra task backlog... https://bugs.launchpad.net/openstack-ci/ (bugs tagged low-hanging-fruit are hopefully things which any contributor can pitch in on without needing root access to infra's servers) | 14:43 |
| *** jorisroovers has joined #openstack-infra | 14:44 | |
| fungi | ruhe: https://bugs.launchpad.net/openstack-ci/+bugs?field.tag=low-hanging-fruit | 14:44 |
| anteaya | SergeyLukjanov: encouraging folks to game the system, which is a big problem in Neutron which I and others are trying to address, please don't add to the problem | 14:44 |
| anteaya | fungi: thank you | 14:44 |
| fungi | ruhe: we also try to have bug triage days once per milestone or so, to make sure those are properly tagged for most efficient outside contribution | 14:44 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Add support for parameters in pipeline publisher. https://review.openstack.org/65719 | 14:44 |
| *** eharney has joined #openstack-infra | 14:45 | |
| kruskakli | anteaya: perhaps I misunderstand what you are saying but if we need to extend our driver we will do that, if someone changes something that require us to change our driver, we will do that, if something changes that breaks our driver we will "raise a flag"/vote so that the problem can be resolved, any other openstack dev. is not in our plans at the moment | 14:45 |
| ruhe | fungi: thank you | 14:45 |
| *** jroovers has quit IRC | 14:46 | |
| anteaya | kruskakli: your use of 3rd party testing tools and others have alerted us to a big problem, you are creating feedback with no knowledge of how that feedback is used and consumed by the openstack developers | 14:48 |
| sdague | oh look, someone poorly configured zuul - https://review.openstack.org/#/c/63647/ | 14:48 |
| anteaya | this is a big problem | 14:48 |
| fungi | kruskakli: i have to agree with anteaya... if a vendor's main contribution is to write a driver which supports their proprietary hardware/systems, that's somewhat anti-social in a free software project. gaining familiarity with the ongoing development of the project is encouraged for effective integration with the community | 14:48 |
| *** gokrokve has joined #openstack-infra | 14:48 | |
| anteaya | while not a requirement yet, we are heading in the direction of requiring two active contributores per 3rd party test system for voting to be enabled | 14:49 |
| SergeyLukjanov | anteaya, +1 for such requirement | 14:49 |
| anteaya | otherwise the messs you can make with your system because you don't know how the results are consumed are just too difficult for the workflow of the developers affected | 14:49 |
| fungi | it helps to have enough familiarity with openstack internals to be able to point out *why* a change is breaking your black box, and to suggest possible alternative solutions | 14:50 |
| *** jorisroovers has quit IRC | 14:50 | |
| fungi | and that is best accomplished through being an active contributor to the core components with which your device or software is interacting in openstack | 14:50 |
| kruskakli | anteaya: I'm sorry to hear that, I've been contributing to open source as long as the concept has existed, but always on a voluntary basis according to what effort I've been able to muster, I need to ponder over this... | 14:51 |
| *** burt has joined #openstack-infra | 14:52 | |
| anteaya | kruskakli: we aren't targeting you, please no that | 14:52 |
| anteaya | there is a large influx of people asking for 3rd party testing permission who also have no prior experience contributed to openstack and we are trying to figure out how to address it | 14:53 |
| *** gokrokve has quit IRC | 14:53 | |
| kruskakli | anteaya: sure, I'm taking it personal, but 2 person means 20% of our dev. capacity | 14:53 |
| anteaya | honestly we didn't expect people who have never contributed to openstack would be operating these systems | 14:54 |
| anteaya | we have to be consistent with all the 3rd party test accounts and there are a large number of them now | 14:54 |
| kruskakli | sorry I meant I'm not taking it personal | 14:54 |
| anteaya | I'm glad to hear that | 14:54 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Add SQLalchemy database model https://review.openstack.org/62239 | 14:54 |
| anteaya | you need to understand what a disruption this is for our devs | 14:55 |
| fungi | it has placed a surprising additional amount of workload on the infra team and, to a great extent, other subprojects that we're trying to teach the third-party ci operators about the openstack development process and workflow, particularly challenging if they're not expecting to contribute directly to the internals of the project | 14:55 |
| anteaya | we have quite a few tests systems run by people with no appreciation of how their logs or lack of them are interupted development | 14:56 |
| kruskakli | ok, I'll bring it up with my bozz and we'll see how to proceed | 14:56 |
| anteaya | kruskakli: also know we are still figuring out how to approach this from our end | 14:57 |
| kruskakli | anyway, have a good weeken to you all I'm going home :-) | 14:57 |
| anteaya | we didn't expect what we are having to deal with so we didn't have any prior plans as to how to approach it | 14:57 |
| *** pcrews has joined #openstack-infra | 14:58 | |
| anteaya | kruskakli: have a good weekend as well | 14:58 |
| fungi | kruskakli: good weekend to you too! | 14:58 |
| *** CaptTofu has quit IRC | 14:59 | |
| *** dcramer_ has quit IRC | 15:00 | |
| *** mriedem has joined #openstack-infra | 15:01 | |
| *** Ajaeger1 has joined #openstack-infra | 15:02 | |
| *** mtreinish has quit IRC | 15:04 | |
| *** jergerber has joined #openstack-infra | 15:04 | |
| *** jergerber has quit IRC | 15:04 | |
| *** mtreinish has joined #openstack-infra | 15:06 | |
| *** coolsvap has joined #openstack-infra | 15:09 | |
| *** dizquierdo has joined #openstack-infra | 15:14 | |
| *** rwsu has quit IRC | 15:15 | |
| *** mrodden has joined #openstack-infra | 15:17 | |
| *** CaptTofu has joined #openstack-infra | 15:20 | |
| *** jgrimm has joined #openstack-infra | 15:24 | |
| *** rwsu has joined #openstack-infra | 15:26 | |
| *** pblaho has joined #openstack-infra | 15:26 | |
| *** rcleere has joined #openstack-infra | 15:27 | |
| sdague | jeblair: as I'm looking over the ever increasing complexity of some of our javascript tools, thoughts on doing something like this instead - http://javascriptissexy.com/handlebars-js-tutorial-learn-everything-about-handlebars-js-javascript-templating/ | 15:27 |
| *** jgrimm has quit IRC | 15:28 | |
| *** jasondotstar has quit IRC | 15:29 | |
| bknudson | dojo javascript library also has templates -- http://dojotoolkit.org/reference-guide/1.8/dojox/dtl.html -- haven't used it other than in widgets. | 15:31 |
| *** jgrimm has joined #openstack-infra | 15:31 | |
| *** ^d has joined #openstack-infra | 15:32 | |
| *** markmcclain has joined #openstack-infra | 15:36 | |
| *** pliszka has left #openstack-infra | 15:36 | |
| ruhe | fungi: i'm working on CI/CD for storyboard. puppet module is already on review (https://review.openstack.org/#/c/65017/). the next step is to add storyboard.openstack.org to site.pp | 15:38 |
| ruhe | fungi: should i file a ticket for a new server vm for storyboard? | 15:38 |
| ruhe | ticket in laucnhpad.net/openstack-ci | 15:39 |
| *** russellb is now known as rustlebee | 15:39 | |
| *** gokrokve has joined #openstack-infra | 15:39 | |
| *** mfink has quit IRC | 15:39 | |
| *** kraman1 has quit IRC | 15:40 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Use venv to build documentation https://review.openstack.org/65987 | 15:40 |
| *** mfink has joined #openstack-infra | 15:40 | |
| *** mrodden1 has joined #openstack-infra | 15:42 | |
| *** wenlock has quit IRC | 15:42 | |
| *** alaski has quit IRC | 15:43 | |
| *** mrodden has quit IRC | 15:43 | |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 15:44 |
| *** dansmith is now known as damnsmith | 15:44 | |
| *** GheRivero_afk is now known as GheRivero | 15:46 | |
| *** ymr1 has quit IRC | 15:47 | |
| zul | arrgh | 15:47 |
| *** dizquierdo has quit IRC | 15:47 | |
| *** jasondotstar has joined #openstack-infra | 15:48 | |
| fungi | ruhe: nah, i can just launch a server and either apply the change once merged or in a separate development puppet env to check it out | 15:55 |
| *** herndon_ has joined #openstack-infra | 15:56 | |
| ruhe | fungi: that's great. i'll get back to you once storyboard is ready to be deployed :) | 15:57 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: provide time in queue in zuul ui https://review.openstack.org/65993 | 15:57 |
| sdague | ok, I'm pretty happy with that one :) | 15:58 |
| fungi | ruhe: let me make sure i find some time to review it, and then i'll test it out | 15:58 |
| fungi | ruhe: it's on my list for later today, after i get more done on the server config management for our upcoming openid provider service | 15:58 |
| *** dangers_away is now known as dangers | 15:59 | |
| *** flaper87 is now known as flaper87|afk | 15:59 | |
| fungi | ruhe: i'd love for you to have a demo storyboard already up in our infra by the time the sprint happens | 15:59 |
| *** thedodd has joined #openstack-infra | 15:59 | |
| ruhe | fungi: yeah, that's my (and everybody else's) to have it running before the sprint | 15:59 |
| SergeyLukjanov | sdague, oh, it looks interesting ;) | 15:59 |
| ruhe | *my goal | 15:59 |
| fungi | ruhe: it's a laudable one | 16:00 |
| sdague | SergeyLukjanov: http://ubuntuone.com/1Wi72AV0ELHD3V1MX5KcjZ | 16:00 |
| fungi | i won't be at the sprint, but i gather some other infra folk may be popping over from fosdem for it | 16:00 |
| ruhe | fungi: afaik mordred and jeblair will be there | 16:01 |
| *** herndon_ has quit IRC | 16:01 | |
| fungi | ruhe: perfect | 16:02 |
| SergeyLukjanov | sdague, thx | 16:02 |
| SergeyLukjanov | sdague, I'll take a look on your change today, it looks very useful | 16:03 |
| sdague | yeh, then I stop having to do math to see how bad our queue is | 16:03 |
| sdague | and, right now, top of queue entered 15+ hrs ago | 16:03 |
| fungi | i love that we have so many people contributing usability improvements to the zuul status screen now. that tells me there are *tons* of people who find it useful already too | 16:03 |
| SergeyLukjanov | fungi, I'd like to rewrite the js code to the more readable/patchable style, but I'm afraid that my knowledge of js isn't good enough now | 16:05 |
| fungi | though the test nodes graph has me worried that we have a lot of our aggregate quota full of stale entries stuck in a building or deleting state... checking up on that now | 16:05 |
| fungi | SergeyLukjanov: nor mine either, but sounds like an excellent opportunity to get better at it ;) | 16:05 |
| SergeyLukjanov | fungi, I'll probable return back to this idea after a pack of minor improvements to the server side zuul | 16:06 |
| sdague | SergeyLukjanov: actually, look at http://javascriptissexy.com/handlebars-js-tutorial-learn-everything-about-handlebars-js-javascript-templating/ | 16:10 |
| openstackgerrit | Michael Still proposed a change to openstack-infra/zuul: Implement a simple mysql reporter. https://review.openstack.org/65885 | 16:10 |
| sdague | I think that would go a huge way | 16:10 |
| SergeyLukjanov | sdague, I think that templating will not be enough to make it more readable | 16:11 |
| *** mdenny has joined #openstack-infra | 16:11 | |
| SergeyLukjanov | sdague, I'm looking on something like angular.js or knockout.js | 16:11 |
| SergeyLukjanov | both of them supports templating | 16:11 |
| SergeyLukjanov | and will provide us with ability to not rewrite the whole html for all changes | 16:12 |
| SergeyLukjanov | only changed elements | 16:12 |
| SergeyLukjanov | AFAIR | 16:12 |
| sdague | SergeyLukjanov: yeh, that would be a far more extensive undertaking | 16:13 |
| sdague | I think you could get rid of a ton of debt just getting to templates | 16:13 |
| sdague | and then do a further conversion in the future | 16:13 |
| SergeyLukjanov | sdague, yup, that's not so destructive as a fully rewrite ;) | 16:13 |
| sdague | SergeyLukjanov: and could be done in bite sized chunks | 16:13 |
| sdague | honestly, if jeblair isn't against it, I was going to start trickling in templated stuff over time | 16:14 |
| *** gokrokve has quit IRC | 16:18 | |
| *** gokrokve has joined #openstack-infra | 16:18 | |
| *** pblaho has quit IRC | 16:20 | |
| *** UtahDave has joined #openstack-infra | 16:21 | |
| *** gokrokve has quit IRC | 16:23 | |
| fungi | yeah, so roughly half of the nodes nodepool knew about were >3 hours old and not picking up any jobs. i've got 5 parallel delete loops going over the list of stale nodes now, to get us back to capacity, then i'll restart nodepool gracefully and start watching for any new signs of a persistent leak | 16:25 |
| fungi | some of the nodes were >100 hours old | 16:25 |
| *** alaski has joined #openstack-infra | 16:26 | |
| fungi | which takes us back to the jenkins restarts early in the week when we cleared everything | 16:26 |
| *** reed has joined #openstack-infra | 16:26 | |
| *** beagles is now known as beagles_brb | 16:27 | |
| markmcclain | so I've got a question about list.o.o | 16:28 |
| fungi | markmcclain: ask away | 16:29 |
| markmcclain | I updated my subscriber address yesterday and it seems to take effect immediately | 16:29 |
| markmcclain | and then this morning mail started showing up on the old address | 16:29 |
| *** SergeyLukjanov has quit IRC | 16:30 | |
| fungi | that's weird. old dates on the headers? | 16:30 |
| fungi | wondering if those might have been stuck in a deferred queue somewhere | 16:30 |
| markmcclain | all current messages | 16:30 |
| fungi | does the subscriber address in mailman show your old one again? | 16:31 |
| *** kraman1 has joined #openstack-infra | 16:32 | |
| jog0 | http://status.openstack.org/elastic-recheck/ is showing no gate hits | 16:32 |
| *** gyee_ has joined #openstack-infra | 16:32 | |
| jog0 | which I find hard to believe | 16:32 |
| fungi | jog0: do i need to check the listener daemon? | 16:33 |
| jog0 | fungi: that is graphite data | 16:33 |
| markmcclain | fungi: yes | 16:33 |
| fungi | jog0: how long has it been? | 16:33 |
| fungi | markmcclain: i'll check for any strange errors on the list server, or signs anyone might have been doing maintenance | 16:34 |
| jog0 | http://graphite.openstack.org/graphplot/?from=-36hours&height=200&until=now&width=600&bgcolor=ffffff&fgcolor=000000&hideGrid=true&target=color(alias(hitcount(sum(stats.zuul.pipeline.gate.job.gate-tempest-dsvm-neutron.{SUCCESS,FAILURE}),'5hours'),%20'gate-tempest-dsvm-neutron'),'purple')&target=color(alias(hitcount(sum(stats.zuul.pipeline.gate.job.gate-tempest-dsvm-full.{SUCCESS,FAILURE}),'5hours'),%20'gate-tempest-dsvm-full'),'orange')&target=color(alias | 16:34 |
| fungi | maybe someone restored an earlier subscriber database or something | 16:34 |
| jog0 | a few hours | 16:34 |
| markmcclain | fungi: thanks.. I'll just go ahead and update the address again | 16:34 |
| fungi | markmcclain: sorry about that--i agree that's not how it should have to work ;) | 16:35 |
| markmcclain | no worries | 16:35 |
| *** derekh has quit IRC | 16:36 | |
| fungi | markmcclain: yeah, nobody's logged into a shell on that server for nearly 4 weeks | 16:36 |
| *** AaronGr_Zzz is now known as AaronGr | 16:36 | |
| portante | jog0 that logstash appears cut off ... | 16:37 |
| jog0 | portante: logstash? that was a graphite link | 16:37 |
| *** CaptTofu has quit IRC | 16:38 | |
| portante | sorry | 16:38 |
| *** jcoufal has quit IRC | 16:38 | |
| portante | that link above ... :) | 16:38 |
| fungi | however, portante, i agree it's truncated | 16:38 |
| fungi | ...20'gate-tempest-dsvm-full'),'orange')&target=color(alias | 16:39 |
| fungi | and it ends abruptly there | 16:39 |
| jog0 | anyway see graphs at status.openstack.org/elastic-recheck/ | 16:41 |
| openstackgerrit | Mate Lakat proposed a change to openstack-infra/nodepool: Readme enhancements https://review.openstack.org/66011 | 16:41 |
| *** jog0 is now known as flashgordon | 16:44 | |
| *** changbl has quit IRC | 16:46 | |
| *** markmcclain has quit IRC | 16:49 | |
| fungi | [savior of the universe] | 16:53 |
| * fungi seems to get queen's flash gordon theme stuck in his head every friday, like clockwork | 16:53 | |
| fungi | node deletes are still going, but the test nodes graph is already starting to look a lot more sane | 16:54 |
| fungi | and we've got waaay more changes being tested at a time now as a result | 16:55 |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Get rid of Django https://review.openstack.org/66016 | 16:56 |
| *** ruhe is now known as _ruhe | 16:59 | |
| guitarzan | fungi! king of the impossible! | 16:59 |
| *** rwsu has quit IRC | 17:00 | |
| *** Ajaeger1 has quit IRC | 17:02 | |
| *** dcramer_ has joined #openstack-infra | 17:04 | |
| *** wenlock has joined #openstack-infra | 17:06 | |
| *** bnemec is now known as beekneemech | 17:07 | |
| *** _ruhe is now known as ruhe | 17:07 | |
| *** pcrews has quit IRC | 17:07 | |
| *** gokrokve has joined #openstack-infra | 17:08 | |
| *** dcramer_ has quit IRC | 17:09 | |
| fungi | nope, that's jog0's job title | 17:10 |
| openstackgerrit | Dirk Mueller proposed a change to openstack/requirements: Allow newer versions of psutil as well https://review.openstack.org/66022 | 17:10 |
| *** rnirmal has joined #openstack-infra | 17:11 | |
| zaro | fungi: _david_ requested repro steps for this.. https://code.google.com/p/gerrit/issues/detail?id=2377 | 17:13 |
| zaro | fungi: i'm not sure how we setup the CLA, would you be able to provide info? | 17:13 |
| fungi | zaro: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/gerrit.pp#n349 | 17:17 |
| fungi | zaro: that runs http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/gerrit_set_agreements.sh.erb | 17:18 |
| *** SergeyLukjanov has joined #openstack-infra | 17:19 | |
| zaro | fungi: i'll take a look. thanks. | 17:19 |
| *** mancdaz is now known as mancdaz_away | 17:23 | |
| *** beagles_brb is now known as beagles | 17:24 | |
| *** hashar has quit IRC | 17:25 | |
| *** madmike has joined #openstack-infra | 17:25 | |
| *** CaptTofu has joined #openstack-infra | 17:26 | |
| *** jpich has quit IRC | 17:27 | |
| *** mfink has quit IRC | 17:27 | |
| fungi | zaro: at the time we set it up, the extent of the instructions was the "auth.contributorAgreements" entry in https://review.openstack.org/Documentation/config-gerrit.html#_a_id_auth_a_section_auth | 17:28 |
| fungi | zaro: "the admin must also insert one or more rows into contributor_agreements and create agreement files under '$site_path'/static, so users can actually complete one or more agreements." | 17:29 |
| *** dstanek has joined #openstack-infra | 17:29 | |
| fungi | zaro: so that's basically what that script is doing | 17:29 |
| zaro | fungi: seems like we all this is done thru db. is there an equivalent way to do this stuff from gerrit ui? | 17:30 |
| fungi | zaro: nope, or at least there was none at the time | 17:30 |
| fungi | as it was, i had to contribute patches upstream (which landed in 2.5) and backport them to 2.4 because the feature was still half-baked | 17:30 |
| sdake | recently I changed permissions on the #heat channel to add logging support for the channel, where are the irc logs stored? | 17:31 |
| zaro | fungi: do you remember if this was our hack or was it something recommended by gerrit docs? | 17:31 |
| fungi | sdake: http://eavesdrop.openstack.org/irclogs/%23heat/ | 17:31 |
| sdake | thanks fungi | 17:31 |
| fungi | zaro: it was our automation of the (very minimal) gerrit docs | 17:31 |
| *** mriedem has quit IRC | 17:32 | |
| fungi | zaro: the bugs i fixed were unrelated, the cla signing interface in 2.4 was simply broken when we started trying to use it | 17:32 |
| *** markmcclain has joined #openstack-infra | 17:32 | |
| fungi | zaro: specifically bugs with the non-group based self-verifying cla functionality. so it's possible gerrit devs might consider a broken feature to not have been an actual released feature, since you couldn't use it without patching and rebuilding gerrit | 17:34 |
| fungi | i think one of the bugs was there, and one of the bugs was actually related to the contact information form being broken/unusable instead | 17:34 |
| zaro | fungi: wow, seems like it would be difficult for me to repro on 2.4 | 17:35 |
| zaro | fungi: and it might not even be a bug against upstream at all because we are upgrading from our forked gerrit. | 17:36 |
| fungi | it may not be worth our time to reproduce, since they fixed it in 2.5 but did not backport fixes to 2.4 | 17:36 |
| fungi | zaro: well, it's a failure to support upgrading from a broken feature which nobody could have used without backporting the fixes themselves, as we did | 17:36 |
| fungi | gerrit fixed them upstream, but they fixed them in 2.5 | 17:37 |
| openstackgerrit | Sushil Kumar proposed a change to openstack-dev/hacking: Add -U to pip install command in tox.ini https://review.openstack.org/66026 | 17:37 |
| fungi | and i think they may also have dropped support for non-group-backed contributor agreements in 2.5 as well | 17:37 |
| *** dstanek has quit IRC | 17:37 | |
| *** blamar has joined #openstack-infra | 17:40 | |
| *** dpyzhov has joined #openstack-infra | 17:41 | |
| zaro | not sure what to do about this. i don't think this is an upstream bug at all now. | 17:42 |
| *** dcramer_ has joined #openstack-infra | 17:42 | |
| *** ruhe is now known as _ruhe | 17:43 | |
| fungi | zaro: well, it's an upstream bug, i think, but i can understand them not caring about it | 17:43 |
| zaro | fungi: do we have to automate the upgrade from openstack gerrit 2.4 to upstream gerrit 2.8? | 17:44 |
| fungi | zaro: as i said, the feature was broken in 2.4, and they opted not to support it, but rather replaced bits of it in 2.5. i think that's more or less the resolution | 17:44 |
| zaro | fungi: my thought was that we do manual upgrade from 2.4 to 2.8, then automated from 2.8 and beyond. since 1st upgrade is such a big spread. | 17:45 |
| fungi | zaro: i'm still in favor of just working around it, so long as the situation's documented upstream for the benefit of others who may have also hacked the broken cla support in 2.4 into a working state and then later decide to upgrade | 17:45 |
| *** che-arne has quit IRC | 17:45 | |
| *** che-arne has joined #openstack-infra | 17:45 | |
| zaro | fungi: yeah, i agree. there was another problem as well where we needed to truncate a table before the upgrade. | 17:46 |
| fungi | zaro: we can discuss upgrade automation with the group. we'll want to script it regardless, whether it's in the form of scripts in our puppet repo or prewritten scripts in pastes or an etherpad | 17:46 |
| fungi | because we want it to be exactly repeatable in case we end up having to roll back, revise and try again | 17:46 |
| *** dcramer_ has quit IRC | 17:47 | |
| zaro | fungi: cool. i'll update the bug with more accurate info, like the fact that we are upgrading from our forked gerrit 2.4. | 17:47 |
| *** BobBall is now known as BobBallAway | 17:47 | |
| *** harlowja_away is now known as harlowja | 17:47 | |
| fungi | specifically that we hand-patched 2.4 with fixes backported from 2.5 to get it working | 17:47 |
| fungi | because it had been released in a broken state in 2.4 | 17:48 |
| zaro | ok. i have a better picture of what happened now. i'll update the bug to see if _david_ wants to pursue. i'm thinking it's unlikely that he would. | 17:50 |
| fungi | right, he seemed to be pretty insistent that it should get fixed upstream if it's a bug, but in this case i think it's a design flaw which upstream knew about and decided not to fix back in 2.5 | 17:51 |
| fungi | instead they redid how some of it worked, and seemed to have decided not to provide an upgrade path from the nonworking implementation in 2.4 | 17:52 |
| fungi | which makes sense. it's a pragmatic choice | 17:52 |
| fungi | to have actually used it in 2.4 you would have had to be running a fork | 17:53 |
| fungi | just to get it usable | 17:53 |
| *** dstanek has joined #openstack-infra | 17:56 | |
| *** hogepodge has quit IRC | 17:57 | |
| *** nati_ueno has joined #openstack-infra | 18:01 | |
| *** nati_uen_ has joined #openstack-infra | 18:02 | |
| reed | when will people be able to create new projects on stackforge? | 18:02 |
| *** nati_uen_ has quit IRC | 18:03 | |
| fungi | reed: i've been reviewing the new project proposals and looking for one with no further issues i can test with. i didn't get a chance to look back at them again since wednesday when i went through them last, but hopefully i can try to put one through today | 18:03 |
| *** mriedem has joined #openstack-infra | 18:04 | |
| *** w_ has joined #openstack-infra | 18:04 | |
| fungi | right now trying to get the gate back into a happy state (nodepool is taking far too long to gracefully stop) | 18:04 |
| *** nati_uen_ has joined #openstack-infra | 18:04 | |
| reed | fungi, check this one https://review.openstack.org/#/c/61954/ :) | 18:04 |
| *** nati_ueno has quit IRC | 18:05 | |
| fungi | reed: i'd missed that one since it got auto-abandoned | 18:05 |
| fungi | adding to my review | 18:05 |
| *** GheRivero is now known as GheRivero_afk | 18:05 | |
| reed | fungi, thanks | 18:05 |
| *** mriedem has quit IRC | 18:06 | |
| *** olaph has quit IRC | 18:06 | |
| fungi | i've restored it. i'll give it a quick rebase | 18:06 |
| fungi | actually, it needs editing too. i'll leave comments | 18:07 |
| *** luqas has quit IRC | 18:07 | |
| *** jerryz has joined #openstack-infra | 18:08 | |
| *** markmc has quit IRC | 18:09 | |
| *** pcrews has joined #openstack-infra | 18:12 | |
| flashgordon | fungi: heh | 18:12 |
| *** vipuls is now known as vipuls-away | 18:15 | |
| *** vipuls-away is now known as vipuls | 18:15 | |
| *** Bada has quit IRC | 18:16 | |
| *** mgagne1 has joined #openstack-infra | 18:16 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: maven-deploy: don't require id https://review.openstack.org/66033 | 18:17 |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id https://review.openstack.org/66034 | 18:17 |
| *** mgagne2 has joined #openstack-infra | 18:18 | |
| *** thuc has joined #openstack-infra | 18:18 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Explain howto use maven's pre/postbuilders https://review.openstack.org/66035 | 18:18 |
| *** mgagne has quit IRC | 18:19 | |
| *** thuc has quit IRC | 18:19 | |
| *** thuc has joined #openstack-infra | 18:19 | |
| *** thuc has quit IRC | 18:20 | |
| *** thuc has joined #openstack-infra | 18:21 | |
| *** mgagne1 has quit IRC | 18:21 | |
| *** thuc has quit IRC | 18:21 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: project_maven: Don't require artifact-id and group-id https://review.openstack.org/66036 | 18:22 |
| *** thuc has joined #openstack-infra | 18:22 | |
| *** gyee_ has quit IRC | 18:23 | |
| *** hogepodge has joined #openstack-infra | 18:27 | |
| *** sarob has joined #openstack-infra | 18:28 | |
| *** sarob has quit IRC | 18:28 | |
| *** sarob has joined #openstack-infra | 18:28 | |
| *** mgagne has joined #openstack-infra | 18:30 | |
| *** mgagne is now known as mgagne1 | 18:30 | |
| *** mgagne2 has quit IRC | 18:30 | |
| harlowja | hmmm, qq, it seems like most of the stackforge jobs (and alot of others) are sitting at 100% done (but not going past 100%), something is up i guess right? | 18:34 |
| fungi | harlowja: yes, that's me | 18:34 |
| fungi | i'm trying to gracefully restart nodepool | 18:34 |
| harlowja | kk | 18:34 |
| harlowja | thx fungi | 18:34 |
| fungi | should pick back up here momentarilyu | 18:34 |
| fungi | nothing's broken, its just we had a lot of cruft i'm trying to clean out to get our capacity back up | 18:35 |
| harlowja | cool, for those that use the czuul tool that i made (trunk has the following that i added, czuul --project "stackforge*") | 18:35 |
| harlowja | to watch certain projects only :-P | 18:35 |
| *** SergeyLukjanov has quit IRC | 18:35 | |
| fungi | harlowja: basically, we're out of available nodepool-managed nodes for a bit while it's restarting | 18:36 |
| *** dizquierdo has joined #openstack-infra | 18:36 | |
| harlowja | :-/ | 18:36 |
| fungi | zuul gobbled up the remaining supply while nodepool was busy trying to work through provider nova api timeouts | 18:36 |
| harlowja | neat stuff, i saw that jeblair i think was doing some conference about zuul,openstackinfra, is that video online anywhere? | 18:36 |
| harlowja | http://linux.conf.au/schedule/30168/view_talk?day=friday (maybe its happening right now?) | 18:37 |
| fungi | harlowja: if you're talking about lca this week, not sure if they've got any vids up. the presentation slides are probably mostly reflected in http://docs.openstack.org/infra/publications/ | 18:38 |
| harlowja | ah, thx | 18:38 |
| harlowja | k | 18:38 |
| *** SergeyLukjanov has joined #openstack-infra | 18:38 | |
| flashgordon | fungi: having fun as the infra skeleton crew | 18:43 |
| *** Ryan_Lane2 has joined #openstack-infra | 18:45 | |
| fungi | flashgordon: of course | 18:45 |
| fungi | also, nodepool is back up again | 18:45 |
| fungi | within the next 10 minutes we should have nodes running more tests | 18:45 |
| Ryan_Lane2 | on a scale of 10 - not interested, if I added github support to zuul would my changes be accepted upstream? | 18:45 |
| fungi | Ryan_Lane2: curious what you mean by "support" (github triggers or some such?) | 18:46 |
| Ryan_Lane2 | github triggers and reporting back into github | 18:46 |
| Ryan_Lane2 | basically gating support for it. | 18:46 |
| fungi | Ryan_Lane2: i suspect it would need to come with a fake backend and a fair battery of tests since we'd not want to connect to github when running zuul unit tests | 18:47 |
| Ryan_Lane2 | indeed | 18:47 |
| fungi | Ryan_Lane2: but i'll defer to jeblair on it | 18:47 |
| fungi | as to whether or not we'd accept patches for support in-tree | 18:48 |
| *** DinaBelova has quit IRC | 18:48 | |
| Ryan_Lane2 | I'm not working somewhere not using gerrit, but am quickly approaching the limits of what I can do in jenkins | 18:48 |
| Ryan_Lane2 | s/not/now/ | 18:48 |
| fungi | seems like a reasonable request to me as long as we can be assured we don't break it and don't need to touch github ourselves, but i'd seek additional consensus when people aren't in post-lca travel | 18:48 |
| Ryan_Lane2 | sounds good. it may be that I don't end up using zuul, but I wanted to check before I went down that rabbit hole | 18:49 |
| fungi | Ryan_Lane2: looked at turbo-hipster? (non-jenkins job runner written in python by some of our contributors) | 18:49 |
| zaro | fungi: about discussion yesterday regarding Change Owner owner_group_id is set to 0 in the gerrit DB.. | 18:50 |
| fungi | zaro: yeah, did you ever get additional details on that? | 18:50 |
| zaro | fungi: did some testing and found out that gerrit 2.8 doesn't even have that column anymore. | 18:50 |
| fungi | zaro: i'm going to guess 0 means "no owner" | 18:50 |
| fungi | oh! | 18:50 |
| zaro | fungi: in fact for 2.9 they moved all users from the DB. | 18:51 |
| Ryan_Lane2 | I think we're fine using jenkins, but wanted to handle gating better than jenkins itself can handle | 18:51 |
| zaro | fungi: So the migrated tables don't even get used anymore, i think everthing got moved from db to git repo. | 18:51 |
| fungi | zaro: any idea when they're targeting for a 2.9 release? it sounds like they're doing such heavy refactoring we might be better trying to work on an upgrade to 2.9 if it's soon, and save ourselves extra work | 18:52 |
| Ryan_Lane2 | fungi: do you know if zuul could have a merge event in master trigger events in open change requests? | 18:52 |
| zaro | fungi: which means i don't have to patch that at all :) | 18:52 |
| fungi | Ryan_Lane2: i'd need to know what "trigger events in open change requests" meant in this case | 18:52 |
| fungi | zaro: heh | 18:52 |
| Ryan_Lane2 | well, I'd like to do something like rebase and recheck open change requests, so they are always tested against master | 18:53 |
| *** mfer has joined #openstack-infra | 18:54 | |
| Ryan_Lane2 | my goal is continuous deployment, where all change requests are fully tested against the master and are always ready to merge and deploy | 18:55 |
| fungi | Ryan_Lane2: jeblair was talking about possibly adding a timed trigger to retest open changes in zuul and report on them if they're stale | 18:56 |
| Ryan_Lane2 | that would be awesome | 18:56 |
| fungi | basically as a low-priority pipeline | 18:56 |
| fungi | so it gets available workers if there's nothing else taht needs them | 18:56 |
| Ryan_Lane2 | in my case I'd have it as a trigger that occurs on master merges and retests. I just need to add a trigger for that, I'd imagine? | 18:57 |
| fungi | Ryan_Lane2: if you're interested in supporting that (i know it was on sdague's wishlist too) you should talk to jeblair about implementation details | 18:57 |
| *** dpyzhov has quit IRC | 18:57 | |
| fungi | Ryan_Lane2: that sounds sane... if master moves and there are changes which are still open, retest and report | 18:57 |
| Ryan_Lane2 | well, let me ensure I'm going to use this before I make any promises :) | 18:58 |
| fungi | maybe with a throttle so it won't do it unless it's been for a configurable period of time | 18:58 |
| Ryan_Lane2 | yeah, that's a good idea | 18:58 |
| fungi | and maybe only have it leave a comment if it's failing | 18:58 |
| fungi | or rather, only comment if it would be a state/vote change | 18:59 |
| * Ryan_Lane2 nods | 18:59 | |
| fungi | to catch open changes which go from broken to working without a new patchset because master grew a feature they needed or something | 18:59 |
| *** rwsu has joined #openstack-infra | 18:59 | |
| Ryan_Lane2 | I can already tell this is going to be a pain in github since it doesn't have gating support | 18:59 |
| fungi | Ryan_Lane2: you know you love travis-ci. don't deny it ;) | 19:00 |
| Ryan_Lane2 | we're not using it | 19:00 |
| *** herndon_ has joined #openstack-infra | 19:01 | |
| *** Ryan_Lane2 has quit IRC | 19:02 | |
| *** herndon_ has quit IRC | 19:02 | |
| *** andreaf has quit IRC | 19:03 | |
| mfer | fungi the other day you provided feedback on https://review.openstack.org/#/c/62069/. i'm looking for a bit of direction because what you said appears to differ from what's documented. i'm trying to understand it before i make any changes | 19:04 |
| mfer | fungi can you take a look? | 19:04 |
| *** Ryan_Lane2 has joined #openstack-infra | 19:04 | |
| fungi | mfer: sure! i'm behind on circling back to check feedback on reviews where i commented | 19:04 |
| fungi | the week's been a bit crazy | 19:05 |
| mfer | fungi i understand that. i've gone long periods just hoping to catch up. | 19:05 |
| *** rossella_s has quit IRC | 19:05 | |
| *** johnthetubaguy has quit IRC | 19:06 | |
| *** mrodden1 has quit IRC | 19:06 | |
| *** Ryan_Lane2 has quit IRC | 19:06 | |
| *** Ryan_Lane2 has joined #openstack-infra | 19:07 | |
| flashgordon | fungi: I assume you already knwo but no movement in the gate queue | 19:07 |
| *** dizquierdo has quit IRC | 19:07 | |
| fungi | flashgordon: it's picking back up now. i was restarting nodepool to clear out cruft nodes which had us only using half our capacity | 19:07 |
| *** mrodden has joined #openstack-infra | 19:08 | |
| flashgordon | fungi: ack | 19:08 |
| *** dizquierdo has joined #openstack-infra | 19:08 | |
| fungi | flashgordon: it needs a few more minutes to spin up additional nodes so it can catch back up | 19:08 |
| *** mriedem has joined #openstack-infra | 19:09 | |
| flashgordon | cool wasn't sure how long ramp up was supposed to take | 19:09 |
| sdague | flashgordon: btw - https://review.openstack.org/#/c/65700/ | 19:09 |
| fungi | flashgordon: spinning up new nodes takes on the order of 15 minutes, and it gets throttled a little so we don't invoke the wrath of our generous cloud providers | 19:10 |
| sdague | that will move the overall gate stats to a dedicated page | 19:10 |
| flashgordon | sdague: woot | 19:10 |
| *** Ryan_Lane2 has quit IRC | 19:10 | |
| fungi | flashgordon: due to a gate reset when it was winding down, zuul's allocating a bunch of the initial pool to changes in the check pipeline | 19:10 |
| *** herndon_ has joined #openstack-infra | 19:11 | |
| flashgordon | fungi: huh? | 19:11 |
| sdague | oh man - I so *want* https://review.openstack.org/#/c/65993/ in :) | 19:11 |
| sdague | I'm using it locally now, so much more handy to figure out the state of things | 19:11 |
| flashgordon | so whats the time in queue for gate? | 19:12 |
| fungi | flashgordon: changes get tested first-come-first-served, so a gate reset means that all the restacked changes in the gate pipeline get to wait their turn after changes which were already in the check pipeline waiting on nodes | 19:12 |
| flashgordon | fungi: O_o oh | 19:13 |
| *** dstanek has quit IRC | 19:13 | |
| flashgordon | fungi: can we weigh gate somehow | 19:13 |
| flashgordon | actually scratch that | 19:13 |
| flashgordon | thats a can of worms | 19:14 |
| sdague | flashgordon: we did that once, then we starved out check | 19:14 |
| *** ivar-lazzaro has joined #openstack-infra | 19:14 | |
| fungi | flashgordon: we tried priorities, but it meant that when the gate got deep enough to use all available nodes, check pipeline changes sat indefinitely waiting for resources and devs never got any feedback | 19:14 |
| sdague | flashgordon: top of gate is 16hrs | 19:14 |
| flashgordon | sdague: yeah can o worms | 19:14 |
| flashgordon | sdague: 16 hours nice! | 19:14 |
| flashgordon | so the answer would be prioritize the top x in gate or something like that | 19:14 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: Capture libvirtd logs https://review.openstack.org/65833 | 19:14 |
| sdague | flashgordon: pull - https://review.openstack.org/#/c/65993/ and gvfs-open status.html | 19:15 |
| fungi | flashgordon: it might be possible to do a weighted distribution algorithm, like you said, instead of straight prioritization, but that would need new code in zuul | 19:15 |
| flashgordon | fungi: right, maybe just prioritize top 1 of gate | 19:15 |
| *** w_ is now known as olaph | 19:15 | |
| sdague | http://ubuntuone.com/2Piox3MVZ9VB4EP5AyMIFB | 19:15 |
| flashgordon | sdague: who nice! | 19:16 |
| flashgordon | fungi: but this is an unsual case and meh | 19:17 |
| fungi | flashgordon: yeah, i suspect we still have a node leak, probably manifests around jenkins master restarts, but i want to watch it closely now that i've cleaned it up and see what starts to pile up when | 19:18 |
| *** _ruhe is now known as ruhe | 19:18 | |
| flashgordon | cool, thanks for running the infra all by your self btw | 19:18 |
| fungi | but you can see on the graph now we've got far more nodes "in use" than we did a few hours ago | 19:18 |
| fungi | flashgordon: well, i'm not all by myself, just happen to be by myself for a good chunk of the day in this timezone | 19:19 |
| *** ryanpetrello has quit IRC | 19:19 | |
| flashgordon | fungi: I think the in-use graph is wrong | 19:19 |
| fungi | the other root admins have been pitching in from au as they got time (admirably missing meals and recreation i expect) | 19:19 |
| flashgordon | because those in-use numbers were like that when no new jobs were running | 19:20 |
| flashgordon | fungi: right, thank you au folks | 19:20 |
| fungi | flashgordon: the in use graph will be more right in a bit. used nodes get switched to deleting as nodepool gets a moment to catch its breath | 19:20 |
| *** ryanpetrello has joined #openstack-infra | 19:20 | |
| fungi | so those were left "in use" while nodepool was gracefully winding down, and only how has a chance to start deleting them | 19:21 |
| fungi | er, only now | 19:21 |
| flashgordon | ahh | 19:21 |
| *** dpyzhov has joined #openstack-infra | 19:23 | |
| *** sarob_ has joined #openstack-infra | 19:23 | |
| *** ryanpetrello has quit IRC | 19:25 | |
| fungi | there's several hundred previously used nodes which are in the process of being deleted and replaced by building/ready ones | 19:27 |
| *** sarob has quit IRC | 19:27 | |
| fungi | (though those quickly switch from ready to in use as they get assigned jobs) | 19:27 |
| *** sarob_ has quit IRC | 19:28 | |
| *** herndon_ has quit IRC | 19:28 | |
| *** reed has quit IRC | 19:28 | |
| flashgordon | fungi: I am amazed this stuff ever works. at this scale things are hard. its pretty amazing | 19:29 |
| * fungi couldn't agree more | 19:29 | |
| fungi | i'm keeping tabs on the nodes which were in used state prior to the nodepool restart and making sure they're dropping steadily, so it seems to be working out as intended | 19:30 |
| *** sarob has joined #openstack-infra | 19:32 | |
| fungi | hopefully with twice as many workers back into the pool, we'll burn through what's in the gate a bit faster | 19:32 |
| sdague | fungi: yeh, we all seriously owe you beers for holding down the fort by yourself this week | 19:32 |
| fungi | sdague: nah, it's always someone's turn to shore up the timbers when everyone else needs to travel | 19:32 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Collect and report last reconfigured timestamp https://review.openstack.org/63849 | 19:33 |
| fungi | i'm banking up karma for when i'm going to be on a two-week wedding travel hiatus during the release. everyone will hate me then, so this works on balance | 19:33 |
| *** dizquierdo has quit IRC | 19:33 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Add zuul_version to status.json samples https://review.openstack.org/66043 | 19:34 |
| *** dizquierdo has joined #openstack-infra | 19:35 | |
| *** mrodden has quit IRC | 19:38 | |
| *** mrodden has joined #openstack-infra | 19:39 | |
| *** praneshp has joined #openstack-infra | 19:41 | |
| *** denis_makogon_ has joined #openstack-infra | 19:43 | |
| *** denis_makogon_ is now known as denis_makogon | 19:45 | |
| *** dims is now known as dimsum | 19:45 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack-dev/hacking: Fix typo in package summary https://review.openstack.org/66046 | 19:48 |
| *** thedodd has quit IRC | 19:49 | |
| *** dpyzhov_ has joined #openstack-infra | 19:49 | |
| *** dpyzhov has quit IRC | 19:49 | |
| *** dpyzhov_ is now known as dpyzhov | 19:49 | |
| *** yolanda has quit IRC | 19:49 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Add zuul_version to status.json samples https://review.openstack.org/66043 | 19:51 |
| *** ruhe is now known as _ruhe | 19:51 | |
| *** thuc has quit IRC | 19:53 | |
| *** thuc has joined #openstack-infra | 19:53 | |
| *** ryanpetrello has joined #openstack-infra | 19:56 | |
| *** ryanpetrello has quit IRC | 19:57 | |
| sdague | fungi: how long after a pypi land will we get things in our mirror? | 19:57 |
| *** thuc has quit IRC | 19:58 | |
| *** rfolco has quit IRC | 19:59 | |
| *** thuc has joined #openstack-infra | 19:59 | |
| *** _david_ has joined #openstack-infra | 19:59 | |
| *** vipuls is now known as vipuls-away | 20:00 | |
| *** vipuls-away is now known as vipuls | 20:00 | |
| fungi | sdague: by pypi land, you mean we uploaded a release to pypi or one of our requirements uploaded a new release? | 20:00 |
| *** rockyg has joined #openstack-infra | 20:00 | |
| *** dizquierdo has quit IRC | 20:01 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make a dedicated page for gate status https://review.openstack.org/65700 | 20:03 |
| sdague | fungi: as in taskflow just hit pypi | 20:03 |
| *** ^d has quit IRC | 20:03 | |
| sdague | how long before a requirements recheck will pull that version? | 20:03 |
| *** ^d has joined #openstack-infra | 20:03 | |
| fungi | i'll have to see if they're configured to trigger a mirror refresh on release... checking | 20:04 |
| fungi | sdague: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n3905 | 20:05 |
| fungi | sdague: note that they don't have their project configured to run mirror jobs after a pypi upload in the release pipeline | 20:05 |
| sdague | ok | 20:05 |
| sdague | harlowja: can you look into doing that ^^ | 20:06 |
| *** mrmartin has joined #openstack-infra | 20:06 | |
| harlowja | k | 20:06 |
| fungi | sdague: so it will be the daily (which i think hits at 0500 utc?) or the next time someone else tags a release or merges an openstack/requirements change, whichever comes first | 20:06 |
| harlowja | fungi just 'post-mirror-python26' and such? | 20:06 |
| fungi | harlowja: yup, there are adjacent examples in that file | 20:07 |
| harlowja | k | 20:07 |
| *** mozawa has quit IRC | 20:07 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: add in the optional ; everywhere https://review.openstack.org/65703 | 20:07 |
| *** ryanpetrello has joined #openstack-infra | 20:07 | |
| *** lcestari has quit IRC | 20:07 | |
| *** dpyzhov has quit IRC | 20:09 | |
| sdague | having that time in gate right on the UI exposes some really interesting things | 20:11 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Ensure taskflow upload affects the mirrors https://review.openstack.org/66051 | 20:11 |
| sdague | like the heat change that's top of the check queue that entered almost 4 hrs ago, and is still waiting on getting a devstack node for pg | 20:11 |
| *** Ryan_Lane2 has joined #openstack-infra | 20:11 | |
| sdague | basically it looks like the top 5 check queue items are lost | 20:12 |
| fungi | sdague: yeah, there are times when jenkins fails to run a job for whatever reason and zuul readds that to the work queue, but it's waiting in line for things which came in after zuul realized that | 20:12 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Fix typo in package summary https://review.openstack.org/66046 | 20:12 |
| sdague | yeh | 20:13 |
| fungi | er, which came in before zuul realized that | 20:13 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Add openstack-doc-tools https://review.openstack.org/65361 | 20:13 |
| *** wenlock_ has joined #openstack-infra | 20:13 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Display zuul_version in zuul status page https://review.openstack.org/66052 | 20:13 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/zuul: Display last_reconfigured in zuul status page https://review.openstack.org/66053 | 20:13 |
| fungi | but the first ~ 50% of the gate pipeline has workers assigned now | 20:14 |
| *** dpyzhov has joined #openstack-infra | 20:14 | |
| sdague | yeh, it looks like under current load it's ~2hrs until you get a dsvm node | 20:14 |
| sdague | fungi: are we still bursting? | 20:14 |
| *** vipuls is now known as vipuls-away | 20:15 | |
| fungi | we're maxing out the ~500 vm aggregate quota we've set across the various providers/zones | 20:15 |
| *** herndon_ has joined #openstack-infra | 20:15 | |
| sdague | ok, so now it's straight up load | 20:15 |
| fungi | yep | 20:16 |
| fungi | actually, looks like the aggregate quota is closer to 450 | 20:16 |
| *** wenlock_ has quit IRC | 20:17 | |
| dimsum | guess we'll need the weekend to clear the queue? | 20:19 |
| flashgordon | dimsum: like the nick | 20:19 |
| dimsum | flashgordon, thanks! flashgordon brings back memories too... | 20:20 |
| fungi | i've got one hpcloud vm which is gumming up the works. nodepool delete times out on it, nova list says it's in deleting state and has been for ages, i was able to ssh into it so i did a sudo poweroff and it shut down, but nova still things it's trying to delete. weird | 20:20 |
| *** Ajaeger has joined #openstack-infra | 20:20 | |
| *** praneshp has quit IRC | 20:21 | |
| *** praneshp has joined #openstack-infra | 20:22 | |
| flashgordon | looks like we found a nova bug :) | 20:22 |
| *** Ryan_Lane2 has quit IRC | 20:22 | |
| *** sarob has quit IRC | 20:23 | |
| *** sarob has joined #openstack-infra | 20:23 | |
| *** gokrokve has quit IRC | 20:25 | |
| Ajaeger | flashgordon: found - or fixed? ;) | 20:25 |
| flashgordon | Ajaeger: found a bug in hpclouds nova | 20:26 |
| Ajaeger | Ooops ;( | 20:26 |
| *** hashar has joined #openstack-infra | 20:26 | |
| *** sarob has quit IRC | 20:28 | |
| SergeyLukjanov | sdague, here is your change about adding time in queue live - http://f.slukjanov.name/w/review.o.o/65993/1/status.html | 20:28 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: make merge conflict changes black https://review.openstack.org/66056 | 20:30 |
| fungi | flashgordon: yeah, it's been in a deleting state for 38 days according to the uptime command i ran before shutting it down to see whether that would help | 20:30 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: clean up possible js incompatibilities https://review.openstack.org/66057 | 20:31 |
| sdague | SergeyLukjanov: cool | 20:31 |
| *** prad has quit IRC | 20:32 | |
| flashgordon | fungi: if only we new people at HP | 20:32 |
| fungi | sdague: btw, that heat change finally got a worker for its one missing job run | 20:32 |
| sdague | fungi: cool | 20:32 |
| fungi | flashgordon: well, people at hp who weren't about to head out for friday afternoon beer | 20:32 |
| *** dkliban is now known as dkliban_afk | 20:33 | |
| flashgordon | we are all on the west coast now I think | 20:33 |
| fungi | yeah, it's after noon there, right? ;) | 20:33 |
| sdague | flashgordon: so I don't think I'll be able to get enough mental energy to deal with the grenade patches today. Lets just set asside a couple hours in montreal and run through them side by side | 20:33 |
| sdague | flashgordon: I thought HP just moved all your offices to delta terminals? | 20:34 |
| fungi | sdague: budget cuts... greyhound terminals | 20:34 |
| flashgordon | sdague: works for me | 20:35 |
| flashgordon | btw the grapite server seems to be failing to serve images | 20:36 |
| flashgordon | occasionally | 20:36 |
| sdague | yeh | 20:36 |
| *** changbl has joined #openstack-infra | 20:37 | |
| flashgordon | not to hammer the fungi too much | 20:37 |
| fungi | flashgordon: sdague: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=439&rra_id=all | 20:38 |
| fungi | looks like we give it quite the workout | 20:38 |
| marun | does testr run tests in parallel using multiple processes, or...? | 20:39 |
| fungi | might be i/o-bound... http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=440&rra_id=all | 20:39 |
| flashgordon | marun: multi process | 20:40 |
| sdague | SergeyLukjanov: the change I stuck on top of that also color codes out the merge conflicts as black | 20:40 |
| flashgordon | but if lifeless is around he will correct me | 20:40 |
| marun | flashgordon: danke | 20:40 |
| sdague | so you can pick them out easier from straight fails | 20:41 |
| sdague | marun: it's multi process | 20:41 |
| *** thedodd has joined #openstack-infra | 20:41 | |
| marun | sdague: i figured as much, but wanted to be sure. someone was worried about runtime configuration changes affecting other tests but that's impossible afaik | 20:41 |
| marun | (so long as teardown calls reset()) | 20:41 |
| SergeyLukjanov | sdague, I've already +1'ed it :) live http://f.slukjanov.name/w/review.o.o/66056/1/status.html | 20:42 |
| flashgordon | marun: as long as those are running in seperate threads | 20:42 |
| sdague | marun: as long as you aren't using filesystem shared state | 20:42 |
| sdague | or ports or something | 20:42 |
| flashgordon | so concurrency=4 is 4 threads | 20:42 |
| flashgordon | err processes | 20:42 |
| sdague | flashgordon: 4 processes | 20:42 |
| marun | right | 20:42 |
| sdague | marun: in the cases where we know we might have a global resource in tempest, we use the oslo lockutils | 20:43 |
| sdague | with external=True | 20:43 |
| sdague | like doing agregates manipulation | 20:43 |
| marun | sdague: external=True? presumably that uses file-based locking to ensure cross-process concurrency, but what does external=False do? | 20:43 |
| flashgordon | marun: correct file-based | 20:44 |
| marun | concurrency => synchronization | 20:44 |
| sdague | marun: external=False (which is the default) will do a lock which is safe for multi threaded environments, but not multi process | 20:44 |
| *** Ryan_Lane2 has joined #openstack-infra | 20:44 | |
| marun | sdague: ah, good to know. | 20:44 |
| fungi | so graphite... 4x vcpu, one of which is entirely pegged on iowait | 20:45 |
| *** Ryan_Lane2 has quit IRC | 20:47 | |
| *** sarob has joined #openstack-infra | 20:49 | |
| harlowja | mordred yt (question about testr + setup.py) | 20:50 |
| *** sarob_ has joined #openstack-infra | 20:51 | |
| harlowja | maybe someone else knows also | 20:51 |
| *** dprince has quit IRC | 20:52 | |
| fungi | harlowja: quite probably. may not be me, but you never know unless you ask | 20:52 |
| harlowja | ok, probably simple, so is it possible to get the colorized output via running python setup.py with a testr enabled project | 20:53 |
| harlowja | like python setup.py --verbose test --no-parallel doesn't seem to do it | 20:53 |
| harlowja | *maybe its a feature which doesn't exist yet | 20:53 |
| fungi | harlowja: i think nova has a color filter which you can feed the subunit stream through | 20:53 |
| harlowja | right, any idea if that can just be integrated into testr directly? | 20:54 |
| *** dpyzhov has quit IRC | 20:54 | |
| harlowja | so it could be like python setup.py --verbose test --no-parallel --colorized | 20:54 |
| fungi | harlowja: for the most part, sitting and staring at the process of test running is not part of the intended design of testr | 20:54 |
| harlowja | durn it, haha | 20:54 |
| *** sarob has quit IRC | 20:54 | |
| harlowja | thats why it could be an option :-P | 20:54 |
| harlowja | bb | 20:54 |
| fungi | it's geared toward running all or specific sets of tests, evaluating the results, and then has a robust set of options to rerun tests based on the results of previous runs | 20:54 |
| fungi | so you can, for example, tell it to just re-run failing tests | 20:55 |
| *** sandywalsh has quit IRC | 20:55 | |
| fungi | useful for tight iteration when debugging your code | 20:55 |
| *** ^d is now known as ^demon|away | 20:56 | |
| sdague | fungi: well given that testr regex defaults to or, you can't do that via tox in a lot of projects | 20:56 |
| fungi | sdague: true | 20:57 |
| fungi | you'd have to activate your venv and call testr directly | 20:57 |
| sdague | fungi: right | 20:57 |
| sdague | at which point I want to start stabbing people | 20:57 |
| sdague | and bring back run_tests.sh | 20:57 |
| sdague | harlowja: there are some wrappers in tempest as well | 20:58 |
| sdague | harlowja: so I think they should be fixed in testr, however testr is currently in a bzr tree, and I don't have patience for that. So I'll probably start a stackforge project with a common set of testr wrappers | 21:00 |
| flashgordon | fungi: so libvirt | 21:01 |
| flashgordon | https://review.openstack.org/#/c/65267/23 | 21:01 |
| flashgordon | we have a patch to make nova new libvirt friendly | 21:01 |
| sdague | flashgordon: did you test it on a cloud archive enabled box yet? | 21:02 |
| flashgordon | sdague: not yet, trying to get someone who has one set up to do that | 21:02 |
| fungi | i was about to ask exactly that | 21:02 |
| sdague | because while we *know* that was a fail | 21:02 |
| sdague | we don't know it's the only one | 21:02 |
| sdague | flashgordon: don't you work for a company with a public cloud... ;) | 21:02 |
| flashgordon | dimsum: ^ | 21:03 |
| *** prad has joined #openstack-infra | 21:03 | |
| fungi | sdague: well, both nova unit tests and tempest fail due to what seems like the new libvirt in uca | 21:03 |
| flashgordon | sdague: I don't know what your talking abuot ;) | 21:03 |
| fungi | so either would be a good test | 21:03 |
| fungi | both would be better | 21:03 |
| sdague | actually, I did set up a cloud archive rax guest | 21:03 |
| sdague | last night | 21:03 |
| flashgordon | sdague: awesome thanks | 21:03 |
| fungi | i have scripts to autobuild one and a stub which will add uca before running tempest, if it becomes necessary | 21:04 |
| sdague | yeh, let me at least run the unit tests | 21:04 |
| fungi | but my bandwidth is pretty low right now, so i probably wouldn't get to it right away | 21:04 |
| *** reed has joined #openstack-infra | 21:05 | |
| fungi | so, on graphite, the iowait was almost entirely the kjournald thread according to iotop. on a whim i remounted / with relatime and now the iowait is gone, and i can see that same cpu is pegged with user cycles instead, running carbon-cache.py | 21:06 |
| flashgordon | sdague: what version of cloud archive? | 21:06 |
| *** smarcet has left #openstack-infra | 21:06 | |
| flashgordon | sudo add-apt-repository cloud-archive:havana | 21:06 |
| sdague | flashgordon: havana | 21:06 |
| sdague | on 12.04 | 21:06 |
| *** sandywalsh has joined #openstack-infra | 21:06 | |
| fungi | flashgordon: i was doing this before installing other packages on the vm: http://paste.openstack.org/show/61008/ | 21:07 |
| fungi | that should be all you need to activate it | 21:08 |
| flashgordon | fungi: cool thanks | 21:08 |
| *** vipuls-away is now known as vipuls | 21:08 | |
| * flashgordon uses his free cloud resources | 21:08 | |
| fungi | adding that would cause my tempest runs to consistently fail, and i attached nova logs to the bug too if they're of interest | 21:08 |
| fungi | nova and libvirtd logs attached to https://launchpad.net/bugs/1228977 | 21:10 |
| fungi | it very well may be that the libvirt hangs were masking an incompatibility in nova, and once fixed that's where we ended up | 21:11 |
| *** Ryan_Lane2 has joined #openstack-infra | 21:12 | |
| _david_ | zaro, fixed https://code.google.com/p/gerrit/issues/detail?id=2382 : https://gerrit-review.googlesource.com/53610 | 21:12 |
| flashgordon | yeah there are two nova patches up to for new libvirt | 21:12 |
| fungi | also, the change at the head of the gate just timed out on a tempest-postgres job, resetting all ~100 changes... *sigh* | 21:13 |
| *** sarob_ has quit IRC | 21:15 | |
| flashgordon | was that a rax or hp server | 21:15 |
| fungi | flashgordon: i believe i reproduced the issue on both | 21:15 |
| *** sarob has joined #openstack-infra | 21:15 | |
| fungi | for tempest runs anyway | 21:15 |
| flashgordon | fungi: neato | 21:16 |
| flashgordon | so the question is | 21:16 |
| flashgordon | https://review.openstack.org/#/c/65360/2 | 21:16 |
| flashgordon | https://review.openstack.org/#/c/65267/23 | 21:17 |
| flashgordon | do those two fix it | 21:17 |
| fungi | in combination, or is this an either/or fix? | 21:17 |
| *** DennyZhang has joined #openstack-infra | 21:18 | |
| flashgordon | in combination | 21:18 |
| *** sarob has quit IRC | 21:19 | |
| flashgordon | fungi: I can run with the libvirt stuff if you want | 21:19 |
| *** sarob_ has joined #openstack-infra | 21:19 | |
| *** Ryan_Lane2 has quit IRC | 21:19 | |
| *** Ryan_Lane2 has joined #openstack-infra | 21:20 | |
| fungi | flashgordon: please do. i only originally weighed in on this bug because someone wanted to re-add uca to our slaves after it broke everything last time, and i wanted to make sure it didn't merge until we had confidence that wasn't going to happen again | 21:20 |
| *** Ajaeger has quit IRC | 21:20 | |
| flashgordon | yup thats why I am manaully checking | 21:20 |
| fungi | basically i asked on the review whether it had been tested, and there was silence, but it kept accumulating positive votes, and i was worried | 21:21 |
| flashgordon | fungi: which review? | 21:21 |
| fungi | it's probably been abandoned since november, don't exactly remember | 21:23 |
| flashgordon | anyway i'll run with this, you have enough on your plate | 21:23 |
| fungi | basically i'm blocking any proposals to add uca to our slaves again until someone confirms nova unit tests and tempest work | 21:23 |
| *** CaptTofu has quit IRC | 21:24 | |
| flashgordon | fungi: that is exactly what I intend to do | 21:24 |
| *** CaptTofu has joined #openstack-infra | 21:24 | |
| fungi | for the tempest test, i did https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n100 but with that snipped i pasted you earlier before installing git | 21:25 |
| fungi | s/snipped/snippet/ | 21:25 |
| arosen | Hi, I'm running into this error today trying to install nova's test-requirements.txt i was wondering if anyone has hit this yet: http://codepad.org/HABSpojX | 21:26 |
| fungi | arosen: looks like maybe pylint released a broken package? | 21:27 |
| fungi | "unknown encoding: IBO-8859-1" | 21:27 |
| *** mgagne1 has quit IRC | 21:27 | |
| fungi | yeah, i don't know that encoding either | 21:27 |
| fungi | i assume someone fat-fingered ISO-8859-1 | 21:28 |
| arosen | yea, i saw that.. I guess the gate isn't hitting it because it hasn't pulled in the latest pylint and it's using a cached copy? | 21:28 |
| *** SergeyLukjanov has quit IRC | 21:29 | |
| fungi | quite possible--we may want to emergency cap it. looking to see if 0.25.0 is shiny-new | 21:29 |
| arosen | Hrm nova specifies pylint==0.25.2 | 21:29 |
| fungi | yeah, current pylint is weeks old 1.1.0 | 21:29 |
| fungi | maybe a new release of something else specified a broken pylint as a requirement | 21:30 |
| fungi | oh | 21:30 |
| fungi | nova | 21:30 |
| fungi | sitepackages=true there | 21:30 |
| fungi | "Found existing installation: pylint 0.25.0" | 21:30 |
| fungi | sdague: example of what i was talking about the other day ^ | 21:31 |
| sdague | yeh | 21:31 |
| sdague | it is another pip frustration | 21:31 |
| fungi | nova says it requires pylint==0.25.2 but the local 0.25.0 on the system gets used instead | 21:31 |
| sdague | yep | 21:32 |
| fungi | arosen: try uninstalling your system package of pylint | 21:32 |
| arosen | If i remove pylint 0.25.0 i get this error: http://codepad.org/W0837GFr | 21:32 |
| sdague | it completely sucks that pip under tox can't give you the right thing | 21:32 |
| *** starmer has joined #openstack-infra | 21:33 | |
| zaro | _david_: cool. thanks. i'll give a try | 21:33 |
| fungi | arosen: i'd suggest `fing /usr -name "*pylint*"` and see if there's bits left behind | 21:34 |
| fungi | er, find | 21:34 |
| _david_ | zaro, Fixing gerrit-javamelody plugin now: https://gerrit-review.googlesource.com/#/admin/projects/plugins/javamelody | 21:34 |
| *** starmer has quit IRC | 21:34 | |
| fungi | arosen: because you're installing pylint into a virtualenv and it's complaining about pylint imports it found in /usr/local | 21:34 |
| fungi | arosen: oh, wait, you're not | 21:35 |
| fungi | you're installing them system-wide with pip instead | 21:35 |
| *** vipuls is now known as vipuls-away | 21:36 | |
| *** vipuls-away is now known as vipuls | 21:36 | |
| fungi | arosen: so i guess pip uninstall pylint and then use find to go on a hunting expedition for pieces which might have been left behind | 21:36 |
| *** Ryan_Lane2 has left #openstack-infra | 21:36 | |
| fungi | though i think the "package init file './test/__init__.py' not found (or not a regular file)" is probably the real problem | 21:38 |
| arosen | fungi: yup same thing: http://codepad.org/Rsa0cWhw see line 1 shows i have no more pylint | 21:38 |
| fungi | oh, or maybe that's just a stray, benign message | 21:39 |
| arosen | fungi: the problem is it causes testtools not to be installed. | 21:40 |
| fungi | it's complaining about syntax errors in /usr/local/lib/python2.7/dist-packages/pylint/test/input/func_w0705.py which was not there before you started to install according to your find comment | 21:40 |
| *** ryanpetrello has quit IRC | 21:40 | |
| fungi | so either that file is from a corrupt archive, or your interpreter is wrong | 21:41 |
| *** ryanpetrello has joined #openstack-infra | 21:41 | |
| fungi | when it's done it claims "Successfully installed pylint testrepository testtools" though | 21:42 |
| fungi | so maybe the syntax errors in the pylint test are really there upstream in that version, and they just get ignored (but complain loudly at you anyway)? | 21:42 |
| arosen | fungi: thanks i fixed it. There was actually a bug in the code //usr/local/lib/python2.7/dist-packages/pylint/test/input/func_w0705.py once i fixed the that it worked.... Very odd. | 21:45 |
| fungi | yeah, that's a little strange | 21:46 |
| *** dstanek has joined #openstack-infra | 21:46 | |
| *** DennyZhang has quit IRC | 21:46 | |
| *** DennyZhang has joined #openstack-infra | 21:47 | |
| fungi | maybe you want to check the tarball against the checksum on pypi to make sure it's not been deviled with | 21:47 |
| fungi | https://pypi.python.org/simple/pylint/ says pylint-0.25.2.tar.gz should have an md5sum of d878d7688a4f5290dc5b53a836872400 | 21:49 |
| fungi | you should probably check your pip cache to see if you get the same | 21:49 |
| * fungi gets weirdly paranoid about things like that | 21:50 | |
| *** vipuls is now known as vipuls-away | 21:51 | |
| arosen | fungi: Hrm i'm not finding the pip cache where is that? | 21:51 |
| fungi | arosen: i'm guessing /root/.pip/cache/ but if you don't have one, then pip's probably not caching downloads | 21:53 |
| fungi | could also be some artifacts in /tmp/pip-build-root | 21:54 |
| arosen | root@arosen-desktop:~/.pip_download_cache# md5sum https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpylint%2Fpylint-0.25.2.tar.gz | 21:54 |
| arosen | d878d7688a4f5290dc5b53a836872400 https%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fp%2Fpylint%2Fpylint-0.25.2.tar.gz | 21:54 |
| arosen | Yea it matches the hash | 21:54 |
| fungi | aha. and okay, good | 21:55 |
| fungi | so it's probably just a bug in 0.25.2 which is spewing nonsense during install but gets ignored by pip | 21:55 |
| arosen | Had to remove it and set export PIP_DOWNLOAD_CACHE . Gets me into the same situation where i need to edit /usr/local/lib/python2.7/dist-packages/pylint/test/input/func_w0705.py | 21:55 |
| fungi | the errors all seemed to be in unit tests for pylint, not in actual run-time code | 21:55 |
| fungi | so i'm guessing pip is running the pylint's provided test suite, which is broken, but pip could care less anyway | 21:56 |
| fungi | or rather the setup.py for pylint is running the test suite, after being invoked by pip, but you get the idea | 21:57 |
| fungi | and setup.py probably spews test errors on stdout/stderr but still exits 0 | 21:57 |
| *** CaptTofu has quit IRC | 21:58 | |
| *** gokrokve has joined #openstack-infra | 21:58 | |
| *** CaptTofu has joined #openstack-infra | 21:58 | |
| *** che-arne has quit IRC | 21:59 | |
| *** jasondotstar has quit IRC | 21:59 | |
| *** eharney has quit IRC | 22:03 | |
| *** mrmartin has quit IRC | 22:07 | |
| fungi | we're using 83% of the 315gb root partition on graphite. checking to see why/where | 22:08 |
| *** DennyZhang has quit IRC | 22:10 | |
| *** iccha has joined #openstack-infra | 22:12 | |
| *** sarob_ is now known as sarob | 22:12 | |
| *** pballand has joined #openstack-infra | 22:14 | |
| *** dimsum has quit IRC | 22:15 | |
| pballand | after starting from a cookie cutter project, how does one configure the project to copy appropriate source to .tox (running unit tests via 'tox -epy27' ? | 22:15 |
| *** jgrimm has quit IRC | 22:16 | |
| sdague | fungi: so there are a ton of pg jobs that are getting killed because of timeouts | 22:16 |
| sdague | when they were about 2minutes from success | 22:16 |
| sdague | that's what reset the whole queue just now | 22:16 |
| sdague | but you can see a bunch of it in the check queue as well | 22:17 |
| pballand | never mind - it must have been something in my environment; please disregard my previous question | 22:17 |
| fungi | pballand: disregarded! | 22:17 |
| fungi | most of them happening on rax? we probably just need to bite the bullet and up the timeouts there unless someone can make the job run faster? | 22:17 |
| fungi | sdague: ^ | 22:18 |
| flashgordon | can't we yell at rax a bit for fun too? | 22:18 |
| sdague | no it's all hp nodes | 22:18 |
| flashgordon | sdague: interesting | 22:19 |
| sdague | we need to stop blaming rax, hp nodes are taking 50 minutes to run a tempest run today | 22:19 |
| * flashgordon looks at the slowest tempest test list | 22:19 | |
| *** burt has quit IRC | 22:19 | |
| fungi | sdague: any particular az? | 22:20 |
| sdague | fungi: no, it's be on a bunch of them | 22:21 |
| *** mfer has quit IRC | 22:21 | |
| fungi | huh. that's... stellar | 22:21 |
| flashgordon | 2014-01-10 22:13:36.408 | tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,image,network,volume] 265.925 | 22:21 |
| flashgordon | lol | 22:21 |
| fungi | still, my question stands. do we up the timeout, or is this a legit issue? | 22:21 |
| fungi | (legit issue with the tests/job) | 22:22 |
| fungi | guess that can also be all of the above | 22:22 |
| *** alexpilotti has joined #openstack-infra | 22:22 | |
| flashgordon | make tempest faster list http://paste.openstack.org/raw/61010/ | 22:22 |
| fungi | it's possible the runtime on tempest has just been slowly creeping up with new jobs, and we're finally bumping up against the ceiling there? | 22:23 |
| flashgordon | fungi: jenkins plots that right? | 22:23 |
| *** hashar has left #openstack-infra | 22:23 | |
| fungi | flashgordon: not long-term (we run too many jobs, so i think we only keep a day history) | 22:24 |
| fungi | i'm double-checking though | 22:24 |
| flashgordon | jenkins data is too dirty | 22:25 |
| flashgordon | https://jenkins02.openstack.org/job/gate-tempest-dsvm-postgres-full/buildTimeTrend | 22:25 |
| flashgordon | do we graphite this? | 22:25 |
| flashgordon | or logstash | 22:25 |
| * flashgordon checks logstash | 22:25 | |
| fungi | oh, also jenkins trending is worthless because of the way we're constantly aborting/cancelling jobs in the gate | 22:25 |
| openstackgerrit | Aaron Greengrass proposed a change to openstack-infra/config: Remove hardcoded config assumptions, cleanup variables https://review.openstack.org/66072 | 22:26 |
| fungi | graphite's probably a slow angle of investigation at the moment given the load/iowait there, but i also think we maybe haven't been trending job duration yet (i could be wrong) | 22:26 |
| flashgordon | message:"tests in" AND message:"Ran" AND build_name:"check-tempest-dsvm-postgres-full" | 22:26 |
| flashgordon | spot check is showing things are worse | 22:28 |
| sdague | fungi: if you filter jenkins by only success runs, it's ok | 22:28 |
| sdague | that's what my scraper used to do | 22:28 |
| sdague | but with this many aborts in there, it's a little tougher to get good data | 22:29 |
| flashgordon | message:"tests in" AND message:"Ran" AND build_name:"check-tempest-dsvm-postgres-full" AND build_status:"SUCCESS" | 22:29 |
| flashgordon | works pretty well | 22:29 |
| flashgordon | a lot of jobs around 2000 seconds | 22:29 |
| fungi | which is still well under the timeout | 22:30 |
| Alex_Gaynor | mordred: Where do I file bug reports against pbr? Specifically if try ot install a package that depends on pbr, and you don't have git installed you get a horrendous error message "file does not exist" | 22:30 |
| flashgordon | fungi: thats just tempest not setup | 22:30 |
| fungi | ohhhhh | 22:30 |
| fungi | yeah then it's probably very close | 22:30 |
| *** katyafervent has quit IRC | 22:30 | |
| *** katyafervent has joined #openstack-infra | 22:31 | |
| fungi | Alex_Gaynor: that sounds familiar--i thought it was fixed, maybe just not yet released. anyway https://bugs.launchpad.net/pbr/ | 22:31 |
| flashgordon | 2231 tests today | 22:31 |
| flashgordon | 2193 around new years | 22:31 |
| *** jerryz has quit IRC | 22:32 | |
| fungi | Alex_Gaynor: anyway, what are you doing with a machine with no git installed? don't they just embed it in the kernel these days ;) | 22:32 |
| *** dimsum has joined #openstack-infra | 22:32 | |
| Alex_Gaynor | fungi: Totally empty docker image | 22:33 |
| flashgordon | not a big bump in tests | 22:33 |
| fungi | flashgordon: so maybe openstack got slower | 22:34 |
| fungi | flashgordon: or maybe openstack-as-run-by-one-of-our-providers got slower | 22:34 |
| flashgordon | or something in devstack got slower | 22:35 |
| flashgordon | why is this seen more in postgres? | 22:35 |
| flashgordon | sdague: whats your take on bumping the timeout? | 22:36 |
| sdague | flashgordon: +1 on bumping timeouts | 22:36 |
| flashgordon | it may be a decent short term solution | 22:36 |
| flashgordon | sdague: by how much 5 minutes? | 22:37 |
| sdague | remember though that you need to handle it in the inner script as well | 22:37 |
| sdague | if you are going to bump them, add 30 minutes | 22:37 |
| * flashgordon looks at fungi | 22:37 | |
| flashgordon | we should also look into getting these tests faster http://paste.openstack.org/raw/61010/ | 22:37 |
| fungi | flashgordon: i can bump the timeout(s both places) | 22:38 |
| fungi | i'll set them to 95 outer, 90 inner i guess | 22:38 |
| fungi | just postgres or are we flying into the danger zone anywhere else obvious? | 22:39 |
| flashgordon | sdague: oh watcher of zuul | 22:40 |
| *** vipuls-away is now known as vipuls | 22:40 | |
| *** dangers is now known as dangers_away | 22:41 | |
| sdague | flashgordon: well, some of those take real time | 22:42 |
| flashgordon | sdague: agreed, but how much time | 22:42 |
| sdague | that amount of time | 22:42 |
| *** thomasem has quit IRC | 22:43 | |
| sdague | grrr... I've been waiting around for 15 minutes now to get a 10th jenkins gerrit event | 22:43 |
| sdague | flashgordon: after a week of frustration, I'm getting all the fundamentals addressed in ER, like actually having frameworks for spitting up fake, but real looking, gerrit events, so we can actually test everything as unit tests in ER | 22:45 |
| sdague | so I can get this passing - https://review.openstack.org/#/c/65961/ | 22:46 |
| flashgordon | sdague: sweet | 22:46 |
| flashgordon | we can have a review party next week and get that all in | 22:46 |
| sdague | sounds good | 22:47 |
| *** oubiwann has quit IRC | 22:47 | |
| sdague | that will let us really easily add more jobtypes | 22:47 |
| sdague | actually, it defaults to working on all the job types, we can add the readiness files to make them less likely to miss | 22:48 |
| fungi | okay, so bump postgres-full to 95(90) and... do we care about postgres (non-full) timeout? | 22:48 |
| sdague | I think it's functionally correct, but our unit tests are such a wreck that they require uber hackiness | 22:48 |
| sdague | fungi: I don't think so | 22:49 |
| fungi | k, patch inbound | 22:49 |
| *** prad has quit IRC | 22:49 | |
| sdague | also, glanceclient is failing unit tests a lot | 22:49 |
| flashgordon | sdague: I noticed that lolz | 22:49 |
| flashgordon | thats embarrassing for them | 22:50 |
| sdague | flashgordon: I don't think they know | 22:50 |
| flashgordon | thats even worse | 22:52 |
| pballand | are 3rd party packages cached somewhere? (How do the build system/developers deal with 3rd-party packages in pypi where the server is down?) | 22:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Increase gate tempest-dsvm-postgres-full timeout https://review.openstack.org/66076 | 22:53 |
| *** prad has joined #openstack-infra | 22:53 | |
| fungi | sdague: flashgordon: if you'll both +1 ^ i'll self-approve it in | 22:53 |
| *** markmcclain has quit IRC | 22:53 | |
| lifeless | flashgordon: hi? | 22:54 |
| fungi | pballand: it depends on the project. for openstack official project jobs, we force them to use a limited-content pypi mirror we maintain with a set of automated jobs. for stackforge projects and some leaf infra/dev projects we just hit pypi directly and deal with the false negative results when it goes down | 22:54 |
| *** alexpilotti has quit IRC | 22:55 | |
| flashgordon | lifeless: o/ I used your name in vane with regard to a question about testr | 22:55 |
| flashgordon | vain | 22:55 |
| sdague | fungi: done | 22:55 |
| fungi | vein too ;) | 22:55 |
| flashgordon | fungi: done | 22:55 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Bump version of sphinxcontrib-pecanwsme https://review.openstack.org/66077 | 22:56 |
| sdague | fungi: it would be nice if we could automate that inner / outer part | 22:56 |
| fungi | complicity confirmed | 22:56 |
| pballand | fungi: that makes sense. Any recommendations for a stackforge project that relies on a 3rd party package from a flaky server? (is it acceptable to cache the package in the project?) | 22:56 |
| sdague | any chance the yaml values could be sent down automatically? | 22:56 |
| fungi | sdague: agreed, it could probably be done with a parameter and some shell math or pipe through bc if you want to get old-school | 22:56 |
| flashgordon | so gate delay is 16 hours not too bad | 22:57 |
| anteaya | flashgordon: you were correct the first time, vane | 22:57 |
| anteaya | I'm wrong, vain | 22:57 |
| anteaya | http://homepage.smc.edu/quizzes/cheney_joyce/vainvanevein.html | 22:58 |
| flashgordon | I just spent the last 5 minutes googling that too | 22:58 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack/requirements: Bump version of sphinxcontrib-pecanwsme https://review.openstack.org/66077 | 22:58 |
| *** madmike has quit IRC | 22:58 | |
| fungi | pballand: well, pypi has gotten much more reliable now that they've put their service behind a cdn. what package? convince them to start uploading directly to pypi if they don't (otherwise pip 1.5 will also need scary-looking options to install them anyway) | 22:59 |
| pballand | fungi: the package is antlr3: https://pypi.python.org/pypi/antlr_python_runtime/3.1.2 | 23:00 |
| *** CaptTofu has quit IRC | 23:01 | |
| fungi | sdague: for bonus points, in devstack-gate we could find a way to check the job start time against when the timeout command gets invoked and take the difference into account, so we only need fudge time for file uploads at the end and and whatnot | 23:01 |
| fungi | pballand: yep... https://pypi.python.org/simple/antlr_python_runtime/ | 23:02 |
| fungi | pballand: talk them into uploading to pypi | 23:02 |
| anteaya | flashgordon: :D | 23:02 |
| fungi | pballand: right now, with latest pip, you'd have to add --allow-external antlr_python_runtime --allow-unverified antlr_python_runtime | 23:02 |
| _david_ | zaro, https://gerrit-review.googlesource.com/#/c/53623 | 23:03 |
| fungi | pballand: basically, packages which can only be installed through external pypi linking need to get publicly shamed into uploading their tarballs. if some of the pip/pypi maintainers had their way, that would already be disallowed | 23:04 |
| *** thuc has quit IRC | 23:04 | |
| *** thuc has joined #openstack-infra | 23:05 | |
| *** CaptTofu has joined #openstack-infra | 23:06 | |
| *** banix has quit IRC | 23:08 | |
| *** thuc has quit IRC | 23:09 | |
| fungi | pballand: and it's not just for the sake of stability. most of those externally-linked packages are just served up via plain http, so no endpoint validation to even confirm you're not the victim of a man-in-the-middle attack feeding you a compromised piece of software | 23:10 |
| pballand | fungi: okay, there are already concerns regarding the stability of that package - there is a history of introducing bugs in new versions, so perhaps caching the exact version of the code is the best way forward for now | 23:11 |
| fungi | pballand: or switching to an alternative implementation if there is one | 23:11 |
| *** jerryz has joined #openstack-infra | 23:12 | |
| *** _david_ has quit IRC | 23:12 | |
| fungi | pballand: or going rogue and uploading a slightly renamed fork of it to pypi, then depending on that, if they're stubborn and you're so inclined | 23:12 |
| pballand | fungi: :-) | 23:13 |
| fungi | pballand: SpamapS recently (a couple days ago) had luck talking the netaddr devs into uploading their releases to pypi, so it's not necessarily a futile endeavor. some upstreams are just unaware of what's going on in the present python packaging revolution | 23:14 |
| *** pcrews has quit IRC | 23:15 | |
| pballand | fungi: I'll give it a shot; nice to know the alternatives if it doesn't work out | 23:16 |
| *** GheRivero_afk is now known as GheRivero | 23:16 | |
| *** rcleere has quit IRC | 23:16 | |
| lifeless | flashgordon: ah, so yes you are correct | 23:16 |
| pballand | fungi: first, I need to wait for the site to come back online to find the author :-/ | 23:18 |
| fungi | pballand: yeah, that sucks. maybe if you mention to them that had they uploaded copies directly to pypi like most projects, you could have continued to install their software while they were busy dealing with (hard drive crash|power outage|law enforcement) | 23:22 |
| *** herndon_ has quit IRC | 23:22 | |
| *** rnirmal has quit IRC | 23:22 | |
| flashgordon | fungi: so with libvirt 1.1.1 I couldn't get nova-cpu to crash | 23:23 |
| *** thedodd has quit IRC | 23:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Increase gate tempest-dsvm-postgres-full timeout https://review.openstack.org/66076 | 23:25 |
| fungi | flashgordon: was that with or without those patches? | 23:28 |
| david-lyle | I'm trying to upload a new version of django_openstack_auth to pypi, can someone add me as a maintainer? I've pushed a tag on the gerrit side, now I need to publish. current maintainers are gabrielhurley and openstackci | 23:28 |
| fungi | david-lyle: jenkins should have uploaded it for you automatically. i'll look into why it didn't | 23:29 |
| *** ryanpetrello has quit IRC | 23:30 | |
| david-lyle | fungi: thanks, didn't realize that was automated | 23:31 |
| fungi | david-lyle: 1.1.4 right? | 23:31 |
| david-lyle | yes | 23:31 |
| fungi | looks properly signed | 23:31 |
| fungi | i don't see the tarball at http://tarballs.openstack.org/django_openstack_auth/ | 23:32 |
| fungi | checking into the tarball job | 23:32 |
| david-lyle | I may have confirmed my key after pushing the tag, don't know if that would cause issues | 23:32 |
| *** GheRivero is now known as GheRivero_afk | 23:33 | |
| fungi | shouldn't | 23:34 |
| *** CaptTofu has quit IRC | 23:34 | |
| *** pballand has quit IRC | 23:34 | |
| fungi | david-lyle: it's probably just zuul being backed up, since its in the throes of a fairly large gate reset right now and i don't see any signs the job got triggered on any of our jenkins masters yet. i'll check the zuul log to make sure it's getting enqueued | 23:36 |
| david-lyle | fungi: ok, I thought I would have to push, so was looking for a way to do that, but I understand zuul is loaded right now, wasn't trying to be impatient | 23:37 |
| *** bauzas has quit IRC | 23:37 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: wip: moving readiness checks into stream https://review.openstack.org/65961 | 23:37 |
| fungi | david-lyle: no problem at all, i want to make sure the automation is actually working as designed (but it seems to have successfully uploaded the previous releases, so i expect it's probably okay) | 23:38 |
| sdague | well the check queue is basically empty, that's good to no longer have things there for hours | 23:39 |
| *** rockyg has quit IRC | 23:41 | |
| *** vipuls is now known as vipuls-away | 23:44 | |
| *** UtahDave has quit IRC | 23:47 | |
| jeblair | fungi: what happened to nodepool? | 23:48 |
| *** banix has joined #openstack-infra | 23:49 | |
| *** mrodden has quit IRC | 23:49 | |
| sdague | fungi: btw - I updated the indentation on https://review.openstack.org/#/c/65700/2/modules/openstack_project/files/gatestats/index.html | 23:51 |
| fungi | jeblair: roughly 50% of the aggregate quota were between 8 hours and 4 days old | 23:52 |
| jeblair | fungi: ready or used? | 23:52 |
| fungi | jeblair: one vm was undeletable, in hpcloud, stuck in deleting state according to novaclient (for over a month since i logged into it and uptime said 38 days before i shut it down) | 23:52 |
| fungi | jeblair: a mix | 23:53 |
| fungi | some claimed to be in ready, some delete, some used | 23:53 |
| *** sarob has quit IRC | 23:53 | |
| *** sarob has joined #openstack-infra | 23:54 | |
| fungi | i did a graceful shutdown and deleted all the nodes which were more than a few hours old since they clearly weren't picking up any jobs. that put the gate behind a couple hours all total | 23:54 |
| fungi | though the graceful shutdown never completed. i nodepool deleted all the non-used nodes at the end, except for the one which would never delete, then killed nodepoold and cleared the leftover pidfile | 23:55 |
| fungi | the undeletable hpcloud slave is still there... devstack-precise-hpcloud-az2-777371 | 23:56 |
| jeblair | fungi: you cane run 'nodepool delete' on old nodes without stopping nodepool in the future | 23:56 |
| fungi | jeblair: right, i'm aware. i was worried though since it wasn't deleting them on its own, dating back to around the jenkins master restarts | 23:57 |
| fungi | it was deleting some, but had a bunch which the cleanup thread should have taken care of | 23:57 |
| fungi | so i didn't want them piling right back up again | 23:57 |
| *** sarob has quit IRC | 23:57 | |
| *** wenlock has quit IRC | 23:58 | |
| fungi | though the graph leading up to when i started working on it shows a marked decline in capacity (slow trending increase in both building and deleting count, leaving a narrower and narrower envelope of in use nodes) | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!