| Krinkle | mordred: Tell me more about zuulv3? | 00:04 |
|---|---|---|
| *** dboik has joined #openstack-infra | 00:04 | |
| mordred | Krinkle: jeblair wrote an email to the list - lemme find a link | 00:04 |
| *** dims has quit IRC | 00:04 | |
| mordred | Krinkle: http://lists.openstack.org/pipermail/openstack-infra/2015-February/002471.html | 00:05 |
| Krinkle | mordred: I suppose one theory would be to have all jobs be a generic "shell" and send the whole thing as a parameter? | 00:05 |
| *** dboik_ has joined #openstack-infra | 00:05 | |
| mordred | Krinkle: nope. going much further down the rabbit hole than that | 00:05 |
| mordred | Krinkle: this is part of operation "get rid of jenkins" | 00:05 |
| Krinkle | mordred: I like the sound of that | 00:06 |
| Krinkle | mordred: We're seeing scale issues at Wikimedia as well. | 00:06 |
| mordred | it has scale issues, it breaks all the time - and all we use it for is remote shell execution | 00:06 |
| mordred | it's kinda heavyweight for that | 00:06 |
| Krinkle | mordred: I think our average build complexity is lower (and we're not using nodepool yet, though soon will be). But possibly our project/job count is much higher. | 00:07 |
| fungi | zuul v2 (e.g. gearman support) came about because we couldn't scale jenkins | 00:07 |
| mordred | Krinkle: the project/job structure intended for zuul v3 would be potentially especially nicer for you | 00:07 |
| mordred | Krinkle: instead of doing templating to expand out all of the project/job possibilities | 00:07 |
| Krinkle | One thing we did recently is consolidate our jobs a lot. E.g. from "project-foo-task-bar" to "task-bar" triggered as different ZUUL_PROJECt. | 00:08 |
| *** tiswanso has joined #openstack-infra | 00:08 | |
| mordred | Krinkle: we're hoping to have, for instance, a job called "python-27-unittests" | 00:08 |
| Krinkle | So that we didn't have 2000 mwext-{name}-jshint jobs | 00:08 |
| mordred | yah | 00:08 |
| mordred | that'll be fairly well baked in as a design concept | 00:08 |
| Krinkle | mordred: Which also amounted to an enormous workspace leak | 00:08 |
| *** dboik has quit IRC | 00:09 | |
| jeblair | yeah, that's a jenkins-ism that got baked into zuulv2 that i want to get rid of | 00:09 |
| Krinkle | which in turn was somewhat desirable since we don't want to re-clone every build (not until we have better gerrit replication) | 00:09 |
| jeblair | (it also has implications for how zuul builds shared change queues) | 00:09 |
| *** achanda has joined #openstack-infra | 00:09 | |
| Krinkle | The majority if our build now no longer use zuul-cloner but instead jenkins git plugin as shallow. And re-uses the workspaces for different projects. | 00:09 |
| Krinkle | Much lighter weight | 00:09 |
| Krinkle | But for more complex builds that need other projects checked out we obviously need the power of zuul-cloner and its references etc. | 00:10 |
| jeblair | Krinkle: if you use nodepool, you'll be able to pre-cache git repos like we do which should make it easier to use zuul-cloner everywhere | 00:10 |
| mordred | Krinkle: yah - that's one of the nice parts about us pre-caching all of our repos into our base images with nodepool :) | 00:10 |
| mordred | jeblair: jinx | 00:10 |
| jeblair | :) | 00:10 |
| mordred | jeblair: OR ... OR ... | 00:10 |
| mordred | jeblair: AFS | 00:10 |
| Krinkle | jeblair: Right, that's the unintended side-effect we ran into this month. Our disks are no longer full, but now our gate pipeline is useless | 00:10 |
| Krinkle | Zuul thingks everything depends on everything because they share a common job | 00:11 |
| *** medieval1 has joined #openstack-infra | 00:11 | |
| jeblair | yup | 00:11 |
| mordred | oh wow | 00:11 |
| mordred | yeah | 00:11 |
| fungi | hrm, we may need to roll back the nodepool upgrade. not sure yet which patch is to blame since we jumped forward at least several weeks worth of new commits just now, but seeing a lot of http://paste.openstack.org/show/198002/ | 00:12 |
| Krinkle | We're fighting fires everyday (usuaully the same fires). It's getting rather frustrating | 00:12 |
| *** FatBack has quit IRC | 00:13 | |
| Krinkle | jeblair: Yeah. How does that work exactly? Do you have nodepool create the snapshot-template with a complete gerrit/* replication on-board? | 00:13 |
| fungi | i'm going to git blame around jenkins.node_exists() and things calling into it to see if anything's changed in that vicinity | 00:13 |
| jeblair | Krinkle: so we won't have that assumption in v3, change queues will be manually specified. if you need that sooner, that's probably not a very hard patch to write for v2 | 00:13 |
| mordred | Krinkle: I will give you link - one sec ... | 00:13 |
| Krinkle | jeblair: Yeah, that's what we thought too. Specify dependnt projects explicitly. | 00:13 |
| Krinkle | Looking forward to that, eagerly. | 00:13 |
| Krinkle | :) | 00:13 |
| jeblair | fungi: we may also have upgraded python-jenkins | 00:13 |
| mordred | Krinkle: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/cache_git_repos.py | 00:14 |
| fungi | jeblair: good point, i'll double check the log for what got upgraded there | 00:14 |
| mordred | Krinkle: obviously that's a bit hardcoded for us - but not terribly hard to change | 00:14 |
| mordred | Krinkle: we run that in our image builds | 00:14 |
| mordred | Krinkle: I have a diskimage-builder element version of it too | 00:14 |
| mordred | Krinkle: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/openstack-repos | 00:15 |
| mordred | jeblair: why does something break every time we update anything that touches a jenkins | 00:15 |
| *** medieval1 has quit IRC | 00:16 | |
| jeblair | fungi: it's also possible that it's not more broken than before | 00:16 |
| jeblair | fungi: i am seeing successful node deletes | 00:17 |
| fungi | jeblair: i'm checking the nodes it's complaining about | 00:17 |
| fungi | i think they may be missing from the jenkins masters but still in nodepool's dv | 00:17 |
| fungi | db | 00:17 |
| fungi | there are only 13 but they're repeating as it retries | 00:17 |
| mordred | jeblair: oh wow. I didn't think about it before - but we get task counters for jenkins operations now too | 00:19 |
| jeblair | oh yeah | 00:19 |
| fungi | no real commonality between them, but i'm going to see if i can manually clear them out of the db and see if we grow new ones (some have been in a delete state for a week) | 00:19 |
| *** tiswanso has quit IRC | 00:19 | |
| Krinkle | mordred: Cool. Thanks :) | 00:20 |
| * Krinkle saves links | 00:20 | |
| jeblair | fungi: yeah, they have no 'jenkins' name in the db, so nodepool is correctly passing None | 00:20 |
| Krinkle | We're still creating new tasks, but here's roughly our board to implement nodepool in our established infrastructure: https://phabricator.wikimedia.org/tag/continuous-integration-isolation/ | 00:20 |
| jeblair | fungi: i think the thing to look for is perhaps a change in whether nodepool would have skipped the jenkins removal based on the DB value | 00:20 |
| fungi | jeblair: odd. they have a target in the nodepool list output | 00:21 |
| jeblair | fungi: i think your 'delete db records' approach is good for fixing immediate | 00:21 |
| jeblair | fungi: yeah but no "NodeName" | 00:21 |
| fungi | aha | 00:21 |
| fungi | yeah, i see that | 00:21 |
| fungi | okay, makes more sense now | 00:21 |
| fungi | i wonder how we accumulate those | 00:21 |
| *** dims has joined #openstack-infra | 00:22 | |
| clarkb | that moment when you discover that a set of tests have never tested what they claim to test | 00:22 |
| clarkb | or at least don't appear to currently test it | 00:22 |
| jeblair | clarkb: it's a beautiful moment. but hey, it happens all the time so ain't that special really ;) | 00:23 |
| jeblair | Krinkle: hehe, you updated https://phabricator.wikimedia.org/T87294 faster than i could confirm my email addr with phabricator :) | 00:24 |
| Krinkle | And https://phabricator.wikimedia.org/T94715 yesterday :) | 00:24 |
| *** annegentle has joined #openstack-infra | 00:24 | |
| Krinkle | I'm splitbrain on CI maintenance. If I don't write it down, I forget when I return from my other half. | 00:25 |
| *** YorikSar has quit IRC | 00:25 | |
| Krinkle | spread a bit thin this quarter. | 00:25 |
| jeblair | Krinkle: one thing you may want to do is ensure your CI account can upload images to glance. we're really focusing on using diskimage-builder and uploading the resulting images directly (rather than the image snapshot approach we're mostly using now) | 00:25 |
| mordred | Krinkle: we're very close | 00:26 |
| jeblair | Krinkle: so if you are able, i would recommend planning for that and deploying after we finish that work (as mordred says, very close). | 00:27 |
| fungi | jeblair: as suspected, the nodepool log is much cleaner since manually deleting those problem rows from the node table. i'll continue to keep an eye on it for other oddities | 00:27 |
| jeblair | Krinkle: (we're using the diskimage-builder + glance approach for some images on one of our providers now, but are still working on compatability with the other) | 00:27 |
| mordred | fungi: the new metrics are all in graphite now | 00:27 |
| fungi | pewpewmetricslasers | 00:27 |
| Krinkle | Hm.. I've not followed that part of the roadmap yet. Is there something that suggests we weren't going that route already? | 00:27 |
| mordred | timrc: ^^ | 00:27 |
| jeblair | Krinkle: no; i haven't read all your tasks yet, sorry, and i'm about to run out the door and just wanted to make sure that wasn't overlooked. :) | 00:28 |
| Krinkle | jeblair: no worries! Cool. | 00:29 |
| Krinkle | jeblair: If you're doing a review/recommendation, I'd check https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Isolation instead. that's our main roadmap for this. The tasks are a bit behind. | 00:29 |
| Krinkle | it seems that document mentions glance, not snapshot. Though I do recall something about snapshots. | 00:30 |
| Krinkle | Hm.. I'm curious what the difference is. So right your nodepool is creating an otherwise unused instance, sets it all up, and then snapshots the active state to use as boilerplate for new instances, right? | 00:31 |
| *** tkelsey has joined #openstack-infra | 00:32 | |
| mordred | that's right | 00:32 |
| fungi | Krinkle: correct. at least that was the old mechanism. we've already stopped doing that for some of our images | 00:32 |
| Krinkle | I suppose it still has to periodically recreate the base image. So what are you (going to be) doing instead? | 00:32 |
| mordred | Krinkle: we build a new base image using diskimage-builder - and then we upload the exact same image to all of our clouds | 00:33 |
| fungi | Krinkle: the new mechanism uses a tool called diskimage-builder to create images from scratch by installing things in a chroot and then converting that into a bootable image we upload into glance in our providers | 00:33 |
| Krinkle | Hm.. 'from scratch'. What base image does it use? E.g. the linux install. | 00:34 |
| *** smyers has quit IRC | 00:35 | |
| fungi | Krinkle: not necessarily any base image. instead root bootstrapping tools for each distro | 00:35 |
| fungi | e.g. debootstrap | 00:35 |
| *** TheJulia has quit IRC | 00:35 | |
| *** TheJulia_ has joined #openstack-infra | 00:35 | |
| Krinkle | Hmm I think I've reached the boundary of my server knowledge :D | 00:36 |
| timrc | mordred, fungi Woohoo | 00:36 |
| fungi | akin to automating a distro installer rather than starting from an existing install | 00:36 |
| *** smyers has joined #openstack-infra | 00:36 | |
| *** tkelsey has quit IRC | 00:37 | |
| *** Swami has quit IRC | 00:37 | |
| fungi | sort of like what happens when you install a physical server from a cd or usb stick installer, rather than writing a preinstalled image directly onto the hard drive | 00:37 |
| *** pritesh has quit IRC | 00:37 | |
| Krinkle | So in our case the wmflabs cloud (used for various production tier 2 projects, unrelated to CI) has three base images (trusty, precise, jessie). Which ops updates about every other month or so. | 00:38 |
| fungi | and this would allow you to avoid using those entirely, as long as they have a working glance service you can upload into | 00:39 |
| Krinkle | We currently have a nova project 'integration' with a couple dozen persistent instances given a puppet role in openstack that sets everything up and used as jenkins slaves. | 00:39 |
| *** gibi has quit IRC | 00:40 | |
| *** ildikov_ has quit IRC | 00:40 | |
| Krinkle | Ah, right. So nodepool would be creating the instance on its own server outside openstack (though firewall allowing, nodepool itself coudl also run on an instance I guess), and then uploads it as base image. | 00:40 |
| fungi | yep. part of what drove us to this is that the images available in our providers are not ideal for our needs, since they're tuned to support more general customer use cases | 00:40 |
| Krinkle | I don't actually know.. do base images have the same performance as snapshots (e.g. copy from a hot-running paused state) | 00:41 |
| fungi | often not, in fact. it depends entirely on how the cloud was built though | 00:41 |
| Krinkle | I guess this is fine since nodepool boots them ahead of time | 00:42 |
| fungi | some providers, we've seen, cache normal images on or close to the nova controller where they're being booted, while snapshots end up on slow/poorly-connected storage making them slower to boot as a result | 00:42 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon https://review.openstack.org/165228 | 00:42 |
| *** annegentle has quit IRC | 00:42 | |
| Krinkle | Hm.. "providers", what is that again in this context> | 00:42 |
| fungi | rackspace, hpcloud, et cetera. public cloud service providers donating accounts to us | 00:43 |
| Krinkle | Right | 00:43 |
| Krinkle | I guess we're our own provider? | 00:43 |
| *** bknudson has joined #openstack-infra | 00:43 | |
| fungi | in a sense | 00:43 |
| *** ChuckC has quit IRC | 00:44 | |
| fungi | depends on how you scope "we" (if you're not part of the cloud operations team there, then you perhaps don't have direct control over some of the things you might prefer worked differently) | 00:44 |
| Krinkle | I was referring to Wikimedia | 00:44 |
| fungi | yep | 00:45 |
| fungi | you were just referring to your ops team as if they were a separate group from the one you're in | 00:45 |
| *** shashankhegde has quit IRC | 00:45 | |
| Krinkle | No, I mean, Wikimedia's openstack cloud ("Wikimedia Labs") is run by our ops team. Not hosted at a third party provider. | 00:46 |
| fungi | right, but you're not in the ops team? so in a sense, they are your provider | 00:46 |
| openstackgerrit | liuyang1 proposed openstack-infra/git-review: Add utf-8 char support https://review.openstack.org/170324 | 00:46 |
| fungi | just an internal one | 00:46 |
| Krinkle | fungi: Sure :) | 00:46 |
| Krinkle | fungi: It's maintenance is quite viral though. | 00:46 |
| *** medieval1 has joined #openstack-infra | 00:47 | |
| *** gulic has joined #openstack-infra | 00:47 | |
| fungi | understandable. we're getting close to embarking on running one within the openstack community on donated hardware too | 00:47 |
| fungi | to augment the virtual resources we have from public service providers | 00:47 |
| Krinkle | Interesting. | 00:48 |
| Krinkle | dontated hardware central or colocation? | 00:48 |
| fungi | donated colocation and donated hands-on datacenter ops to troubleshoot/reboot hardware | 00:48 |
| Krinkle | Would be interesting to see a cloud with its instances spread across different physical clusters elsewhere. | 00:49 |
| Krinkle | I mean, I guess AWS does that.. | 00:49 |
| Krinkle | Was fun to "simply" move a couple instances from US to Asia with a few clicks. | 00:49 |
| fungi | we'll likely bring on more racks in different places as additional donors offer up similar deals to us | 00:50 |
| Krinkle | Very exciting. | 00:50 |
| fungi | but a lot of openstack-based public cloud service providers do that themselves already | 00:51 |
| Krinkle | Hm.. what kind of audience would you be targeting? I assume not free-for-all? | 00:51 |
| fungi | for example we have instances running in rackspace some of which are in virginia, some in texas, some in illinois | 00:51 |
| fungi | i expect the primary workload is going to be ci jobs | 00:52 |
| *** sdake_ has joined #openstack-infra | 00:54 | |
| *** tiswanso has joined #openstack-infra | 00:54 | |
| fungi | Krinkle: the plan for that is loosely explained at http://lists.openstack.org/pipermail/openstack-infra/2015-February/002459.html | 00:56 |
| Krinkle | Hehe, it's funny how there's more and more urls where openstack can be substituted for wikimedia and visa-versa and get almost a valid url. | 00:57 |
| *** sdake has quit IRC | 00:57 | |
| *** tiswanso has quit IRC | 00:57 | |
| *** sputnik13 has quit IRC | 00:58 | |
| *** tiswanso has joined #openstack-infra | 00:58 | |
| fungi | we use and run a lot of the same tools, and chose some reasonably obvious hostnames for them ;) | 00:58 |
| fungi | also, we run some of your software, so it's only fair you should run some of ours ;) | 00:59 |
| *** JayJ_ has joined #openstack-infra | 00:59 | |
| *** JayJ_ has quit IRC | 00:59 | |
| *** redrobot has quit IRC | 00:59 | |
| *** sdake has joined #openstack-infra | 01:00 | |
| *** tteggel_ has quit IRC | 01:00 | |
| *** woodster_ has quit IRC | 01:00 | |
| *** asahlin has quit IRC | 01:00 | |
| *** mordred has quit IRC | 01:01 | |
| *** jgrimm has quit IRC | 01:01 | |
| *** asahlin has joined #openstack-infra | 01:01 | |
| *** tteggel has joined #openstack-infra | 01:01 | |
| *** Krinkle has quit IRC | 01:02 | |
| *** gilliard has quit IRC | 01:02 | |
| *** jgrimm has joined #openstack-infra | 01:02 | |
| *** JoshNang has quit IRC | 01:02 | |
| *** ildikov_ has joined #openstack-infra | 01:03 | |
| *** sdake_ has quit IRC | 01:03 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/os-loganalyze: Have swift pass through non log data unoutched https://review.openstack.org/170234 | 01:03 |
| openstackgerrit | Clark Boylan proposed openstack-infra/os-loganalyze: Fix swiftclient tests https://review.openstack.org/170326 | 01:03 |
| clarkb | jhesketh: ^ your review on the fix swiftclient tests is much appreciated | 01:04 |
| clarkb | the other change sdague wants to hold off on a bit but in my work to test it discovered that the swift tests weren't working correctly | 01:04 |
| *** JoshNang has joined #openstack-infra | 01:06 | |
| *** redrobot has joined #openstack-infra | 01:06 | |
| fungi | i'm cleaning up lingering alien nodes now. nodepool seems to be operating within normal expectations, so i'm going to transition to more eveningish endeavors, but will continue to keep an eye on it for a while and check back in on it first thing when i wake up tomorrow | 01:06 |
| *** mordred has joined #openstack-infra | 01:06 | |
| *** redrobot is now known as Guest48074 | 01:06 | |
| *** gilliard has joined #openstack-infra | 01:07 | |
| *** stevemar has quit IRC | 01:07 | |
| fungi | clarkb: if you feel up for +2'ing 170314, and assuming everything still looks good tomorrow, i'll go ahead and approve that and make sure nodepool still successfully connects to its db | 01:08 |
| *** Krinkle has joined #openstack-infra | 01:08 | |
| fungi | but for now i'll let it coast on the new nodepool version wit the old config | 01:09 |
| clarkb | fungi: want me to hold off on approving? | 01:09 |
| *** SumitNaiksatam has joined #openstack-infra | 01:10 | |
| fungi | clarkb: well, you can if you want. i don't expect it to take effect until the next nodepool restart anyway | 01:10 |
| *** yamamoto has joined #openstack-infra | 01:10 | |
| fungi | but i'm perfectly happy to approve tomorrow when i'm around to restart and make sure it's still working | 01:10 |
| *** otter768 has joined #openstack-infra | 01:11 | |
| fungi | thanks! | 01:11 |
| *** claudiub has quit IRC | 01:13 | |
| *** woodster_ has joined #openstack-infra | 01:14 | |
| *** otter768 has quit IRC | 01:16 | |
| *** unicell has quit IRC | 01:25 | |
| clarkb | ok time to call it a day. This os_loganalyze stuff melted my brain a bit | 01:26 |
| *** yamamoto has quit IRC | 01:26 | |
| clarkb | the good news is the code works when the tests are fixed :) | 01:26 |
| *** kiseok7_ has joined #openstack-infra | 01:33 | |
| *** tqtran_ has quit IRC | 01:37 | |
| *** sdake_ has joined #openstack-infra | 01:39 | |
| *** sdake has quit IRC | 01:43 | |
| *** annegentle has joined #openstack-infra | 01:43 | |
| *** amitgandhinz has quit IRC | 01:48 | |
| *** annegentle has quit IRC | 01:48 | |
| *** amitgandhinz has joined #openstack-infra | 01:49 | |
| *** annegentle has joined #openstack-infra | 01:50 | |
| *** harlowja is now known as harlowja_away | 01:52 | |
| *** yamamoto has joined #openstack-infra | 01:54 | |
| *** yamahata has quit IRC | 01:59 | |
| *** marun has quit IRC | 02:00 | |
| *** annegentle has quit IRC | 02:00 | |
| *** elarson has quit IRC | 02:00 | |
| *** markvoelker has quit IRC | 02:01 | |
| *** shashankhegde has joined #openstack-infra | 02:02 | |
| *** nelsnelson has joined #openstack-infra | 02:02 | |
| *** markvoelker has joined #openstack-infra | 02:03 | |
| *** bhunter71 has joined #openstack-infra | 02:04 | |
| *** shashankhegde has quit IRC | 02:10 | |
| *** baoli has joined #openstack-infra | 02:13 | |
| *** jerryz has joined #openstack-infra | 02:28 | |
| *** koolhead17 has joined #openstack-infra | 02:31 | |
| *** dboik_ has quit IRC | 02:35 | |
| *** dboik has joined #openstack-infra | 02:36 | |
| *** otter768 has joined #openstack-infra | 02:37 | |
| *** jcoufal has joined #openstack-infra | 02:39 | |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: Status: Expose $.zuul object in zuul_start to allow extension https://review.openstack.org/170337 | 02:45 |
| *** mwagner_lap has quit IRC | 02:46 | |
| *** amitgandhinz has quit IRC | 02:48 | |
| *** rbradfor has joined #openstack-infra | 02:49 | |
| *** baoli has quit IRC | 02:52 | |
| *** mwagner_lap has joined #openstack-infra | 02:59 | |
| *** amitgandhinz has joined #openstack-infra | 03:02 | |
| *** achanda has quit IRC | 03:03 | |
| *** unicell has joined #openstack-infra | 03:03 | |
| *** koolhead17 has quit IRC | 03:06 | |
| *** jcoufal has quit IRC | 03:12 | |
| *** amitgandhinz has quit IRC | 03:12 | |
| *** achanda has joined #openstack-infra | 03:16 | |
| *** sputnik13 has joined #openstack-infra | 03:32 | |
| *** Longgeek has joined #openstack-infra | 03:34 | |
| *** achanda has quit IRC | 03:36 | |
| *** tqtran has joined #openstack-infra | 03:40 | |
| *** links has joined #openstack-infra | 03:45 | |
| *** achanda has joined #openstack-infra | 03:45 | |
| *** stevemar has joined #openstack-infra | 03:46 | |
| *** chlong has joined #openstack-infra | 03:48 | |
| *** chlong has quit IRC | 03:49 | |
| *** sdake has joined #openstack-infra | 03:51 | |
| *** otter768 has quit IRC | 03:53 | |
| *** rbradfor has quit IRC | 03:54 | |
| *** carl_baldwin has joined #openstack-infra | 03:55 | |
| *** sdake_ has quit IRC | 03:55 | |
| *** Longgeek has quit IRC | 03:59 | |
| *** Longgeek has joined #openstack-infra | 03:59 | |
| *** woodster_ has quit IRC | 04:00 | |
| *** achanda has quit IRC | 04:02 | |
| *** tqtran has quit IRC | 04:05 | |
| *** subscope has quit IRC | 04:07 | |
| *** bhunter71 has quit IRC | 04:10 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Replace the Gerrit WIP icon https://review.openstack.org/165228 | 04:14 |
| *** nelsnelson has quit IRC | 04:17 | |
| *** amitgandhinz has joined #openstack-infra | 04:23 | |
| *** jlvillal has quit IRC | 04:25 | |
| *** gpocentek has quit IRC | 04:26 | |
| *** amitgandhinz has quit IRC | 04:27 | |
| *** YorikSar has joined #openstack-infra | 04:27 | |
| *** jlvillal has joined #openstack-infra | 04:28 | |
| *** prmtl has joined #openstack-infra | 04:28 | |
| *** sarob has joined #openstack-infra | 04:28 | |
| *** sarob has quit IRC | 04:33 | |
| *** tkelsey has joined #openstack-infra | 04:33 | |
| *** btully has quit IRC | 04:33 | |
| *** tkelsey has quit IRC | 04:38 | |
| *** changbl_ has quit IRC | 04:40 | |
| *** sarob has joined #openstack-infra | 04:43 | |
| *** achanda has joined #openstack-infra | 04:45 | |
| *** mrmartin has joined #openstack-infra | 04:46 | |
| *** briancurtin has quit IRC | 04:46 | |
| *** dougwig has quit IRC | 04:46 | |
| *** ameade has quit IRC | 04:46 | |
| *** zhiyan has quit IRC | 04:46 | |
| *** j^2 has quit IRC | 04:46 | |
| *** ctracey has quit IRC | 04:46 | |
| *** rackertom has quit IRC | 04:46 | |
| *** ameade has joined #openstack-infra | 04:47 | |
| *** zhiyan has joined #openstack-infra | 04:47 | |
| *** briancurtin has joined #openstack-infra | 04:47 | |
| *** j^2 has joined #openstack-infra | 04:47 | |
| *** dougwig has joined #openstack-infra | 04:48 | |
| *** ctracey has joined #openstack-infra | 04:49 | |
| *** rackertom has joined #openstack-infra | 04:50 | |
| *** hdd has joined #openstack-infra | 04:55 | |
| *** YorikSar has quit IRC | 04:58 | |
| *** carl_baldwin has quit IRC | 05:02 | |
| *** unicell1 has joined #openstack-infra | 05:05 | |
| *** unicell has quit IRC | 05:07 | |
| *** hdd has quit IRC | 05:14 | |
| *** lpetrut has joined #openstack-infra | 05:20 | |
| openstackgerrit | Kei YAMAZAKI proposed openstack-infra/jenkins-job-builder: Update gitignore https://review.openstack.org/170347 | 05:32 |
| openstackgerrit | Kei YAMAZAKI proposed openstack-infra/jenkins-job-builder: Add support slack plugin https://review.openstack.org/170348 | 05:32 |
| openstackgerrit | Kei YAMAZAKI proposed openstack-infra/jenkins-job-builder: Add support slack plugin https://review.openstack.org/170348 | 05:37 |
| openstackgerrit | Kei YAMAZAKI proposed openstack-infra/jenkins-job-builder: Update gitignore https://review.openstack.org/170349 | 05:37 |
| *** koolhead17 has joined #openstack-infra | 05:38 | |
| *** mpaolino has joined #openstack-infra | 05:39 | |
| *** ildikov_ has quit IRC | 05:40 | |
| *** koolhead17 has quit IRC | 05:42 | |
| openstackgerrit | Kei YAMAZAKI proposed openstack-infra/jenkins-job-builder: Add support slack plugin https://review.openstack.org/170348 | 05:46 |
| *** marcusvrn has quit IRC | 05:49 | |
| *** otter768 has joined #openstack-infra | 05:54 | |
| *** achanda has quit IRC | 05:57 | |
| *** YorikSar has joined #openstack-infra | 05:58 | |
| *** otter768 has quit IRC | 05:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/169624 | 06:01 |
| *** btully has joined #openstack-infra | 06:02 | |
| *** armax has quit IRC | 06:07 | |
| *** woodster_ has joined #openstack-infra | 06:13 | |
| *** sarob has quit IRC | 06:13 | |
| *** sarob has joined #openstack-infra | 06:14 | |
| *** Krinkle is now known as Krinkle|detached | 06:15 | |
| *** sarob has quit IRC | 06:18 | |
| openstackgerrit | Vadim Rovachev proposed openstack-infra/project-config: Enable ceilometer functional tests https://review.openstack.org/170359 | 06:20 |
| *** ildikov_ has joined #openstack-infra | 06:23 | |
| *** achanda has joined #openstack-infra | 06:26 | |
| *** jerryz has quit IRC | 06:28 | |
| *** markvoelker has quit IRC | 06:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/169624 | 06:29 |
| *** ajmiller__ has joined #openstack-infra | 06:31 | |
| *** ihrachyshka has joined #openstack-infra | 06:31 | |
| *** aswadr has joined #openstack-infra | 06:32 | |
| *** ajmiller_ has quit IRC | 06:34 | |
| *** _nadya_ has joined #openstack-infra | 06:36 | |
| *** doude has joined #openstack-infra | 06:39 | |
| *** _nadya_ has quit IRC | 06:41 | |
| *** ttx has quit IRC | 06:44 | |
| *** ttx has joined #openstack-infra | 06:44 | |
| *** ihrachyshka has quit IRC | 06:47 | |
| *** _nadya_ has joined #openstack-infra | 06:57 | |
| *** markvoelker has joined #openstack-infra | 06:59 | |
| *** otter768 has joined #openstack-infra | 07:00 | |
| *** mmedvede has joined #openstack-infra | 07:02 | |
| *** dtantsur|afk is now known as dtantsur | 07:02 | |
| *** markvoelker has quit IRC | 07:04 | |
| *** otter768 has quit IRC | 07:05 | |
| *** HeOS has quit IRC | 07:06 | |
| *** e0ne has joined #openstack-infra | 07:09 | |
| *** gibi has joined #openstack-infra | 07:10 | |
| *** akislitsky has left #openstack-infra | 07:12 | |
| *** _nadya_ has quit IRC | 07:13 | |
| *** stevemar has quit IRC | 07:18 | |
| *** e0ne has quit IRC | 07:20 | |
| *** amotoki has joined #openstack-infra | 07:23 | |
| *** e0ne has joined #openstack-infra | 07:24 | |
| *** jlibosva has joined #openstack-infra | 07:28 | |
| *** gaelL has quit IRC | 07:28 | |
| *** jistr has joined #openstack-infra | 07:29 | |
| *** arxcruz has joined #openstack-infra | 07:30 | |
| *** jcoufal has joined #openstack-infra | 07:32 | |
| *** achanda has quit IRC | 07:33 | |
| *** achanda has joined #openstack-infra | 07:33 | |
| *** ihrachyshka has joined #openstack-infra | 07:37 | |
| *** _nadya_ has joined #openstack-infra | 07:38 | |
| *** e0ne has quit IRC | 07:40 | |
| *** tchaypo has quit IRC | 07:47 | |
| *** ihrachyshka has quit IRC | 07:47 | |
| *** hashar has joined #openstack-infra | 07:50 | |
| *** tchaypo has joined #openstack-infra | 07:51 | |
| *** _nadya_ has quit IRC | 08:00 | |
| *** markvoelker has joined #openstack-infra | 08:00 | |
| *** dims has quit IRC | 08:03 | |
| *** markvoelker has quit IRC | 08:05 | |
| *** vlaza has joined #openstack-infra | 08:06 | |
| *** derekh has joined #openstack-infra | 08:11 | |
| *** Bsony has joined #openstack-infra | 08:12 | |
| *** mpaolino has quit IRC | 08:15 | |
| *** woodster_ has quit IRC | 08:20 | |
| *** gaelL has joined #openstack-infra | 08:21 | |
| *** marun has joined #openstack-infra | 08:23 | |
| *** lpetrut has quit IRC | 08:24 | |
| *** e0ne has joined #openstack-infra | 08:27 | |
| *** e0ne has quit IRC | 08:27 | |
| *** marun has quit IRC | 08:27 | |
| *** e0ne has joined #openstack-infra | 08:28 | |
| *** Hal has joined #openstack-infra | 08:28 | |
| *** Hal is now known as Guest78790 | 08:29 | |
| *** sarob has joined #openstack-infra | 08:30 | |
| *** sarob has quit IRC | 08:35 | |
| openstackgerrit | Merged openstack/requirements: Bump last version of ceilometer-client https://review.openstack.org/167527 | 08:35 |
| *** Longgeek has quit IRC | 08:36 | |
| *** Longgeek has joined #openstack-infra | 08:37 | |
| *** mpaolino has joined #openstack-infra | 08:39 | |
| *** salv-orlando has quit IRC | 08:40 | |
| openstackgerrit | Sean M. Collins proposed stackforge/gertty: Revert "Add support for starred changes" https://review.openstack.org/170394 | 08:48 |
| *** e0ne is now known as e0ne_ | 08:48 | |
| *** e0ne_ is now known as e0ne | 08:49 | |
| *** lpetrut has joined #openstack-infra | 08:49 | |
| *** andreykurilin_ has joined #openstack-infra | 08:50 | |
| *** ihrachyshka has joined #openstack-infra | 08:52 | |
| *** yamamoto has quit IRC | 08:52 | |
| *** achanda has quit IRC | 08:55 | |
| *** achanda has joined #openstack-infra | 08:55 | |
| *** oomichi_ has quit IRC | 08:56 | |
| *** SumitNaiksatam has left #openstack-infra | 08:57 | |
| *** salv-orlando has joined #openstack-infra | 08:57 | |
| *** oomichi has joined #openstack-infra | 08:58 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 | 08:58 |
| *** oomichi has quit IRC | 09:01 | |
| *** markvoelker has joined #openstack-infra | 09:01 | |
| *** otter768 has joined #openstack-infra | 09:01 | |
| *** HeOS has joined #openstack-infra | 09:02 | |
| *** otter768 has quit IRC | 09:05 | |
| *** markvoelker has quit IRC | 09:05 | |
| *** imcsk8 has quit IRC | 09:06 | |
| *** imcsk8 has joined #openstack-infra | 09:06 | |
| *** skolekonov has joined #openstack-infra | 09:07 | |
| *** pelix has joined #openstack-infra | 09:21 | |
| *** wicope has joined #openstack-infra | 09:23 | |
| *** dimtruck is now known as zz_dimtruck | 09:24 | |
| *** wicope has joined #openstack-infra | 09:24 | |
| *** achanda has quit IRC | 09:25 | |
| *** achanda has joined #openstack-infra | 09:28 | |
| *** sdake has quit IRC | 09:28 | |
| *** achanda has quit IRC | 09:28 | |
| *** achanda has joined #openstack-infra | 09:30 | |
| *** bnemec has quit IRC | 09:33 | |
| *** tkelsey has joined #openstack-infra | 09:33 | |
| *** tkelsey has quit IRC | 09:38 | |
| *** Krinkle|detached is now known as Krinkle | 09:49 | |
| *** andreykurilin_ has quit IRC | 09:50 | |
| *** _nadya_ has joined #openstack-infra | 09:58 | |
| *** yamamoto has joined #openstack-infra | 10:00 | |
| *** dtantsur is now known as dtantsur|lunch | 10:00 | |
| *** markvoelker has joined #openstack-infra | 10:02 | |
| *** achanda has quit IRC | 10:02 | |
| *** markvoelker has quit IRC | 10:06 | |
| *** prmtl has quit IRC | 10:08 | |
| *** Guest78790 has quit IRC | 10:10 | |
| *** dims has joined #openstack-infra | 10:12 | |
| *** aswadr has quit IRC | 10:13 | |
| *** dims has quit IRC | 10:13 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** bnemec has joined #openstack-infra | 10:19 | |
| *** dims has joined #openstack-infra | 10:19 | |
| *** Krinkle is now known as Krinkle|detached | 10:21 | |
| *** claudiub has joined #openstack-infra | 10:36 | |
| *** pc_m has joined #openstack-infra | 10:38 | |
| *** sputnik13 has quit IRC | 10:52 | |
| *** mpaolino has quit IRC | 10:53 | |
| *** jcoufal_ has joined #openstack-infra | 10:55 | |
| *** jcoufal has quit IRC | 10:58 | |
| *** otter768 has joined #openstack-infra | 11:02 | |
| *** markvoelker has joined #openstack-infra | 11:02 | |
| *** dtantsur|lunch is now known as dtantsur | 11:03 | |
| *** otter768 has quit IRC | 11:07 | |
| *** markvoelker has quit IRC | 11:07 | |
| fungi | awesome, nodepool seems to have not fallen over (yet anyway) | 11:07 |
| *** pblaho has joined #openstack-infra | 11:12 | |
| *** prmtl has joined #openstack-infra | 11:14 | |
| sdague | fungi: any chance I could get you to approve this? - https://review.openstack.org/#/c/166396 | 11:21 |
| *** e0ne is now known as e0ne_ | 11:25 | |
| *** ujuc has joined #openstack-infra | 11:25 | |
| *** sarob has joined #openstack-infra | 11:31 | |
| *** sarob has quit IRC | 11:35 | |
| *** e0ne_ has quit IRC | 11:35 | |
| *** yamamoto has joined #openstack-infra | 11:36 | |
| *** prmtl has quit IRC | 11:38 | |
| *** alexpilotti has joined #openstack-infra | 11:39 | |
| *** tiswanso has quit IRC | 11:44 | |
| *** markvoelker has joined #openstack-infra | 12:03 | |
| *** nibalizer has quit IRC | 12:05 | |
| *** hichihara has quit IRC | 12:07 | |
| *** markvoelker has quit IRC | 12:08 | |
| *** bhunter71 has joined #openstack-infra | 12:11 | |
| *** nibalizer has joined #openstack-infra | 12:12 | |
| *** bswartz has quit IRC | 12:18 | |
| *** adam_g is now known as adam_g_out | 12:20 | |
| *** e0ne has joined #openstack-infra | 12:21 | |
| *** markvoelker has joined #openstack-infra | 12:21 | |
| *** sdake has joined #openstack-infra | 12:24 | |
| *** pblaho_ has joined #openstack-infra | 12:27 | |
| *** jlibosva1 has joined #openstack-infra | 12:27 | |
| *** jistr_ has joined #openstack-infra | 12:27 | |
| *** dtantsur_ has joined #openstack-infra | 12:27 | |
| *** jcoufal has joined #openstack-infra | 12:27 | |
| *** dtantsur has quit IRC | 12:28 | |
| *** jcoufal_ has quit IRC | 12:28 | |
| *** jcoufal_ has joined #openstack-infra | 12:28 | |
| *** eglamn3 has joined #openstack-infra | 12:29 | |
| *** ihrachyshka has quit IRC | 12:29 | |
| *** jlibosva has quit IRC | 12:30 | |
| *** smccully has joined #openstack-infra | 12:30 | |
| *** jistr has quit IRC | 12:30 | |
| *** pblaho has quit IRC | 12:31 | |
| *** pblaho__ has joined #openstack-infra | 12:31 | |
| *** jistr has joined #openstack-infra | 12:31 | |
| *** jlibosva1 has quit IRC | 12:31 | |
| eglamn3 | hi, whet is the next step for this review https://review.openstack.org/#/c/163876/ ? everything seems to be ok ? | 12:32 |
| *** dtantsur_ has quit IRC | 12:32 | |
| *** jcoufal has quit IRC | 12:32 | |
| *** pblaho_ has quit IRC | 12:32 | |
| *** jistr_ has quit IRC | 12:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add API method create_subnet() https://review.openstack.org/170217 | 12:33 |
| *** signed8bit has joined #openstack-infra | 12:35 | |
| *** lpetrut has quit IRC | 12:37 | |
| *** e0ne is now known as e0ne_ | 12:37 | |
| *** dtantsur has joined #openstack-infra | 12:37 | |
| *** e0ne_ is now known as e0ne | 12:37 | |
| *** dkliban_afk is now known as dkliban | 12:38 | |
| *** mpaolino has joined #openstack-infra | 12:39 | |
| *** miqui has joined #openstack-infra | 12:42 | |
| *** lpetrut has joined #openstack-infra | 12:45 | |
| *** signed8bit has quit IRC | 12:45 | |
| *** prmtl has joined #openstack-infra | 12:49 | |
| *** baoli has joined #openstack-infra | 12:49 | |
| *** Hal has joined #openstack-infra | 12:49 | |
| *** signed8bit has joined #openstack-infra | 12:49 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add API method delete_subnet() https://review.openstack.org/170477 | 12:49 |
| *** Hal is now known as Guest27546 | 12:49 | |
| *** baoli has quit IRC | 12:53 | |
| *** baoli has joined #openstack-infra | 12:53 | |
| *** _nadya_ has quit IRC | 12:54 | |
| *** weshay has joined #openstack-infra | 12:54 | |
| *** btully has quit IRC | 12:55 | |
| *** bswartz has joined #openstack-infra | 12:55 | |
| *** yamamoto has quit IRC | 12:55 | |
| *** pblaho__ is now known as pblaho | 12:56 | |
| *** pblaho has joined #openstack-infra | 12:56 | |
| *** garyh has joined #openstack-infra | 12:56 | |
| *** prmtl has quit IRC | 12:57 | |
| *** garyh has quit IRC | 12:57 | |
| *** bknudson has quit IRC | 13:00 | |
| *** dims has quit IRC | 13:01 | |
| *** dims has joined #openstack-infra | 13:02 | |
| *** derekh has quit IRC | 13:06 | |
| *** alaski is now known as lascii | 13:07 | |
| *** wenlock has joined #openstack-infra | 13:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Add tempest + postgres job for manila https://review.openstack.org/165935 | 13:13 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Allow passing puppetmaster_server param for jenkins manifests https://review.openstack.org/170487 | 13:15 |
| openstackgerrit | Merged openstack-infra/project-config: Add os-testr commits to gerritbot config https://review.openstack.org/169933 | 13:16 |
| openstackgerrit | Merged openstack-infra/project-config: Remove unneeded 'create' from all tags sections https://review.openstack.org/169853 | 13:17 |
| *** elarson has joined #openstack-infra | 13:17 | |
| *** annegentle has joined #openstack-infra | 13:18 | |
| openstackgerrit | Merged openstack-infra/project-config: saharaclient: enable Python3 jobs https://review.openstack.org/168899 | 13:18 |
| *** sdake_ has joined #openstack-infra | 13:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Improve ec2-api gating jobs for rally testing https://review.openstack.org/166648 | 13:19 |
| *** dkranz has quit IRC | 13:19 | |
| *** dustins has joined #openstack-infra | 13:19 | |
| *** dkranz has joined #openstack-infra | 13:22 | |
| *** sdake has quit IRC | 13:22 | |
| openstackgerrit | Serg Melikyan proposed openstack-infra/project-config: Rename stackforge/murano to openstack/murano https://review.openstack.org/167934 | 13:23 |
| *** ayoung has quit IRC | 13:24 | |
| mescanef | hello, please add me (mmatuszkowiak@mirantis.com) to the gerrit groups in a stackforge/fuel-plugin-cisco-aci (reference: https://review.openstack.org/#/c/166872/) | 13:25 |
| *** miqui_ has joined #openstack-infra | 13:25 | |
| *** mriedem_away is now known as mriedem | 13:26 | |
| *** BobH has joined #openstack-infra | 13:27 | |
| *** amitgandhinz has joined #openstack-infra | 13:28 | |
| *** sdake has joined #openstack-infra | 13:29 | |
| *** miqui has quit IRC | 13:29 | |
| *** bknudson has joined #openstack-infra | 13:30 | |
| *** dboik has quit IRC | 13:32 | |
| *** sdake_ has quit IRC | 13:33 | |
| *** amotoki has quit IRC | 13:33 | |
| *** tiswanso has joined #openstack-infra | 13:34 | |
| *** openstackgerrit has quit IRC | 13:36 | |
| *** openstackgerrit has joined #openstack-infra | 13:36 | |
| fungi | mescanef: i will once that change merges and gets applied to our systems. probably within the next hour | 13:36 |
| mescanef | fungi: ok, thanks :) | 13:38 |
| fungi | wow, 10 out of the first 12 project-config changes in the gate were timing out on centos6 git remote calls | 13:43 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:43 | |
| sdague | fungi: interesting | 13:45 |
| fungi | with a failure rate that high, it should be easy for me to recreate this. i'll see if i have more luck this time around than last time i tried to hunt it down | 13:45 |
| *** sabeen1 has joined #openstack-infra | 13:46 | |
| sdague | I guess I'm glad I flushed so many changes this morning :) | 13:46 |
| *** TheJulia_ is now known as TheJulia | 13:46 | |
| mordred | fungi: do we need to increase the nodes in the git farm? | 13:49 |
| fungi | mordred: i doubt it. otherwise why would it be just our centos6 puppet apply jobs doing this and none of our precise/trusty ones? | 13:50 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Add flag to include hacking https://review.openstack.org/170492 | 13:50 |
| mordred | fungi: nod | 13:50 |
| fungi | this is the problem that cropped up right after the glibc updates for the "ghost" vulnerability | 13:51 |
| fungi | still have a sinking suspicion something changed in git behavior on rhel/centos 6 as a result of that patch | 13:52 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Make saharaclient-dsvm-functional voting https://review.openstack.org/169426 | 13:53 |
| mordred | fungi: *grumble* | 13:53 |
| *** mpaolino has quit IRC | 13:54 | |
| fungi | one other possibility for this only showing up on centos6. since our puppet apply jobs filter the manifests by the platforms on which we run them, this could be an issue with a crappy remote used by some puppet module we only end up using on centos6 | 13:54 |
| mordred | fungi: oh - good point | 13:55 |
| *** AJaeger has joined #openstack-infra | 13:56 | |
| fungi | anyway, i've held one of the workers which is currently running that job, and as soon as (or if) it finishes, i'll rerun the script for it until i can reproduce a hang and then try to inspect it | 13:57 |
| *** tosky has joined #openstack-infra | 13:57 | |
| *** sarob has joined #openstack-infra | 13:57 | |
| tosky | AJaeger: thanks for the review on https://review.openstack.org/169426, even if I will have to wait for the required patches | 13:58 |
| AJaeger | tosky: you're welcome | 13:58 |
| tosky | AJaeger: I have two questions about https://review.openstack.org/#/c/168899/, which lead to my rebase: | 13:58 |
| tosky | - the newly introduced job is "gate-" but in the "check:" group, should it be renamed? | 13:58 |
| tosky | - should the job with the same name in the "experimental" queue (few lines below) removed? | 13:58 |
| *** dboik has joined #openstack-infra | 13:58 | |
| *** ildikov_ has quit IRC | 14:00 | |
| *** btully has joined #openstack-infra | 14:00 | |
| AJaeger | tosky: yeah, remove it from experimental - an oversight to keep it there | 14:00 |
| AJaeger | naming check or gate is not consistent, no problem here | 14:01 |
| tosky | ah, I see | 14:01 |
| AJaeger | If you have a preference, change it - I consider it a nit ;) | 14:01 |
| tosky | I'm starting to know the zuul part, so I was not sure about the naming policy there :) | 14:01 |
| *** sarob has quit IRC | 14:02 | |
| *** btully has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Add license info to subunit2html https://review.openstack.org/169940 | 14:05 |
| openstackgerrit | Merged openstack-infra/project-config: Make the unrecognized exception more verbose https://review.openstack.org/169511 | 14:06 |
| openstackgerrit | Merged openstack-infra/project-config: Added fuel-plugin-cisco-aci fuel plugin to SF https://review.openstack.org/166872 | 14:06 |
| openstackgerrit | Merged openstack-infra/project-config: Add Networking-portforwarding project https://review.openstack.org/169332 | 14:06 |
| openstackgerrit | Merged openstack-infra/project-config: Switch nova-docker-f20 job to centos7 https://review.openstack.org/167437 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Remove the g-r requirements check https://review.openstack.org/169434 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Install required services for Zaqar https://review.openstack.org/169269 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Gnocchi: do not check requirements https://review.openstack.org/169210 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Add check-tempest-dsvm-heat https://review.openstack.org/167802 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Add heat tempest job to heat and tempest https://review.openstack.org/167803 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: enable python-jobs for nova-zvm-virt-driver https://review.openstack.org/167123 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Restore devstack centos7 to non-voting check job https://review.openstack.org/168780 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Also register openstackid jobs with bare-trusty https://review.openstack.org/168739 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: add cue-dashboard project to openstack-cue channel https://review.openstack.org/168455 | 14:07 |
| openstackgerrit | Merged openstack-infra/project-config: Experimental bandit job for keystone https://review.openstack.org/157595 | 14:09 |
| openstackgerrit | Merged openstack-infra/project-config: Exclude the slow tag for the Nova Cells V1 job https://review.openstack.org/168121 | 14:09 |
| openstackgerrit | Merged openstack-infra/project-config: Remove tempest-dsvm-ceilolmeter-mongodb job https://review.openstack.org/167695 | 14:09 |
| mordred | wow | 14:10 |
| mordred | AJaeger: I'm going to assume you did some rocking and rolling this morning | 14:10 |
| AJaeger | mordred: me? Why? | 14:11 |
| fungi | mordred: sdague approved a bunch of project-config changes which already had one other +2 | 14:11 |
| AJaeger | mordred: the above was sdague approving patches that have been waiting for a second core for some time... | 14:11 |
| fungi | i think i've recreated it on the held worker. it's been several minutes now that this has been in the process table: "git fetch-pack | 14:11 |
| mordred | oh! it was sdague | 14:11 |
| fungi | --stateless-rpc --lock-pack --include-tag --thin --no-progress https://git.opens | 14:11 |
| fungi | tack.org/openstack-infra/project-config efs/heads/master | 14:11 |
| mordred | I just assumed because it was early US time it would be the european :) | 14:11 |
| AJaeger | mordred: for many of them I reviewed them some time ago, so the other +2 might have been mine ;) | 14:12 |
| AJaeger | mordred: it was the early sdague :) | 14:12 |
| fungi | i'm not sure if the " efs/heads/master" is just an artifact of the process table or if that's really what it's running | 14:12 |
| mordred | the early sdague is an amazing sight | 14:12 |
| *** teran has joined #openstack-infra | 14:12 | |
| fungi | but yeah it's certainly looking hung on a fetch-pack for the project-config repo from our mirror | 14:13 |
| *** zul has quit IRC | 14:13 | |
| *** amitgandhinz has quit IRC | 14:13 | |
| *** zul has joined #openstack-infra | 14:14 | |
| sdague | mordred: :) | 14:14 |
| fungi | looks like this is a child of a `git fetch origin` run from vcsrepo | 14:14 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Fix exception in update_router() https://review.openstack.org/170510 | 14:14 |
| fungi | happening in the single-use-slave apply | 14:16 |
| kerpan | Guys, I think there's a serious problem with gerrit gui. It doesn't load Clippy anymore. I'm lost without it. | 14:17 |
| fungi | looks like that git process is blocking indefinitely on a read call according to strace, most likely a dead socket | 14:17 |
| *** AJaeger has quit IRC | 14:18 | |
| fungi | kerpan: i see you're trying to clippy... | 14:18 |
| openstackgerrit | Merged openstack-infra/project-config: Murano client is switching to use separated launchpad repo https://review.openstack.org/167156 | 14:18 |
| openstackgerrit | Merged openstack-infra/project-config: Stop running tempest-dsvm-neutron in juno https://review.openstack.org/167674 | 14:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup jenkins/jobs/projects.yaml https://review.openstack.org/168148 | 14:20 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add API method create_subnet() https://review.openstack.org/170217 | 14:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add API method delete_subnet() https://review.openstack.org/170477 | 14:22 |
| mrmartin | er | 14:23 |
| openstackgerrit | Merged openstack-infra/project-config: Adjust regression exceptions for Nova Cells V1 job https://review.openstack.org/166396 | 14:24 |
| mrmartin | fungi: this patch sits in the a queue for a while, and just wait for an approval: https://review.openstack.org/167631 | 14:24 |
| openstackgerrit | Merged openstack-infra/system-config: Switch nodepool from MySQL-python to PyMySQL https://review.openstack.org/170314 | 14:25 |
| *** annegentle has quit IRC | 14:25 | |
| *** armax has joined #openstack-infra | 14:25 | |
| fungi | mrmartin: it's waiting for https://review.openstack.org/168117 to get approved | 14:25 |
| mrmartin | ah really | 14:25 |
| openstackgerrit | Merged openstack-infra/project-config: Add cerberus related projects https://review.openstack.org/163876 | 14:26 |
| mrmartin | ttx are you here? | 14:26 |
| fungi | mrmartin: yep, we don't add new repos to official openstack git namespaces until the technical committee approves them, which usually happens during their weekly irc meetings on tuesdays at 20:00 utc | 14:27 |
| mrmartin | oh my gosh, so we are loosing another week with the backups | 14:27 |
| fungi | mrmartin: i can (and will) make a manual backup anyway. i don't think it's urgent to be able to switch to that server | 14:28 |
| mrmartin | fungi: as I know the switching is scheduled to monday | 14:28 |
| fungi | we can still run there for a week while we wait for official process to happen | 14:28 |
| fungi | automated database dumps don't need to be in place already when we switch | 14:29 |
| fungi | it's non-user-facing | 14:29 |
| ttx | mrmartin: yes | 14:29 |
| *** woodster_ has joined #openstack-infra | 14:29 | |
| openstackgerrit | Jan Provaznik proposed openstack-infra/project-config: Create tripleo-common project under TripleO https://review.openstack.org/165433 | 14:30 |
| mrmartin | ttx: https://review.openstack.org/#/c/168117/ this review is blocking of the db-backup of new ask.o.o instance | 14:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add API method update_subnet() https://review.openstack.org/170516 | 14:30 |
| fungi | i think it got skipped in the governance housekeeping round-up on tuesday | 14:30 |
| fungi | or perhaps just doesn't have enough tc +1 votes yet | 14:30 |
| ttx | fungi: we only vote on stuff that's been on the docket for a few days, that was proposed just the day before | 14:31 |
| mrmartin | it have 4 votes | 14:31 |
| ttx | which is why it was skipped | 14:31 |
| *** rbradfor has joined #openstack-infra | 14:31 | |
| mrmartin | ok, so it means, that on next tuesday we have a good chance to get an approval? | 14:31 |
| fungi | hrm, proposed thursday the 26th, tc meeting was tuesday the 31 | 14:31 |
| *** ildikov_ has joined #openstack-infra | 14:32 | |
| ttx | sure. I'm also fine for fasttracking it if you can get 7 votes | 14:32 |
| fungi | so 5 days (3 weekdays) before the meeting | 14:32 |
| ttx | fungi: yeah, in theory we can only approve stuff that's been there for 4 business days | 14:32 |
| *** links has quit IRC | 14:33 | |
| ttx | so anything proposed after Thu 20:00 gets discounted by the script that gathers the items | 14:33 |
| fungi | ttx: got it. so in the future governance changes need to be up by wednesday to get onto the list for the coming tuesday meeting | 14:33 |
| ttx | but I agree that housekeeping items like this could get a break | 14:33 |
| fungi | (that was thurdsay at 19:24, so technically 36 minutes prior to the cut-off by that measure) | 14:34 |
| mrmartin | :) | 14:34 |
| ttx | I blame dst | 14:34 |
| fungi | i _always_ blame dst | 14:34 |
| mrmartin | yeah, down with dst! I also missed some meeting too :D | 14:34 |
| mrmartin | ttx: are you looking for storyboard alternatives? | 14:35 |
| ttx | mrmartin: as I said, if you can get a couple more TC members to +1 it I 'm fine fasttracking it | 14:35 |
| ttx | I commented as such on that review | 14:35 |
| ttx | Although I'll be leaving in a few hours so better hurry :P | 14:36 |
| fungi | anyway, i don't think we need to delay monday's ask.o.o cut-over maintenance. on the assumption that gets approved in tuesday's meeting we'll have the automated database dumps in place within a couple days after the maintenance | 14:36 |
| ttx | mrmartin: no, but I'm evaluating the alternatives that are proposed to me | 14:36 |
| mrmartin | ttx: cool, do you know about a design summit session where we can discuss the options? Or what is the platform of the election of this toolset? | 14:37 |
| jeblair | mrmartin: i expect there will be an infra session | 14:37 |
| ttx | there should be an infra session on that for sure | 14:37 |
| mrmartin | awesome, I don't like to miss it | 14:37 |
| ttx | jeblair: did you set up the infra planning etherpad yet ? | 14:37 |
| jeblair | ttx: not yet | 14:38 |
| * ttx has been trying to revive odsreg to support cross-project session planning, but I fell in django versioning hell | 14:38 | |
| mordred | mrmartin: I've been investigating phabricator, fwiw | 14:38 |
| mordred | mrmartin: been meaning to send a message to the list about it | 14:38 |
| mrmartin | mordred, cool, php ? | 14:38 |
| mordred | yah | 14:39 |
| mrmartin | :D | 14:39 |
| ttx | deploying multiple django apps on the same host that rely on widely different django versions is, I discovered, non-trivial | 14:39 |
| *** openstackgerrit has quit IRC | 14:39 | |
| mordred | mrmartin: which means there may be some tasks where we could use your help :) | 14:39 |
| * mrmartin is crying | 14:39 | |
| ttx | or at least exceeeds the time I allocated to this interesting problem | 14:39 |
| *** openstackgerrit has joined #openstack-infra | 14:39 | |
| mrmartin | ttx: containers | 14:39 |
| ttx | mrmartin: oh, I know there are plenty of solutions to my problem. Just exceeded the 10 minutes I had dedicated to that effort today | 14:40 |
| mrmartin | I try to allocate some time to investigate some paas / container solution to make the php app deployment a bit simpler than we have today. | 14:40 |
| mrmartin | everything seems to be nice until not need to upgrade the apps. | 14:41 |
| *** yfried|afk has quit IRC | 14:41 | |
| fungi | my biggest concern with containers in production is making sure the various system packages within them are kept updated somehow | 14:41 |
| *** jaypipes has joined #openstack-infra | 14:42 | |
| fungi | at which point, it's not clear to me that one larger virtual machine with lots of containers on it is in any way more effective than a bunch of smaller virtual machines | 14:42 |
| mordred | fungi: yes - which means that if, for instance, we were to use docker - we'd want to have a workflow that allows us to make docker containers in response to source code we're managing | 14:42 |
| mrmartin | fungi: we had the same problems with FreeBSD jails more then 10yrs ago. | 14:43 |
| *** ZZelle has quit IRC | 14:43 | |
| *** ZZelle has joined #openstack-infra | 14:43 | |
| fungi | mordred: what's the workflow for that? just have it redeploy a freshly rebuilt container daily to incorporate the latest system packages since the day before? | 14:43 |
| *** yamamoto has joined #openstack-infra | 14:44 | |
| mordred | fungi: not sure - it's a problem we'd have to solve | 14:44 |
| *** sarob has joined #openstack-infra | 14:44 | |
| fungi | mrmartin: yep, same here. i ran userspace tool/library upgrades independently within each jail for each errata | 14:44 |
| mrmartin | that's an interesting one. | 14:44 |
| fungi | and then in the main system context for kernel errata (which would need rebooting anyway) | 14:45 |
| *** dmsimard has quit IRC | 14:45 | |
| *** dmsimard has joined #openstack-infra | 14:46 | |
| fungi | mordred: yep. mostly just curious whether the container community simply hasn't realized yet that security updates are a thing, or if there's some well-established solution already in use for that | 14:46 |
| mrmartin | yeap, some ksplice alternative could help to avoid the reboots | 14:47 |
| *** banix_ has joined #openstack-infra | 14:47 | |
| *** annegentle has joined #openstack-infra | 14:48 | |
| *** yamamoto has quit IRC | 14:48 | |
| *** armax has quit IRC | 14:49 | |
| *** banix_ has quit IRC | 14:50 | |
| *** sarob has quit IRC | 14:50 | |
| *** amitgandhinz has joined #openstack-infra | 14:50 | |
| *** _nadya_ has joined #openstack-infra | 14:51 | |
| *** claudiub has quit IRC | 14:52 | |
| *** emagana has joined #openstack-infra | 14:55 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/zuul: Update zuul-changes.py doc to match usage https://review.openstack.org/169527 | 14:58 |
| *** koolhead17 has joined #openstack-infra | 14:59 | |
| mordred | mrmartin: ok. just sent email to openstack-infra mailing list with thoughts/details | 15:00 |
| *** Longgeek has quit IRC | 15:01 | |
| *** jaypipes is now known as leakypipes | 15:02 | |
| *** emagana has quit IRC | 15:03 | |
| *** emagana has joined #openstack-infra | 15:03 | |
| *** sdake_ has joined #openstack-infra | 15:03 | |
| *** rhe00 has joined #openstack-infra | 15:03 | |
| *** otter768 has joined #openstack-infra | 15:04 | |
| *** unicell has joined #openstack-infra | 15:04 | |
| *** carl_baldwin has joined #openstack-infra | 15:04 | |
| mrmartin | kanban like workboard cool | 15:04 |
| *** nelsnelson has joined #openstack-infra | 15:05 | |
| *** unicell1 has quit IRC | 15:06 | |
| kerpan | Guys, could you please add me to the gerrit groups in stackforge/networking-portforwarding ? (https://review.openstack.org/#/c/169332/) | 15:06 |
| *** ajmiller__ has quit IRC | 15:06 | |
| *** claudiub has joined #openstack-infra | 15:06 | |
| *** ajmiller has joined #openstack-infra | 15:07 | |
| *** sdake has quit IRC | 15:07 | |
| *** vlaza has quit IRC | 15:07 | |
| mordred | mrmartin: the one definite piece of php we'll need _Someone_ to do that will be unfortunate because it will be throwaway work | 15:07 |
| *** thedodd has joined #openstack-infra | 15:07 | |
| *** EmilienM is now known as EmilienM|afk | 15:07 | |
| mordred | mrmartin: is an openid auth provider that auth's to launchpad | 15:08 |
| jeblair | mordred: that doesn't need to be throwaway | 15:08 |
| jeblair | mordred: openid is an open system that can be pointed at anything; there's nothing launchpad specific about it | 15:08 |
| *** otter768 has quit IRC | 15:08 | |
| *** koolhead17 has quit IRC | 15:08 | |
| fungi | kerpan: done | 15:08 |
| mordred | jeblair: I say throwaway because upstream is not interested in adding openid - they'd rather add openid connect | 15:09 |
| mordred | jeblair: which we can move to with openstackid | 15:09 |
| kerpan | fungi: thanks!! | 15:09 |
| jeblair | mordred: they actively _don't_ want an openid client? | 15:09 |
| mordred | yes | 15:09 |
| mrmartin | mordred: it usually depends on the underlying app platform. | 15:09 |
| fungi | mescanef: i've added you as the initial member in the groups for the fuel plugin you mentioned earlier | 15:09 |
| clarkb | mordred I was afraid youbwere going tonsay they only want facebook auth | 15:09 |
| jeblair | mordred: can you point me at a conversation? | 15:09 |
| mrmartin | so forexample, Drupal had a very nice integrated openid provider module, but oauth2 was a mess, and a little care required here | 15:10 |
| mordred | jeblair: I can find it, yes | 15:10 |
| *** koolhead17 has joined #openstack-infra | 15:11 | |
| mrmartin | the big deal here if we like to move away from launchpad, how we like to do the mapping between launchpad openid and the new implementation. | 15:11 |
| fungi | mordred: i think openstackid doesn't (yet) support openid connect, though i may be wrong about that | 15:11 |
| jeblair | mordred: i'd love to see the reasoning. because if they aren't interested it something like openid, i worry they'll be even less interested in other more substantial changes we might want to make. | 15:12 |
| *** ajmiller has quit IRC | 15:12 | |
| mordred | jeblair: https://secure.phabricator.com/T524 | 15:12 |
| jeblair | (a vibrant upstream developer community is only useful if they actually accept your patches) | 15:12 |
| mordred | jeblair: their main concern is that openid interst is small and waning compared to the complexity of supporting it | 15:12 |
| *** koolhead17 has quit IRC | 15:13 | |
| jeblair | and here i thought it was simple with plenty of well established php code implementing it | 15:13 |
| mordred | considering that once we're off of launchpad we'd also not particularly care about it per-se in favor of openid-connect (which the openstackid folks wanted to do anyway) | 15:13 |
| mordred | althuogh: | 15:14 |
| mordred | " | 15:14 |
| mordred | I'm not necessarily totally opposed to doing this at some point, but interest seems very low and cost relatively high. This can be built as an extension today, so maybe that's the best route forward? Here's my half-implementation:" | 15:14 |
| cinerama | mordred, jeblair: oh how funny, i had just gotten to the "openid??" stage of phabricator install/config | 15:14 |
| *** rhe00 has left #openstack-infra | 15:14 | |
| mordred | jeblair: it seems that it may be more that nobody has actually stepped up to the plate | 15:14 |
| *** hodos has joined #openstack-infra | 15:14 | |
| mordred | and he's not goign to do it himself | 15:14 |
| jeblair | mordred: sure, that's different | 15:14 |
| cinerama | mordred, jeblair that and the php daemon they like you to run :) | 15:14 |
| *** carl_baldwin has quit IRC | 15:14 | |
| jeblair | mordred: i don't think openstackid support openid connect, as fungi just said | 15:15 |
| jeblair | mordred: because the rest of our tools don't | 15:15 |
| mordred | that's correct | 15:15 |
| mrmartin | it is not supporting that as I know | 15:15 |
| *** claudiub has quit IRC | 15:15 | |
| mordred | mrmartin: so - ignore my words "throwaway" | 15:16 |
| BobH | is the preferred approach to set up a 3rd part CI now to use the new puppet module structure and build each piece independently? | 15:16 |
| *** annegentle has quit IRC | 15:17 | |
| jeblair | mordred: does openid-connect support exist for phab? | 15:17 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Add flag to include hacking https://review.openstack.org/170492 | 15:17 |
| *** annegentle has joined #openstack-infra | 15:17 | |
| *** ajmiller has joined #openstack-infra | 15:17 | |
| *** nelsnelson has quit IRC | 15:17 | |
| *** nelsnelson has joined #openstack-infra | 15:18 | |
| BobH | Rather than follow the current instructions out in 'Running your own CI infrastructure? | 15:19 |
| mrmartin | anyway it have a nice amount of supported auth providers: https://github.com/phacility/phabricator/tree/master/src/applications/auth/provider | 15:19 |
| *** zz_dimtruck is now known as dimtruck | 15:19 | |
| jeblair | mordred, mrmartin: openstackid does support oauth2, right? | 15:20 |
| mrmartin | yeap | 15:20 |
| *** salv-orlando has quit IRC | 15:20 | |
| *** armax has joined #openstack-infra | 15:20 | |
| fungi | mrmartin: though i recall there was some mention that it only currently supports using oauth2 for id lookup and not for auth? what were the details there? | 15:21 |
| mordred | jeblair: unsure on openid-connect. I would imagine that cargo-culting one of the oauth2 providers and making it grok openstackid as an endpoint shouldn't be terribly hard | 15:21 |
| *** Kiall has quit IRC | 15:22 | |
| mrmartin | So in a hard-coded form it supports auth: https://github.com/openstack-infra/openstackid/commit/7970a10544c48f6227b8d491e16978c38a65f31e | 15:22 |
| *** Kiall has joined #openstack-infra | 15:22 | |
| fungi | okay, which sounds like was insufficient for using it as an oauth2 backend to keystone like the trystack admins were hoping for | 15:23 |
| *** jcoufal_ has quit IRC | 15:23 | |
| clarkb | BobH we are in a bit of a transition state right now. Either way is likely to work at this point but may require some bug fixing to get the split out repos happy | 15:23 |
| clarkb | BobH others have been tackling this work and have tagged their changes in gerrit with a topic of downstream-puppet | 15:24 |
| jeblair | so i think we have these choices: a) add openid support to phab; b) switch from launchpad auth to phab at the same time and use oauth2 | 15:24 |
| BobH | thanks #clarkb, I've been having issues with it, so I'm wondering if I should change my approach. | 15:24 |
| jeblair | i'm not a fan of (b) as it complicates the transition quite a bit | 15:25 |
| jeblair | but it's certainly possible if we have all hands on deck | 15:25 |
| clarkb | BobH the puppet-openstackci is still very new (I dont know that it has any code yet) but thats where the effort should be going | 15:25 |
| mordred | jeblair: yah - I think adding openid support to phab is a better choice if we can do it - largely because it would allow us to do better testing/analysis | 15:26 |
| jeblair | and if we go with (a) hopefully phab upstream will accept the contribution of openid, and if they don't we can call it throwaway and switch to oauth2 | 15:26 |
| mordred | ++ | 15:26 |
| *** links has joined #openstack-infra | 15:27 | |
| BobH | so, I should continue on my current path, after I look at puppet-openstackci, and just work through the issues | 15:28 |
| jeblair | mordred, fungi, clarkb: i have to run errands this morning, but i'll be back this afternoon and plan to restart zuul then | 15:28 |
| mrmartin | the openid was very stable in openstackid, the only reason that I changed to oauth2 that I cannot retrieve profile pictures with the openid implementation. | 15:28 |
| *** teran has quit IRC | 15:28 | |
| fungi | jeblair: sounds good. i'm about to restart nodepoold now that the updated nodepool.yaml is in place to switch it to going through pymysql | 15:29 |
| jeblair | neat! | 15:29 |
| clarkb | BobH anything you can do to help the transition to making puppet-openstackci useable is much appreciated (testing bugfixes manifests/modules docs etc) | 15:29 |
| BobH | <clarkb> will do and thanks for your help | 15:30 |
| fungi | also i'm having no luck reproducing the git hangs on centos6 again now | 15:31 |
| fungi | i wonder if something about the git client version in centos 6 makes very inefficient pack calls to the server and when we have a dozen or more going at once that's when we see it lock up | 15:32 |
| *** YorikSar has quit IRC | 15:32 | |
| *** marun has joined #openstack-infra | 15:33 | |
| fungi | if we used git protocol instead of smart git over https in our manifests, this might just go away | 15:33 |
| *** amotoki has joined #openstack-infra | 15:35 | |
| clarkb | its possible, also are we using the git cache for all of them? | 15:35 |
| fungi | on the ones where i was able to catch it already hung, they seemed to consistently be stuck updating /etc/project-config | 15:35 |
| fungi | i don't think puppet uses the git cache for vcsrepo | 15:35 |
| *** tkelsey has joined #openstack-infra | 15:35 | |
| *** HeOS has quit IRC | 15:36 | |
| *** tonytan4ever has joined #openstack-infra | 15:36 | |
| fungi | which is where it seems to be happening | 15:36 |
| *** sputnik13 has joined #openstack-infra | 15:36 | |
| clarkb | we could have job setup put repos ahere vcsrepo wants them via git cache | 15:36 |
| *** spzala has quit IRC | 15:37 | |
| fungi | that's not a bad idea at all | 15:37 |
| fungi | even just starting with /etc/project-config would probably help | 15:38 |
| *** EmilienM|afk is now known as EmilienM | 15:39 | |
| fungi | i'll hack something up for that | 15:39 |
| *** tkelsey has quit IRC | 15:40 | |
| clarkb | the zuul cloner map can probably just specify a second set of outputs including /etc/project-config | 15:40 |
| *** arxcruz has quit IRC | 15:41 | |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: doc: set name-filter in example to a real value https://review.openstack.org/170542 | 15:41 |
| clarkb | sdague: when looking at os-loganalyze stuff I figured out that the swift tests were't running against the swift code. https://review.openstack.org/#/c/170326/1 should fix that | 15:41 |
| openstackgerrit | Antoine Musso proposed openstack-infra/nodepool: doc: set name-filter in example to a real value https://review.openstack.org/170542 | 15:42 |
| bknudson | gerrit shows review is being tested when it's complete: https://review.openstack.org/#/c/157930/ | 15:43 |
| clarkb | sdague: and jhesketh points out his change to deal with non text files which I can review with a much better understanding of the problem | 15:44 |
| bknudson | after running "check experimental" | 15:44 |
| *** kerpan has quit IRC | 15:44 | |
| *** packet has joined #openstack-infra | 15:44 | |
| *** dims is now known as dimsum__ | 15:44 | |
| fungi | bknudson: because it still is | 15:45 |
| fungi | bknudson: "experimental-tripleo pipeline" | 15:45 |
| sdague | clarkb: ok | 15:45 |
| bknudson | oh, I get 2 results? | 15:46 |
| fungi | bknudson: search in http://status.openstack.org/zuul/ for 157930 | 15:46 |
| bknudson | jenkins already posted "build succeeded" | 15:46 |
| clarkb | sdague: but in general jhesketh's change is more complete than mine, but I think some of the code refactoring I have done should be incorporated into jheskeths change as it simplifies the swift iteration | 15:46 |
| *** medieval1 has quit IRC | 15:46 | |
| *** AJaeger has joined #openstack-infra | 15:46 | |
| openstackgerrit | Merged openstack-infra/os-loganalyze: Fix swiftclient tests https://review.openstack.org/170326 | 15:46 |
| *** harlowja_at_home has joined #openstack-infra | 15:46 | |
| fungi | bknudson: yep, that was from the experimental pipeline, but it's also triggered jobs in the separate experimental-tripleo pipeline which hasn't reported back yet | 15:46 |
| bknudson | ahh, thanks. | 15:47 |
| fungi | bknudson: looks like the check-tripleo-ironic-overcloud-f20-nonha job is still running on it | 15:47 |
| *** claudiub has joined #openstack-infra | 15:51 | |
| *** jistr has quit IRC | 15:53 | |
| *** jistr has joined #openstack-infra | 15:53 | |
| *** radez is now known as radez_g0n3 | 15:56 | |
| zaro | morning | 15:58 |
| *** unicell1 has joined #openstack-infra | 15:59 | |
| *** pelix has quit IRC | 15:59 | |
| *** unicell has quit IRC | 15:59 | |
| AJaeger | reed: are you around? | 16:00 |
| reed | AJaeger, yep | 16:00 |
| *** pelix has joined #openstack-infra | 16:00 | |
| AJaeger | You accusted the wrong person. | 16:01 |
| AJaeger | The comments you make are relative to patch 8 - see https://review.openstack.org/#/c/138195/8..10/doc/source/sandbox.rst | 16:01 |
| *** harlowja_at_home has quit IRC | 16:01 | |
| AJaeger | And patch 8 was uploaded by you. | 16:01 |
| AJaeger | My change was just moving the toc entry out... | 16:01 |
| zaro | fungi, mordred, clarkb, jeblair : i have a script ready for the gerrit db migration. i have tested on my MySQL ver 5.5 db and it seems to work. If you would like to review you can find it here: https://github.com/zaro0508/gerrit-db-migration | 16:03 |
| zaro | i'm planning to run it on review-dev this weekend just to make sure it works with the same version of MySQL as on review.o.o | 16:03 |
| *** hdd has joined #openstack-infra | 16:04 | |
| reed | AJaeger, I apologize, i'm a dork, sorry | 16:04 |
| zaro | the last step to import is really slow, i don't know the exact time but i think it takes over 2hrs to run thru that script. | 16:04 |
| reed | AJaeger, I must have screwed up when rebasing, pulled a previous patchset | 16:05 |
| AJaeger | reed: Oops. Please readd what you removed... | 16:05 |
| * sdague stares a pypi angrily .... "distutils.errors.DistutilsError: Can't download https://pypi.python.org/packages/2.7/p/pyasn1/pyasn1-0.1.7-py2.7.egg#md5=15f079cabee01402bf86ca8b83356469: 502 Bad Gateway" | 16:05 | |
| sdague | actually, that's even more confusing | 16:06 |
| sdague | http://logs.openstack.org/95/160195/4/gate/gate-tempest-dsvm-postgres-full/17d6c91//logs/devstacklog.txt.gz#_2015-04-03_16_03_08_679 | 16:06 |
| sdague | because... we were using the mirror up until that point | 16:07 |
| *** annegentle has quit IRC | 16:07 | |
| fungi | indeed. it looked in http://pypi.IAD.openstack.org/simple/pyasn1/ | 16:08 |
| *** dtantsur is now known as dtantsur|afk | 16:09 | |
| fungi | interesting that it's grabbing a platform-specific egg rather than an sdist | 16:09 |
| fungi | also someone should tell the pyasn1 authors about wheels | 16:09 |
| mordred | fungi: it will always prefer platform specific eggs | 16:12 |
| mordred | fungi: if they exist | 16:12 |
| mordred | fungi: I'm more concerned that we have distutils downloading something | 16:12 |
| mordred | fungi: and not pip | 16:13 |
| fungi | yeah, just meant i found it interesting they're uploading platform-specific eggs rather than wheels | 16:13 |
| mordred | is pyasn1 a setuprequires? | 16:13 |
| fungi | mordred: i think so. the traceback is coming from under cryptography/setup.py | 16:14 |
| mordred | sigh | 16:14 |
| *** e0ne is now known as e0ne_ | 16:14 | |
| *** pelix has quit IRC | 16:14 | |
| timrc | mordred, So I'm getting Cross-Origin Request Blocked's when trying to access openstack.org graphite data via d3.json() call (that's buried in the API) | 16:14 |
| mordred | This setup.py script uses the setuptools 'setup_requires' feature because | 16:14 |
| mordred | this is required by the cffi package to compile extension modules. | 16:14 |
| mordred | fungi: ^^ that's in the cryptography setup.py | 16:15 |
| fungi | too fun | 16:15 |
| mordred | dstufft: that can't actually be right is it? | 16:15 |
| fungi | i wonder if we could drop cryptography from the list if we pinned to pyOpenSSL | 16:16 |
| fungi | er, to older pyOpenSSL | 16:16 |
| mordred | fungi: I would support that | 16:16 |
| mordred | fungi: we could also put cryptography's setup_requires before cryptography in our install_requires | 16:17 |
| mordred | so that they would already be there and not need to be distutils fetched | 16:17 |
| fungi | i think ianw was leaning toward suggesting that we stick to pyOpenSSL<=0.13 anyway because of the memory consumption problems introduced through cryptography from pycparser | 16:17 |
| *** jistr has quit IRC | 16:17 | |
| mordred | works for me | 16:17 |
| mordred | they also declare a depend on setuptools, fwiw | 16:18 |
| mordred | fungi: fwiw, if we wanted to do the other thing, the list is "six, cffi, enum34, and pyasn1" | 16:19 |
| *** SumitNaiksatam has joined #openstack-infra | 16:19 | |
| clarkb | sdague https://review.openstack.org/#/c/107267/ my comments there got a bit wall of texty, but I think they offer some good suggestions on how to make this better | 16:19 |
| clarkb | mordred: but not when running on pypy because setuptools as a install requires breaks there iirc | 16:20 |
| fungi | sure--that keeps us from grabbing them from not our mirror, though doesn't solve the oom condition we run into installing pyopenssl 0.14 | 16:20 |
| *** e0ne_ is now known as e0ne | 16:20 | |
| clarkb | sdague: but please do respond if you disagree, happy to talk about it | 16:20 |
| *** thedodd has quit IRC | 16:21 | |
| *** carl_baldwin has joined #openstack-infra | 16:21 | |
| sdague | clarkb: my brain is probably not quite up to reasonable digestion on this today. I'll switch gears monday and try to put on my os-loganalyze hat that day | 16:21 |
| *** e0ne has quit IRC | 16:21 | |
| clarkb | sdague: ok, jhesketh is out until late sunday EST anyways so that should be fine | 16:21 |
| *** pblaho has quit IRC | 16:21 | |
| clarkb | sdague: thank you for approving the swift test fix | 16:23 |
| *** _nadya_ has quit IRC | 16:26 | |
| *** thedodd has joined #openstack-infra | 16:27 | |
| *** sarob has joined #openstack-infra | 16:27 | |
| openstackgerrit | Merged stackforge/python-jenkins: Fix examples https://review.openstack.org/165635 | 16:27 |
| krotscheck | Does anyone here sign their git commits? | 16:28 |
| * krotscheck is reading up on signed tags, and it turns out you can sign individual commits | 16:28 | |
| mordred | krotscheck: really? is that new-ish? | 16:29 |
| mordred | krotscheck: I used to sign my bzr commits | 16:29 |
| mordred | but did not when we moved to git | 16:29 |
| krotscheck | mordred: Apparently it's a git 1.7.9 thing (http://git.kernel.org/cgit/git/git.git/plain/Documentation/RelNotes/1.7.9.txt?id=HEAD) | 16:29 |
| hashar | git commit --gpg-sign=<keyid> | 16:29 |
| krotscheck | hashar: Also, git commit -S | 16:30 |
| hashar | or set git config: commit.gpgsign = keyid | 16:30 |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/puppet-graphite: Enable simple CORS requests https://review.openstack.org/170563 | 16:31 |
| reed | mordred, my point on the foundation list is mostly targeted at 'casual contributors' which seems to be the case that is usually quoted as an example (and in particular was brought up by sdague) | 16:31 |
| timrc | mordred, ^^ I *think* that's what will be needed to allow a "third-party" web service to pull data via the API. | 16:31 |
| reed | mordred, and I also hope to hear jbryce's proposal | 16:31 |
| mordred | reed: totally | 16:31 |
| mordred | reed: me too! | 16:31 |
| mordred | reed: it's a good one | 16:31 |
| sdague | reed: my concern is casual contributors is how we build regular contributors is how we build core reviewers | 16:32 |
| krotscheck | timrc: Lemme take a look, I've been digging around in CORS things for a while now. | 16:32 |
| clarkb | timrc: how is zuul status working without that? | 16:32 |
| krotscheck | timrc: I'll trade ya some time if you look at https://review.openstack.org/#/c/170260/ :D | 16:32 |
| reed | sdague, are you cascading? :) | 16:32 |
| sdague | and we keep kneecapping ourselves by being so hostile to casual contributors | 16:32 |
| clarkb | krotscheck: ^ if you have an insight into why some things don't seem to run into CORS trouble I would be interested to know | 16:32 |
| mordred | clarkb: zuul status uses embedded graphite urls | 16:32 |
| timrc | krotscheck, Thanks. | 16:32 |
| krotscheck | clarkb: What's the symptom? | 16:32 |
| clarkb | mordred: but those urls are the api | 16:33 |
| mordred | clarkb: graphene is an external javascript thing that wants to make api calls | 16:33 |
| *** Darkwan has joined #openstack-infra | 16:33 | |
| clarkb | mordred: how does graphenes api call differ from an HTTP GET from status.o.o? | 16:33 |
| mordred | clarkb: yes - but there is a difference when you're fetching a url from javascript to when you're embedding an image link in your html | 16:33 |
| mordred | clarkb: it's not an HTTP GET from status.o.o | 16:33 |
| clarkb | mordred: lets backup one setp | 16:33 |
| clarkb | I thought zuul status was doing js to generate the graphs on the status page | 16:33 |
| * clarkb goes to check | 16:34 | |
| mordred | nope | 16:34 |
| *** stevemar has joined #openstack-infra | 16:34 | |
| reed | sdague, I agree that we have a problem for casual contributors and that's why I propose to split the dco-cla debate off of that and discuss about casual contributions isolating the legal framework | 16:34 |
| mordred | clarkb: zuul status just has static graphite links embedded in the html | 16:34 |
| reed | being a casual contributor myself, I think the legal hurdle is one only and not the toughest | 16:34 |
| clarkb | mordred: http://paste.openstack.org/show/198153/ is what it does | 16:34 |
| sdague | reed: I'm glad you believe that for you personally | 16:35 |
| reed | i think forcing the gerrit workflow on casual contributors has bigger effect IMHO | 16:35 |
| clarkb | mordred: but I am guessing that the render/ path + whatever an Image() object is is equivalent to embedding an img? | 16:35 |
| krotscheck | clarkb: I need to take a look at the actual network request log, is that avialable? | 16:36 |
| reed | sdague, and the gerrit workflow is also the tool that forces the cla to come into play | 16:36 |
| sdague | but I don't believe your personal experience working at an organization that has a legal team created exactly for this is representative | 16:36 |
| sdague | reed: only because we made it | 16:36 |
| mordred | clarkb: yes, I belive so | 16:36 |
| clarkb | krotscheck: just reload the zuul status page with network tracing on | 16:36 |
| reed | sdague, exactly... what if for 5 lines of code we say "put it in a tracker, say it's public domain, I'll merge it??" | 16:36 |
| clarkb | mordred: ok I am less confused now, and I can try to confirm by doing what krotscheck asks about | 16:36 |
| reed | or something similar, like many other open source projects do | 16:37 |
| sdague | reed: because that's completely invisible to normal review process | 16:37 |
| reed | or let's brainstorm on other ways to be friendly to casual contributors | 16:37 |
| krotscheck | Interesting, I'm not seeing a browser preflight options request | 16:37 |
| *** rbradfor has quit IRC | 16:38 | |
| reed | but without entangling ourselves in a huge legal conversation (which will happen anyway but maybe at a different pace) | 16:38 |
| sdague | reed: you do realize that I pointed here that the *only* reason this patch isn't getting into kilo is the CLA, right? | 16:38 |
| clarkb | mordred: krotscheck ya its generating an img tag with the render/ path as the source | 16:38 |
| *** stevemar has quit IRC | 16:38 | |
| sdague | all the other parts were easy | 16:38 |
| sdague | it's blocked on the CLA | 16:38 |
| sdague | which is *my* experience in dealing with contributors that are of the level that they can provide a nova patch in the first place | 16:39 |
| reed | sdague, I see your point, I don't think that the CLA is the root cause, IMHO it's a symptom | 16:39 |
| krotscheck | An img tag? Wat? | 16:39 |
| sdague | reed: I disagree | 16:39 |
| clarkb | krotscheck: ya thats how it gets around it | 16:39 |
| sdague | gerrit might be a stumbling block for folks on projects where advanced git things aren't in their normal workflow | 16:40 |
| *** hashar is now known as hasharAway | 16:40 | |
| reed | i followed closely a similar case, with heat and Kevin Fox, and he was not authorized to merge his patches, DCO or CLA would have not made a difference | 16:40 |
| clarkb | krotscheck: timrc but I imagine that as soon as your js does a GET and not the browser itself, then CORS comes into play | 16:40 |
| clarkb | mordred: ^ so you are correct | 16:40 |
| *** yamamoto has quit IRC | 16:40 | |
| sdague | Kevin was a special case because of the oddities of gov contracting | 16:40 |
| reed | his employer was not authorizing his contributions | 16:40 |
| krotscheck | clarkb: Yep, that's likely. Except the browser's supposed to be smart enough to not give you access to the content of a cross domain image's bitstream. | 16:40 |
| clarkb | krotscheck: not firefox and chromium :) | 16:41 |
| krotscheck | clarkb: Or chrome for that matter. | 16:41 |
| mordred | so - literally no browsers do that | 16:41 |
| reed | sdague, not really, no, that was not exactly it... he simply didn't have authorization by his employer to contribute under the apache license (things changed now, how the employer signed the CCLA) | 16:41 |
| reed | s/how/now | 16:41 |
| mordred | because IE isn't a browser anymore :) | 16:41 |
| *** rbradfor has joined #openstack-infra | 16:41 | |
| krotscheck | mordred: As far as I'm concerned, caniuse.com still lists spartan under 'ie' | 16:42 |
| *** thingee has joined #openstack-infra | 16:42 | |
| reed | sdague, gerrit workflow can be convoluted and confusing and frustrating for casual contributor, I'd suggest looking at it with a critical eye | 16:42 |
| sdague | reed: for people not used to advanced git things, sure | 16:43 |
| mordred | reed: it totally can, I agree | 16:43 |
| sdague | but that *is not* the problem here | 16:43 |
| krotscheck | Correction, they're calling it "IE TP" | 16:43 |
| krotscheck | Technical Preview | 16:43 |
| * krotscheck first thought Toilet Paper | 16:43 | |
| reed | sdague, also for people who don't care about openstack as a long term investment | 16:43 |
| sdague | which is pissing me off that I presented a concrete situation | 16:43 |
| sdague | and you are now saying it's gerrit's fault | 16:43 |
| mordred | reed: right. but we're talking about a specific person | 16:43 |
| sdague | when it's not | 16:43 |
| mordred | ++ | 16:43 |
| *** claudiub has quit IRC | 16:44 | |
| reed | sdague, if he is interested in a long term involvment with openstack then let's talk | 16:44 |
| clarkb | timrc: I am fine with your change, but I also don't claim to be an expert in all of the rules at play here. If fungi and krotscheck are cool with it then my +2 can count for something | 16:44 |
| mordred | reed: he's interested in submitting a patch, and gerrit is not a blocker for him | 16:44 |
| sdague | reed: for f*** sake | 16:44 |
| jeblair | let's just turn the cla enforcement off and let anyone authenticate to gerrit with any openid | 16:44 |
| reed | sdague, if he's a casual contributor, as in 'I have to quickly fix my issue and go do something else', then the case is IMO different | 16:44 |
| jeblair | then we'll see how much of a problem it is | 16:44 |
| mordred | reed: so, while gerrit may be an interesting theoretical conversation, it's irrelevant here | 16:44 |
| *** Swami has joined #openstack-infra | 16:44 | |
| sdague | there is now way we are growing an open community if the answer to become a contributor is to have a 1x1 conversation with you reed | 16:45 |
| mordred | reed: beacuse gerrit IS NOT A BLOCKER in this specific case | 16:45 |
| mordred | jeblair: ++ | 16:45 |
| reed | sdague, that's not what I'm advocating :) I don't scale | 16:45 |
| krotscheck | clarkb timrc: I actually don't think that patch is going to do everything you need it to. Sec, will provide a somewhat sane update taht _might_ work. | 16:45 |
| sdague | "<reed> sdague, if he is interested in a long term involvment with openstack then let's talk" | 16:45 |
| clarkb | krotscheck: cool, thanks for the input | 16:45 |
| sdague | you know the way you build an open community, you accept what people can give you, and when they can give it to you | 16:46 |
| reed | sdague, that sentence means that for long term involvement the CCLA is worth the investement | 16:46 |
| sdague | reed: I do not believe that it is | 16:46 |
| mordred | I don't either | 16:46 |
| reed | means "it's worth talking to his lawyers" | 16:46 |
| mordred | he may not have lawyers | 16:46 |
| mordred | ALSO | 16:46 |
| reed | sorry.. and the DCO is not a legal document by itself? | 16:47 |
| mordred | talking to his lawyers about contribution and talking to his lawyers about having his lawyers sign a thing is different | 16:47 |
| *** medieval1 has joined #openstack-infra | 16:47 | |
| timrc | krotscheck, Cool. Definitely not my area of expertise. | 16:47 |
| mordred | I know this to be true | 16:47 |
| mordred | because I"m going through it right now | 16:47 |
| mordred | "yes, it's fine, go ahead" is different than "yes, our legal team will sign that document" | 16:47 |
| reed | I get that the DCO is a lot simpler, more elegant solution etc... I don't buy the argument that it's that different, it still requires a chat with a lawyer | 16:47 |
| mordred | and the first can be given quite quickly in many cases | 16:47 |
| mordred | chat with lawyer not the problem | 16:48 |
| reed | mordred, I get it | 16:48 |
| mordred | chat with lawyer is easy | 16:48 |
| krotscheck | timrc: I lied. | 16:48 |
| mordred | lawyer singing document on behalf of company VERY HARD | 16:48 |
| mordred | reed: k | 16:48 |
| mordred | if I wasn't going through the hell of the second thing right now, I might not feel so strongly about it | 16:48 |
| reed | :) | 16:48 |
| SpamapS | sdague: the main thing that breaks down there, is that some people might give you what isn't actually theirs to give. But I don't think the CLA does much to ensure that.. they're just violating a license _and_ a CLA when they give you things that aren't theirs after signing the CLA. ;) | 16:48 |
| mordred | SpamapS: ++ | 16:49 |
| sdague | SpamapS: yeh, it's all belts and suspenders and *no pants* | 16:49 |
| reed | SpamapS, yeah, that's another thing :) | 16:49 |
| mordred | so few pants | 16:49 |
| mordred | I'd go so far as to say that CLAs are entirely emporers' new clothes | 16:49 |
| mordred | they make lawyers happy because they ensure that more lawyers get paid | 16:50 |
| *** lpetrut has left #openstack-infra | 16:50 | |
| sdague | so all we are doing is kneecapping ourselves under some inane legal theory that we can go sue Company X when someone that signed all the papers didn't listen to a thing | 16:50 |
| mordred | but they provide literlaly nothing to anyone | 16:50 |
| mordred | other than lawyer happiness | 16:50 |
| sdague | which is never actually going to happen | 16:50 |
| SpamapS | I have always felt that the OpenStack CLA was mainly so OpenStack and it's many corporate entities can point at something and say they did their due dilligence in asking people to assert that they were going to be good, rather than just trusting that they'd be good. | 16:50 |
| mordred | yup | 16:50 |
| clarkb | jeblair: I am going to review the 5s zuul delay change and some of the others stacked atop your find needed by change. hopefully get that in before you restart zuul | 16:50 |
| krotscheck | timrc, clarkb: Added comments | 16:50 |
| krotscheck | +1 as far as I'm concerned. | 16:51 |
| SpamapS | sdague: I never thought it was a counter-attack strategy. Just defense in depth. | 16:51 |
| krotscheck | Unless you want access to wacky headers, in which case my comment also provides a solution via the Expose-Headers bit. | 16:51 |
| sdague | but, more importantly, I basically continues to enforce the fact that non corporate sponsor development is unwelcomed in OpenSTack | 16:51 |
| *** medieval1 has quit IRC | 16:51 | |
| sdague | which, if true, means the life of the project is going to be a lot less. We need to encourage all kinds of new blood, and the CLA drives off a lot of would be contributors | 16:52 |
| *** sdake_ has quit IRC | 16:52 | |
| *** hdd has quit IRC | 16:52 | |
| clarkb | asselin_: small suggestion on https://review.openstack.org/#/c/169527/ | 16:52 |
| SpamapS | sdague: I wonder if we do enough to try and asses who it drives off. | 16:53 |
| clarkb | krotscheck: I don't think the graphite api is that funky, but I may be wrong | 16:53 |
| *** sdake has joined #openstack-infra | 16:53 | |
| clarkb | sdague: ya, its always the least fun part of introducing anyone to openstack dev | 16:53 |
| clarkb | "and now we have to sign a CLA" "WAT" | 16:53 |
| *** hdd has joined #openstack-infra | 16:53 | |
| sdague | SpamapS: how would you accurately measure that? Most of those people never even get to the conversation because they read the CLA and say, f that, I'll go look at mesos or docker | 16:54 |
| sdague | or spend my time somewhere else | 16:54 |
| mordred | to be fair - _I_ say that some days | 16:54 |
| *** patrickeast has joined #openstack-infra | 16:54 | |
| openstackgerrit | Andrew Boik proposed openstack-infra/devstack-gate: WIP: (experiment) Test dual-stack with dependency https://review.openstack.org/168063 | 16:54 |
| mordred | fungi: clarkb and krotscheck I are both good with https://review.openstack.org/170563 ... but I'd rather have you look at it real quick before I +A it | 16:55 |
| clarkb | sdague: wait | 16:55 |
| clarkb | mesos requires a cla right? | 16:55 |
| sdague | clarkb: maybe | 16:55 |
| clarkb | its apache | 16:55 |
| sdague | no, apache projects only have CLA for maintainers | 16:56 |
| SpamapS | sdague: I don't think we can accurately measure it. I'm just thinking we could try to get some idea of who hits the CLA and then bounces off. | 16:56 |
| clarkb | sdague: ah | 16:56 |
| timrc | krotscheck, Thanks much for the comment. I only need GET and AFAICT that's the only thing d3.json() does. I don't see an OPTIONS verb anywhere. | 16:56 |
| sdague | the fact that we have a CLA for everyone is because no one actually read the apache process before they wrote ours | 16:56 |
| *** unicell1 has quit IRC | 16:56 | |
| *** _nadya_ has joined #openstack-infra | 16:57 | |
| sdague | SpamapS: well, I'll tell you right now, I'll never contribute to a project that requires a CLA unless it was actually my job (as in the OpenStack case) | 16:57 |
| clarkb | timrc: ya I think the graphite api is built around crazy parameter lists | 16:57 |
| sdague | so, that's 1 data point :) | 16:57 |
| clarkb | timrc: so you avoid too much trouble with other options that need to be set | 16:57 |
| *** YorikSar has joined #openstack-infra | 16:57 | |
| *** jogo is now known as flashgordon | 16:57 | |
| *** annegentle has joined #openstack-infra | 16:58 | |
| *** yamahata has joined #openstack-infra | 16:58 | |
| *** sabeen1 has left #openstack-infra | 16:58 | |
| sdague | clarkb: if we actually implemented the apache version of this only people with +2 authority would have to sign the CLA. Because it's assumed they can evaluate it before committing the code. | 16:58 |
| clarkb | sdague: I see | 16:58 |
| clarkb | thats still an odd requirement imo | 16:58 |
| *** sabeen1 has joined #openstack-infra | 16:59 | |
| clarkb | "we trust you, but only if you sign this document that isn't directly related to all of what we trust you with" | 16:59 |
| sdague | yeh, that's fine, I'm not sure I agree with it either, but it makes a bit more sense than our crazy pants thing | 16:59 |
| clarkb | yes its much more welcoming to the masses | 17:00 |
| krotscheck | mordred: Comment added | 17:00 |
| *** sarob has quit IRC | 17:02 | |
| *** sarob has joined #openstack-infra | 17:02 | |
| *** harlowja_away is now known as harlowja | 17:02 | |
| *** teran has joined #openstack-infra | 17:03 | |
| *** annegentle has quit IRC | 17:04 | |
| *** annegentle has joined #openstack-infra | 17:04 | |
| *** otter768 has joined #openstack-infra | 17:05 | |
| *** tosky has quit IRC | 17:06 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:07 | |
| *** tonytan4ever has quit IRC | 17:07 | |
| *** otter768 has quit IRC | 17:10 | |
| clarkb | jeblair: see comment on https://review.openstack.org/#/c/170319/ though I may be entirely wrong about that | 17:10 |
| fungi | timrc: mordred: krotscheck: clarkb: i'm pondering it now | 17:11 |
| *** hogepodge has quit IRC | 17:12 | |
| timrc | fungi, Thanks. Definitely need to defer to you guys on this one. | 17:12 |
| *** hogepodge has joined #openstack-infra | 17:14 | |
| fungi | sdague: my take on the cla was that (at least a few of) the member company's lawyers felt it protected them from the risk that someone intentionally introduced patent-encumbered code, waited for it to proliferate, and then sued other member companies over patent infringement while insisting that they never gave up their patent claim because they never signed a contract stating they would | 17:15 |
| fungi | so basically part of an intellectual property arms race between member companies trying to protect themselves from each other, and the individual contributors are the ones caught in the middle agreeing to inane legal documents | 17:16 |
| *** skolekonov has quit IRC | 17:17 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: keystone bandit job on check https://review.openstack.org/170569 | 17:17 |
| *** sarob has quit IRC | 17:17 | |
| fungi | like you, i would never in my free time contribute to a project that required me to sign a cla. i'd love for openstack to eventually not require one (not even a dco, but it seems like that option's being presented as a possible stepping stone away from a more formal contract) | 17:18 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/zuul: Update zuul-changes.py doc to match usage https://review.openstack.org/169527 | 17:18 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: keystone bandit job on check https://review.openstack.org/170569 | 17:18 |
| fungi | i still cling to the idea that people can contribute to a free software project and the project can take it as granted that their contribution is provided under the terms the project itself is licensed (or at least terms stated in the file in which the contribution is situated) | 17:20 |
| mordred | fungi: ++ | 17:20 |
| mordred | fungi: you and every other free software developer | 17:20 |
| fungi | the more projects start to add cla/dco requirements, the more it undermines the legality of the licenses themselves by creating precedent indicating they're insufficient on their own | 17:21 |
| *** leakypipes has quit IRC | 17:21 | |
| fungi | a slippery slope which i worry will lead to us abandoning the licenses altogether, without them ever even being challenged in court in the first place to test their sufficiency | 17:22 |
| *** spzala has joined #openstack-infra | 17:24 | |
| clarkb | fungi: you reviewed https://review.openstack.org/#/c/170319/ as well, any chance you want to check my comment when you have a moment | 17:25 |
| sdague | fungi: yes, I know exactly which corporate lawyers believe that theory | 17:26 |
| clarkb | and now time for breakfast | 17:27 |
| *** links has quit IRC | 17:27 | |
| fungi | sdague: you and me both | 17:27 |
| *** david-lyle has quit IRC | 17:29 | |
| *** dannywilson has joined #openstack-infra | 17:29 | |
| *** sputnik13 has quit IRC | 17:29 | |
| *** ivar-lazzaro has joined #openstack-infra | 17:29 | |
| *** medieval1 has joined #openstack-infra | 17:29 | |
| *** Guest27546 has quit IRC | 17:29 | |
| clarkb | sdague: fyi I have rechecked https://review.openstack.org/#/c/168407/ a second time. I read logs for each failure before doing so and am fairly confident that my change is not to blame. The first fail had to do with a libvirt socket connection error during a tempest run and the second was keystone did not shut down during the neutron grenade run | 17:30 |
| *** melwitt has joined #openstack-infra | 17:30 | |
| clarkb | sdague: just letting you know in case you are worried the above may be related to my change | 17:30 |
| sdague | clarkb: volumes test failures? | 17:31 |
| *** yamahata has quit IRC | 17:31 | |
| clarkb | sdague: let me double check | 17:31 |
| clarkb | sdague: tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_confirm | 17:31 |
| *** yamahata has joined #openstack-infra | 17:32 | |
| clarkb | sdague: from http://logs.openstack.org/07/168407/9/gate/gate-tempest-dsvm-full/c660fe7/console.html | 17:32 |
| clarkb | I would expect my change to break in libvirt much harder. eg the cpu model would be invalid and all libvirt node boots would fail | 17:32 |
| clarkb | which is why I don't think it is related to that fail | 17:32 |
| clarkb | (note the hard failing is what I saw when my change was not correct) | 17:33 |
| fungi | clarkb: see my reply on 170319... am i misreading that method that the chances of piling on sleep calls is higher than i think it is? | 17:33 |
| clarkb | fungi: gah you are right, I didn't realize that that method short circuits | 17:34 |
| clarkb | I will update my review, and thanks for the sanity check | 17:34 |
| sdague | clarkb: yeh, there is a different failing around keystone not stopping, not sure | 17:35 |
| openstackgerrit | Merged openstack-infra/puppet-graphite: Enable simple CORS requests https://review.openstack.org/170563 | 17:36 |
| *** btully has joined #openstack-infra | 17:36 | |
| *** SumitNaiksatam has quit IRC | 17:37 | |
| *** Darkwan has quit IRC | 17:38 | |
| fungi | btw, nodepoold has been running for a while now configured to use pymysql. so far no behaviors out of the ordinary for it | 17:38 |
| fungi | also we're approaching a day's worth of the new stats for it in graphite, though i haven't tried to graph anything interesting from them yet | 17:39 |
| fungi | i assume timrc will have something interesting to show us once he finishes wrestling graphene and cors | 17:39 |
| *** tqtran has joined #openstack-infra | 17:40 | |
| *** btully has quit IRC | 17:41 | |
| clarkb | fungi: neat | 17:41 |
| clarkb | ok really going to go find food now that I have updated my review on the zuul change | 17:42 |
| *** ivar-lazzaro has quit IRC | 17:42 | |
| *** hdd has quit IRC | 17:45 | |
| dstufft | mordred: whats' up? | 17:46 |
| *** hdd has joined #openstack-infra | 17:46 | |
| zaro | waynr: had questions for you in https://review.openstack.org/150597 & 150598 | 17:46 |
| mordred | dstufft: just distutils pain - but I think we sorted it | 17:48 |
| *** bknudson has quit IRC | 17:48 | |
| *** btully has joined #openstack-infra | 17:49 | |
| fungi | "we hateses the setup_requires, we does" | 17:50 |
| dstufft | fungi: mordred ah, lifeless is working on making setup_requires better, though there was stuff that had to be done first | 17:51 |
| fungi | yeah, cryptography setup_requires pyasn1 so setuptools grabbed it from pypi.python.org rather than our mirror, and then promptly got a 5xx error from fastly, presumably | 17:52 |
| sdague | any idea why this isn't going into the gate - https://review.openstack.org/#/c/162612 ? | 17:52 |
| fungi | sdague: Queue lengths: 731 events, 74 results. | 17:54 |
| *** sarob has joined #openstack-infra | 17:54 | |
| fungi | your most recent readding of the approval is likely being worked through in that events pile | 17:55 |
| sdague | fungi: ok, gotcha | 17:55 |
| fungi | i think your original approval on the 26th happened when gerrit was broken | 17:56 |
| *** annegentle has quit IRC | 17:56 | |
| fungi | and rechecking didn't have the desired effect because there was already a +1 from jenkins, so the new +1 left no vote | 17:56 |
| *** achanda has joined #openstack-infra | 17:57 | |
| fungi | yeah, it's up at the top of the gate pipeline now | 17:57 |
| fungi | so seems to have worked | 17:57 |
| *** tonytan4ever has joined #openstack-infra | 17:58 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:03 | |
| *** SumitNaiksatam has left #openstack-infra | 18:03 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:05 | |
| *** akshai has joined #openstack-infra | 18:06 | |
| *** baoli has quit IRC | 18:08 | |
| *** rmcall has joined #openstack-infra | 18:08 | |
| *** armax_ has joined #openstack-infra | 18:09 | |
| *** armax has quit IRC | 18:10 | |
| *** armax_ is now known as armax | 18:10 | |
| *** hdd has quit IRC | 18:10 | |
| *** leakypipes has joined #openstack-infra | 18:10 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add support for Sonar analysis builder https://review.openstack.org/163961 | 18:11 |
| *** dboik_ has joined #openstack-infra | 18:12 | |
| *** sputnik13 has joined #openstack-infra | 18:12 | |
| *** mjordan has joined #openstack-infra | 18:15 | |
| *** dboik has quit IRC | 18:15 | |
| *** armax has quit IRC | 18:17 | |
| sdague | fungi: yep, seems working now, thanks | 18:18 |
| *** sdake_ has joined #openstack-infra | 18:20 | |
| *** achanda has quit IRC | 18:20 | |
| *** ujuc has quit IRC | 18:22 | |
| *** achanda has joined #openstack-infra | 18:24 | |
| *** sdake has quit IRC | 18:24 | |
| *** rmcall has quit IRC | 18:25 | |
| *** yamamoto has joined #openstack-infra | 18:28 | |
| timrc | fungi, I'm hoping :) | 18:31 |
| timrc | fungi, My little dashboard is going to side-to-side comparisons it can be region-to-region. Conceptually you can to a side-by-side of the same region but different periods of time. | 18:32 |
| timrc | to do* | 18:33 |
| timrc | provider-to-provider, rather. | 18:33 |
| *** yamamoto has quit IRC | 18:33 | |
| fungi | gonna go grab a late lunch. back in a little while | 18:33 |
| *** viglesia1 has quit IRC | 18:33 | |
| *** viglesias has joined #openstack-infra | 18:35 | |
| *** annegentle has joined #openstack-infra | 18:35 | |
| clarkb | timrc: we probably want it to be region to region | 18:35 |
| clarkb | timrc: since rax iad != ord | 18:36 |
| clarkb | timrc: or do you mean provider to providerfrom nodepools perspective? since nodepool providers are basically regions or meta regions | 18:36 |
| timrc | clarkb, From nodepools perspective. | 18:37 |
| clarkb | timrc: perfect | 18:37 |
| clarkb | also we overloda all the terms | 18:37 |
| *** alexpilotti has quit IRC | 18:37 | |
| *** annegentle has quit IRC | 18:38 | |
| morganfainberg | sdague, an extra info on why keystone didn't stop and/or bug would be appreciated. | 18:41 |
| timrc | clarkb, Someone will probably need to reload the apache conf for graphite.openstack.org for the change to take effect. | 18:41 |
| clarkb | timrc: puppet should do it for us | 18:42 |
| timrc | clarkb, Ah, okay. | 18:42 |
| clarkb | timrc: http://puppetboard.openstack.org/report/graphite.openstack.org/299b718708f7a3abee8fc10082004092baff6ae3 | 18:42 |
| clarkb | apache why | 18:43 |
| *** annegentle has joined #openstack-infra | 18:43 | |
| clarkb | we need mod_headers enabled | 18:43 |
| timrc | Ah blah.... maybe 'a2enmod headers' is needed? | 18:44 |
| clarkb | timrc: ^ I can manually a2enmod but can you submit a change to do that in puppet? | 18:44 |
| timrc | hivemind. | 18:44 |
| timrc | clarkb, Sure. | 18:44 |
| clarkb | apache is up now after enabling mod headers | 18:44 |
| *** teran has quit IRC | 18:44 | |
| *** thingee has quit IRC | 18:47 | |
| *** sarob has quit IRC | 18:47 | |
| *** sarob has joined #openstack-infra | 18:47 | |
| *** thingee has joined #openstack-infra | 18:47 | |
| *** teran_ has joined #openstack-infra | 18:47 | |
| *** rhe00 has joined #openstack-infra | 18:50 | |
| *** akshai has quit IRC | 18:51 | |
| mrodden | is there an enforced policy of what can or can't go into stackforge? | 18:52 |
| mrodden | other than meeting the open source criteria | 18:52 |
| *** armax has joined #openstack-infra | 18:55 | |
| *** teran_ has quit IRC | 18:55 | |
| mordred | mrodden: not really | 18:56 |
| mordred | mrodden: generally speaking somewhat related to openstack makes sense though | 18:56 |
| *** sdake has joined #openstack-infra | 18:56 | |
| mrodden | yeah i have a python interface for dealing with our internal issue tracker | 18:57 |
| mrodden | and i'd like to get it out in public since its useful | 18:57 |
| mrodden | but i like gerrit | 18:57 |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/puppet-graphite: Ensure the 'headers' module is enabled https://review.openstack.org/170582 | 18:57 |
| clarkb | cinerama: around? want to talk about the zanata apache change | 18:57 |
| clarkb | cinerama: I have one idea I want to bounce off of you | 18:57 |
| cinerama | hi clarkb | 18:57 |
| cinerama | sure | 18:57 |
| clarkb | cinerama: in standalone.xml.erb instead of having 3 different boolean flag to select the port binding. What if we have one variable called socket_binding (or similar) with a default value of "http" but it can also accept "https" and "ajp"? | 18:58 |
| clarkb | cinerama: I always find that a massive number of interrelated boolean flags gets confusing | 18:58 |
| timrc | clarkb, mordred, fungi, krotscheck -- Yay, I can fetch metrics data now :) | 18:59 |
| clarkb | cinerama: but if you use a string var like that you should be able to pass it straight thorugh to the template without any conditionals | 18:59 |
| cinerama | clarkb: that could be good. i'm just trying to think if there's a reason non-infra users might want to have more than one enabled | 19:00 |
| cinerama | clarkb: but i think actually that's the way to go | 19:00 |
| *** sdake_ has quit IRC | 19:00 | |
| *** baoli has joined #openstack-infra | 19:00 | |
| clarkb | cinerama: ok let me write that as a comment, and I think I have one -1 thing in the vhost template | 19:00 |
| clarkb | cinerama: should have them up shortly | 19:00 |
| openstackgerrit | Merged openstack-infra/devstack-gate: Add --assume-yes option to apt-get https://review.openstack.org/162612 | 19:01 |
| cinerama | clarkb: great. i want to start getting these landed even if they're not perfect so we can get on the server and do more/better tests | 19:01 |
| *** hasharAway is now known as hashar | 19:01 | |
| clarkb | cinerama: yup | 19:01 |
| mordred | mrodden: that would probably be the least-related-to-openstack thing :) | 19:02 |
| mrodden | yeah i know | 19:02 |
| mrodden | i'd rather not set up my own external gerrit | 19:02 |
| mrodden | running one is enough work | 19:02 |
| clarkb | cinerama: actually I may not have a -1 for the vhost. /me is going to think out loud on that one here too | 19:02 |
| clarkb | cinerama: so, apache.pp allows for not setting http cert and cert key files | 19:02 |
| clarkb | cinerama: but the vhost makes them required | 19:02 |
| mrodden | guess i'll just deal with github.com pull requests for now | 19:02 |
| mtreinish | mrodden: I think just give it a flashy name and it's fine for stackforge :) | 19:03 |
| mrodden | lol | 19:03 |
| clarkb | cinerama: oh wait nevermind :) | 19:03 |
| mrodden | megawaffleclient9000 | 19:03 |
| clarkb | cinerama: the manifest uses snakeoil certs by default | 19:03 |
| clarkb | cinerama: so its fine to require https. So no -1 just the suggestion from earlier will post now | 19:03 |
| cinerama | clarkb: yeah, otherwise we could let people pick http/https and setup the vhost properly etc. i had put that in the "maybe do later" bin | 19:04 |
| *** sdake_ has joined #openstack-infra | 19:05 | |
| mtreinish | mrodden: heh, I'd give it a +2 if I had +2 on project config (probably a good thing I don't :) ) | 19:05 |
| *** otter768 has joined #openstack-infra | 19:05 | |
| clarkb | cinerama: ya don't worry about it, I just had to work it out for myself. Comments about the other thing posted | 19:07 |
| clarkb | cinerama: I did not approve because I wanted to give you a chance to update the change or decide to go with change as is and fix in followup | 19:08 |
| *** sdake has quit IRC | 19:08 | |
| cinerama | clarkb: i can probably get that turned around fairly shortly | 19:09 |
| *** otter768 has quit IRC | 19:10 | |
| clarkb | fungi: for 164447 I guess the bindep fallback will only ever be as new as the image running the job? | 19:10 |
| clarkb | fungi: shoudl be fine that way since the intent is to have projects specify their own dpes which will be completely self testing | 19:11 |
| clarkb | its fun watching a package you ordered get stuck in german after transiting israel and turkey | 19:13 |
| clarkb | ups tracking numbers should not be this entertaining | 19:13 |
| clarkb | mordred: fungi if we can get 165682 in then I can rebase my multi image format output change on it (but I don't want to rebase until we know which will go first) | 19:15 |
| mordred | clarkb: you could write an IRC bot that informs you when the package is in a new location | 19:15 |
| *** sdake has joined #openstack-infra | 19:15 | |
| *** achanda has quit IRC | 19:15 | |
| clarkb | mordred: thats not a bad idea | 19:16 |
| clarkb | the other fun thing that happens is you will order UPS 2 day from a place in the same city but it will go to kentucky first before showing up at your door | 19:17 |
| *** tonytan4ever has quit IRC | 19:17 | |
| *** sdake_ has quit IRC | 19:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Add fallback bindep manifest https://review.openstack.org/164447 | 19:21 |
| *** wznoinsk has quit IRC | 19:21 | |
| fungi | clarkb: sure thing, looking now | 19:23 |
| *** tonytan4ever has joined #openstack-infra | 19:25 | |
| *** akshai has joined #openstack-infra | 19:26 | |
| *** YorikSar has quit IRC | 19:26 | |
| fungi | clarkb: on 164447 yeah the idea is that is a fallback file we'll use in jobs. as projects start to define their own bindep files the job definition (still in my head at the moment) will use those instead of the fallback. also makes them self-testing | 19:27 |
| *** dustins has quit IRC | 19:27 | |
| fungi | clarkb: i should also comment on that change that i've tested it on trusty and precise now and it does seem to successfully transform devstack-.* nodes to their bare-.* equivalents (centos6 is something i need to hack on today) | 19:28 |
| *** achanda has joined #openstack-infra | 19:29 | |
| fungi | clarkb: though in light of the bikeshed on platform identifiers in this week's meeting, any release-specific platform tags in that file will probably need a patch corresponding to an updated 157979 i haven't written yet | 19:29 |
| clarkb | fungi: rgr | 19:30 |
| fungi | clarkb: as far as bindep changes go, the one which would help the _most_ to get approved is 158098, because without that none of its jobs actually work | 19:30 |
| *** AJaeger has left #openstack-infra | 19:30 | |
| clarkb | fungi: I do find it a bit odd to differentiate on both distro and package format, but I suppose thats a short cut for when all the red hats and suses agree on a package name? | 19:30 |
| fungi | clarkb: you got it | 19:31 |
| clarkb | 158098 isn't in my magical priority task gerrit query | 19:31 |
| fungi | platform:rpm is when everybody we know of using redhat-based packaging uses the same name but debian-derived distros don't | 19:31 |
| *** e0ne has joined #openstack-infra | 19:31 | |
| clarkb | OH! I don't subscribe to bindep yet | 19:31 |
| * clarkb fixes | 19:31 | |
| fungi | seems marginally less magical ;_ | 19:32 |
| *** david-lyle has joined #openstack-infra | 19:33 | |
| *** vponomaryov has quit IRC | 19:34 | |
| *** dboik_ has quit IRC | 19:34 | |
| *** dboik has joined #openstack-infra | 19:35 | |
| *** rbradfor has quit IRC | 19:36 | |
| *** tkelsey has joined #openstack-infra | 19:37 | |
| clarkb | fungi: see tox.ini comments, is cookiecutter doing it wrong? | 19:39 |
| clarkb | fungi: sorry on 158098 | 19:39 |
| clarkb | fungi: if thats from cookiecutter I will approve it then go fix cookiecutter | 19:39 |
| *** mrmartin has quit IRC | 19:40 | |
| *** tkelsey has quit IRC | 19:42 | |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Add apache configuration for zanata https://review.openstack.org/164011 | 19:42 |
| fungi | clarkb: oh, i just responded, wip'd it and said i'd fix cookiecutter next | 19:43 |
| fungi | because they were all good points | 19:43 |
| clarkb | fungi: that works too :) | 19:43 |
| clarkb | fungi: also I am double checking that pbr does the test interception | 19:44 |
| *** spzala has quit IRC | 19:45 | |
| fungi | do i still need to pass in --testr-args='{posargs}' in that case? | 19:45 |
| clarkb | fungi: because it has been confusing enough in the past that I may have remember wrong | 19:45 |
| fungi | i'm testing it now too | 19:45 |
| clarkb | WOW | 19:46 |
| fungi | hrm, how do you ignore all the hacking rules? H* seems to not do it | 19:46 |
| clarkb | so pbr supplies both testr and test | 19:46 |
| fungi | H.*? | 19:46 |
| clarkb | fungi: I think just H | 19:46 |
| fungi | k | 19:46 |
| clarkb | fungi: so using the testr command is fine, pbr is doing that one and the test command both | 19:47 |
| cinerama | clarkb: please wait for the next cut - i've neglected the actual thrust of your comment | 19:47 |
| clarkb | however, if you use test, then nose will work as well | 19:47 |
| clarkb | cinerama: sure | 19:47 |
| *** amotoki has quit IRC | 19:47 | |
| clarkb | fungi: so for cookiecutter `setup.py test` is likely better, but for bindep it doesn't matter | 19:47 |
| fungi | i'm adjusting it in both assuming i can get tests conforming | 19:48 |
| fungi | W391 apparently just showed up out of nowhere | 19:48 |
| clarkb | mordred: ^ do you have an opinion on that? | 19:48 |
| clarkb | fungi: hahahaha | 19:48 |
| mordred | test | 19:48 |
| mordred | I think we prefer test to testr | 19:48 |
| *** annegentle has quit IRC | 19:49 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Sync with openstack-dev/cookiecutter https://review.openstack.org/158098 | 19:49 |
| fungi | clarkb: mordred: in that case ^ | 19:49 |
| fungi | cookiecutter equivalent tweak on the way now | 19:49 |
| clarkb | fungi: also ya, for coverage I don't know why people don't like looking at coverage of tests proper. Its super handy when your tests don't do what you think the should do :) | 19:50 |
| clarkb | the multiprcessing import in setup.py is special too | 19:51 |
| *** krtaylor has quit IRC | 19:51 | |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/cookiecutter: Don't omit tests from coverage https://review.openstack.org/170594 | 19:51 |
| clarkb | does setup.py use multiprocessing? | 19:51 |
| clarkb | mordred: ^ any idea why that exists? | 19:51 |
| *** krtaylor has joined #openstack-infra | 19:52 | |
| mordred | I do not - I think something broke in nova at one point | 19:52 |
| mordred | I do not put that in any of my projects | 19:52 |
| *** pblaho has joined #openstack-infra | 19:52 | |
| *** pblaho has quit IRC | 19:53 | |
| fungi | do i still need to pass in --testr-args='{posargs}' in that case? | 19:53 |
| clarkb | fungi: yes, thats used if test uses testr underneath | 19:54 |
| fungi | okay, just making sure | 19:54 |
| fungi | i suppose if you have to pass testr-specific options then setup.py test isn't necessarily any more portable | 19:55 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Make aiopcpu subnode directory more deterministic https://review.openstack.org/169449 | 19:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-dev/cookiecutter: More portable to use setup.py test not testr https://review.openstack.org/170596 | 19:55 |
| fungi | but at least this ^ way people only have to remove the non-pbr options from the line rather than change the entrypoint name | 19:56 |
| *** rlucio has joined #openstack-infra | 19:56 | |
| *** MarkAtwood has joined #openstack-infra | 19:56 | |
| *** teran has joined #openstack-infra | 19:56 | |
| openstackgerrit | Merged openstack-infra/bindep: Sync with openstack-dev/cookiecutter https://review.openstack.org/158098 | 19:56 |
| *** mmedvede has quit IRC | 19:58 | |
| *** medieval1 has quit IRC | 19:59 | |
| *** sarob has quit IRC | 20:00 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add a --file option https://review.openstack.org/164835 | 20:00 |
| *** Swami has quit IRC | 20:00 | |
| fungi | yikes, the in-browser commit message editor doesn't wrap lines | 20:01 |
| clarkb | mordred: Shrews you all are the openstack *client gurus now. What do you think of jhesketh's -1 on 155770? | 20:01 |
| *** teran has quit IRC | 20:01 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add a --file option https://review.openstack.org/164835 | 20:01 |
| fungi | that's less insane now | 20:02 |
| clarkb | mordred: Shrews jhesketh from what I can tell nova will accept None and False as equivalent, so I guess its a matter of sanity in our own code base? | 20:02 |
| *** mattfarina has quit IRC | 20:02 | |
| clarkb | but we will need that change for ra | 20:02 |
| clarkb | *for rax | 20:02 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/bindep: Add a --file option https://review.openstack.org/164835 | 20:02 |
| clarkb | fungi: 157978 and 157979 are the changes that need to be updated post bikeshedding? | 20:03 |
| fungi | clarkb: just 157979 i think | 20:04 |
| fungi | and then i'll adjust the file that 164447 added with a new change | 20:05 |
| fungi | since there are a couple lines there which need editing | 20:05 |
| clarkb | fungi: were you planning on adding fedora to 157978 or later changes? its not clear to me from your comment | 20:06 |
| fungi | clarkb: depended on whether someone approved that one before i got around to it | 20:06 |
| *** annegentle has joined #openstack-infra | 20:07 | |
| fungi | sort of trying to tackle all the missing bits in parallel without having them depend on each other where possible, so that we can get it eventually consistent with what we're going to need as reviewers have time to look at those changes | 20:07 |
| clarkb | I see, I can dig in after lunch probably | 20:08 |
| clarkb | right now just trying to understand what we have going on | 20:08 |
| fungi | you and me both | 20:08 |
| *** BobH has quit IRC | 20:08 | |
| *** YorikSar has joined #openstack-infra | 20:10 | |
| *** mmedvede has joined #openstack-infra | 20:10 | |
| fungi | mostly just implementing features/support it was missing to do the things i expect our initial use of it to need (running in nodepool prep scripts/dib elements to figure out what to cache, running under ci jobs to install things projects specify or install from a fallback we specify, support for the distros we initially need to run it on) | 20:10 |
| *** e0ne has quit IRC | 20:13 | |
| fungi | i also want to add some new jobs for it which are platform-specific rather than interpreter-specific | 20:13 |
| *** annegentle has quit IRC | 20:13 | |
| clarkb | asselin_: 169117 has a wall of text from me :) one small thing inline for the -1 though then the rest is me trying to plan out a bit more concretely what our module testing might look like | 20:16 |
| clarkb | asselin_: curious to know what you think | 20:16 |
| * asselin_ looking | 20:16 | |
| *** jonesbr1 has joined #openstack-infra | 20:17 | |
| *** jonesbr1 has left #openstack-infra | 20:17 | |
| asselin_ | clarkb, in a meeting...give me a few mins | 20:18 |
| *** annegentle has joined #openstack-infra | 20:19 | |
| armax | clarkb: ping | 20:20 |
| clarkb | armax: pong | 20:20 |
| armax | clarkb: I was wondering if you had a chance to look at https://review.openstack.org/#/c/158525/ | 20:20 |
| armax | clarkb: from my last conversation with swami he told me that you were looking for the possibility to leverage the same gre tunnel setup that n-net used | 20:21 |
| clarkb | armax: no one has checked the experimental jobs on the latest patchset | 20:21 |
| armax | clarkb: done now | 20:21 |
| clarkb | armax: yes, we already set up an overlay layer 2 network to make nova net work | 20:21 |
| clarkb | armax: I don't want to have a second different overlay layer2 network to make neutron work | 20:22 |
| clarkb | whether or not we use gre for that is less important, I just don't want two different everythings | 20:22 |
| armax | clarkb: well that would be acceptable if it worked | 20:22 |
| armax | clarkb: but it can't | 20:22 |
| clarkb | why can't it work? | 20:22 |
| clarkb | (also my original suggestion was to just use a single interface if possible and I never heard anything from that) | 20:23 |
| *** Krinkle|detached is now known as Krinkle | 20:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Cleanup jenkins/jobs/projects.yaml https://review.openstack.org/168148 | 20:23 |
| clarkb | (we can't use a single interface with nova net because ebtables, not sure if a problem with neutron) | 20:23 |
| waynr | zaro: sorry i haven't been responsive, been travelling for and dealing with a death in my family | 20:23 |
| clarkb | anyways I will be -1 on that change as long as it adds a second networking setup without justifying it | 20:24 |
| armax | swami noted that in the review, that said, I appreciate that we don’t want two different everythings but isn’t the objective of this patch? We’re adding neutron support, if we could do the same with one single logic this patch wouldn’t exist! | 20:24 |
| armax | clarkb: he did justify it | 20:24 |
| clarkb | armax: except the overlay networks shouldn't need to be different | 20:24 |
| clarkb | armax: are you telling me that if I had a nova network deployment and wanted to redeploy with neutron dvr I could not do that? | 20:24 |
| armax | neutron’s default is ovs, we can’t use brctl to control the underlying topology | 20:25 |
| *** annegentle has quit IRC | 20:25 | |
| clarkb | this is hopefully a bug | 20:25 |
| armax | of course not | 20:25 |
| armax | but what do you mean exactly? | 20:25 |
| *** JayJ_ has joined #openstack-infra | 20:25 | |
| clarkb | armax: ok lets back up, this has nothing to do with neutron and with nova | 20:25 |
| clarkb | armax: what these two functions do is set up an overlay network because cloud networking is silly | 20:25 |
| clarkb | armax: this gives us control of the network topology that we run neutron and nova net on | 20:25 |
| clarkb | my suggestion is that we come up with a single topology for both managed by a single function | 20:26 |
| clarkb | because it should be possibly to delete nova network and deploy neutron dvr without changing physical hardware layout | 20:26 |
| clarkb | now that may not be the layout we use for nova net today, but such a layout should exist | 20:26 |
| armax | clarkb: right, but even if you look at the way neutron is setup in devstack | 20:26 |
| armax | the logic is different from n-net | 20:26 |
| armax | we have two of those there | 20:26 |
| clarkb | armax: but this isn't a nova network vs neutron service configuration set up step | 20:27 |
| clarkb | armax: this is modeling physical hardware that we then run nova network or neutron on top of | 20:27 |
| armax | I mean the underlying system configuration | 20:27 |
| armax | the creation of bridges etc | 20:27 |
| flashgordon | greghaynes: how is the wheel mirror thing coming? | 20:27 |
| fungi | armax: these functions are here to emulate "someone plugged a crossover cable in between the network interfaces of your two servers" | 20:27 |
| armax | it’s totally differnt | 20:27 |
| armax | fungi: ovs must be made aware of that crossover cabling | 20:28 |
| armax | hence the additional logic | 20:28 |
| fungi | but the crossover cable itself shouldn't need to be removed and replaced with a new one just because you installed different software on the servers | 20:29 |
| clarkb | armax: but ovs is merely an implementation detail. We should be able to use tinc, gre, openvpn, etc and still run nova net/neutron over the top as long as the topology is fine | 20:29 |
| clarkb | fungi: yes this | 20:29 |
| armax | clarkb: we still use gre | 20:29 |
| armax | it’s how the tunnel endpoints are set up that change | 20:29 |
| fungi | can't ovs use bridged gre over top of the existing bridged gre which is smulating the physical network you'd have in an actual cloud (this is the nested virtualization argument) | 20:30 |
| zaro | waynr: sorry to hear that, no hurry on my questions. | 20:30 |
| *** Swami has joined #openstack-infra | 20:31 | |
| armax | fungi: if that was possible, we’d still need to tweak the logic to make adapt it to the environment that neutron expects | 20:31 |
| Swami | armax:hi | 20:31 |
| fungi | we have virtual machines we need to run our tests on, and need to connect them with a virtual network. on top of those virtual machines and virtual networks we want to deploy a cloud (which may use nova net or neutron alternatively) and those things may themselves deploy a virtual network of their own riding on top of the network we have created for this emulated physical environment | 20:31 |
| armax | and I’d rather have the logic completely separated in neutron_dvr_gre_bridge than hijack gre_bridge instead | 20:32 |
| Swami | armax: I agree with your comment | 20:32 |
| clarkb | Swami: armax also flat network as set up by the gre thing is just a name | 20:32 |
| armax | it sounds like gre_bridge was devise to make it work with n-net | 20:32 |
| clarkb | Swami: armax its just a set of bridged interfaces | 20:33 |
| fungi | it was devised so that we could define a common subnet between the virtual machines they could use to communicate with one another | 20:33 |
| armax | I suppose we could try and change gre_bridge to fit both scenarios | 20:33 |
| clarkb | why can't neutron use that? | 20:33 |
| clarkb | say neutron your interface is called flat_if_foo | 20:33 |
| fungi | remember that this is running inside a network environment we don't control, so we already need a level of encapsulation just to be able to have these machines reach one another on a network we define | 20:34 |
| Swami | clarkb: neutron always create br-ex, and it should be connected that would be the neutron's requirement | 20:34 |
| clarkb | Swami: ok then just connect that to flat_if | 20:34 |
| clarkb | Swami: isn't that sufficient? | 20:34 |
| clarkb | Swami: armax all we are doing here is setting up two l2 networks between test nodes | 20:34 |
| clarkb | Swami: armax forget gre/ovs/etc and forget flat/public/br-ex | 20:34 |
| clarkb | neutron should be deployable on two nodes that have two interfaces each on a different shared l2 | 20:35 |
| *** _nadya_ has quit IRC | 20:35 | |
| Swami | clarkb: yes | 20:35 |
| armax | clarkb: agreed | 20:35 |
| clarkb | and if not, there should be some other topology that neutron is deployable on that nova network is also deployable on | 20:35 |
| clarkb | and we will set that up instead | 20:35 |
| clarkb | ok we agree on that, now why is the current setup that configures that network topology insufficient for neutron? | 20:35 |
| armax | ok, so you’d be okay changing the existing gre_bridge function to see if we can accommodate neutron to work as weel as nova | 20:36 |
| armax | ? | 20:36 |
| clarkb | armax: yes absolutely, that is what I asked for before | 20:36 |
| armax | well | 20:36 |
| clarkb | armax: however I do not think it should be necessary | 20:36 |
| clarkb | and am asking for more info on why it might be | 20:36 |
| Swami | clarkb: the issue I had is, automatically the flat_if was not connected to the br-ex | 20:36 |
| Swami | That change has to occur. | 20:36 |
| clarkb | Swami: yes so that should be a devstack config option right? you tell devstack to configure neutron on flat_if | 20:37 |
| clarkb | then devstack bridges the two | 20:37 |
| clarkb | this should be no different than telling neutron to use eth0 | 20:37 |
| Swami | clarkB: The other thing that i don't understand is the public_if. | 20:37 |
| clarkb | Swami: thats just another interface on a different l2 network | 20:37 |
| Swami | Does the public_if correspond to the "br-ex" in neutron. | 20:38 |
| clarkb | Swami: you don't have to use both | 20:38 |
| fungi | those were arbitrary names we picked, they can be renamed if it makes people happier | 20:38 |
| clarkb | yes we should rename then to eth99 and eth42 | 20:38 |
| Swami | So let us assume when you say two l2 networks | 20:38 |
| fungi | they got scribbled on a notepad in the summit lounge and those were the names which sprang to mind | 20:38 |
| Swami | network1 is the internal network that provides the private ip for the VMs. Does it correspond to the flat_if in the nova case. | 20:39 |
| clarkb | Swami: yes | 20:39 |
| Swami | Ok, got it. | 20:39 |
| Swami | Network2 is the external network that provides the gateway and the floatingip to the VMs. Does it correspond to the public_if | 20:39 |
| clarkb | Swami: yes | 20:39 |
| clarkb | and if we need to rename them to make that more clear thats fine | 20:40 |
| clarkb | what I want to stress is that we should configure one overlay network topology and use it for both neutron and nova net | 20:40 |
| fungi | in fact i would say that if the names really smack that much of a nova-net specific deployment design, then we probably should rename them to something more generic so as to avoid future confusion of this sort | 20:40 |
| clarkb | the names and tools matter less to me than achieving that goal | 20:41 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Make tempest-dsvm-heat job voting https://review.openstack.org/170604 | 20:41 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Remove integrated-gate template from heat https://review.openstack.org/170605 | 20:41 |
| Swami | So then we can provide the devstack the details about the "PUBLIC_INTERFACE" as the public_if and there should be a corresponding one to provide the physical interface for internal network. | 20:41 |
| clarkb | however we also know that this topology and toolset works and seems to work quite well | 20:41 |
| clarkb | so I would be careful if changing it | 20:41 |
| Swami | So do you want to stick on to the same names that are there. | 20:41 |
| Swami | In your opinion then no change is required and it should work as such. | 20:42 |
| Swami | Am I right. | 20:42 |
| clarkb | Swami: we can change the names to avoid confusion, but in my opinion the topology itself is fine | 20:42 |
| *** hdd has joined #openstack-infra | 20:42 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/system-config: Install puppet from the Ubuntu apt repository for aarch64/arm64 https://review.openstack.org/170607 | 20:42 |
| Swami | ok then let's use the same topology. | 20:43 |
| *** medieval1 has joined #openstack-infra | 20:43 | |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Remove heat from default configuration https://review.openstack.org/170608 | 20:45 |
| Swami | So I will give it one more try at it to map the existing "flat" to "ethO" and "pub_if" to "br-ex". | 20:45 |
| armax | Swami: besides the naming conventions, we’d need to figure out what extra setup is required and whether that is simply done within devstack itself | 20:45 |
| Swami | armax: Yes I will confirm it. | 20:45 |
| openstackgerrit | Joe Gordon proposed openstack-infra/devstack-gate: Remove heat from default configuration https://review.openstack.org/170608 | 20:46 |
| Swami | If I hit any issues with that I will let you guys know if there any changes required. | 20:46 |
| clarkb | Swami: thanks, also it helps me to explain things in networking terms and not neutron terms | 20:46 |
| Swami | clarkb: understood | 20:46 |
| clarkb | br-ex is mostly meaningless to me but if you say "neutron needs a bridge with an interface on a shared layer2" I can grok that | 20:46 |
| fungi | same. i spent years as a network engineer for some very, very large networks. i've never deployed neutron. i probably should give it a try at some point ;) | 20:47 |
| armax | clarkb: yeah..it’s a matter of also using a naming convention that makes sense for both solutions | 20:47 |
| Swami | Yes neutron by default creates the ovs-bridge "br-ex" for external connectivity. | 20:47 |
| Swami | Somehow we need to connect that ovs-bridge to the interface that "gre-bridge" is creating. | 20:47 |
| clarkb | Swami: I think devstack does that, since that is what it does when you tell it to use eth0 | 20:48 |
| clarkb | you just need to tell it to use something other than eth0, in any case I think we have worked through some of the confusion here and hve a plan. look forward to seeing what you find out | 20:48 |
| armax | clarkb: that’s true, that’s why I was suggesting earlier that there’s might be logic in devstack itself that we can piggyback on | 20:48 |
| Swami | clarkb: Yes devstack does it for neutron when you ask it to create. | 20:48 |
| *** packet has quit IRC | 20:48 | |
| Swami | clarkb: yes I will go re-verify with the current plan and if I have an issue I will ping you. | 20:49 |
| *** akshai has quit IRC | 20:49 | |
| armax | so we might need to edit localrc to pass the intf names that make sense in this type of topology | 20:49 |
| Swami | armax: yes I will try it out. | 20:49 |
| armax | rather than the default ones that are used in single host | 20:49 |
| Swami | armax: sounds like a plan | 20:51 |
| *** rbradfor has joined #openstack-infra | 20:51 | |
| *** harlowja has quit IRC | 20:53 | |
| asselin_ | clarkb, read through your comments. first, glad you agree in the general direction. | 20:54 |
| *** jpeeler has quit IRC | 20:55 | |
| *** david-lyle has quit IRC | 20:55 | |
| asselin_ | clarkb, nibalizer took a quick look and wasn't sure his thoughts exactly. | 20:55 |
| *** MarkAtwood has quit IRC | 20:56 | |
| asselin_ | clarkb, so given that i can iterate a bit more to make it resuable across other puppet modules | 20:56 |
| *** akshai has joined #openstack-infra | 20:57 | |
| *** tiswanso has quit IRC | 20:57 | |
| fungi | going over cacti graphs for nodepool.o.o since the pymysql switch, this is the only obvious performance change http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=987&rra_id=all | 20:57 |
| *** melwitt_ has joined #openstack-infra | 20:58 | |
| *** melwitt has quit IRC | 20:58 | |
| *** teran has joined #openstack-infra | 20:58 | |
| *** sdake has quit IRC | 20:58 | |
| openstackgerrit | Clark Laughlin proposed openstack-infra/system-config: Install puppet from the Ubuntu apt repository for aarch64/arm64 https://review.openstack.org/170607 | 20:58 |
| fungi | it looks like mysql:// was probably using a unix socket file and mysql+pymysql:// is using tcp over the loopback interface | 20:58 |
| *** melwitt_ has quit IRC | 20:58 | |
| clarkb | fungi: interesting, it is pushing more bits too. Ah yup that would probably explain it | 20:59 |
| *** melwitt has joined #openstack-infra | 20:59 | |
| clarkb | asselin_: ya I think iterating is fine, if we get the puppet exit code fix into your script as it is now I am fine merging it | 20:59 |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: web: Allow clients (browsers, proxies) to cache status.json https://review.openstack.org/170081 | 20:59 |
| clarkb | asselin_: then we can followup with changes to make it reconsumable by other modules | 20:59 |
| fungi | part of me wants to see if i can figure out why that is, but a bigger part of me says that's just a few mbps over the loopback interface i don't really care | 20:59 |
| *** sarob has joined #openstack-infra | 21:00 | |
| *** andreykurilin_ has joined #openstack-infra | 21:00 | |
| clarkb | ya its going to be just about as free as using the unix socket | 21:00 |
| fungi | ~3mbps | 21:01 |
| fungi | sustained | 21:01 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Add tempest-dsvm-sahara https://review.openstack.org/167779 | 21:01 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Add sahara tempest job to sahara and tempest https://review.openstack.org/167780 | 21:01 |
| *** patrickeast_ has joined #openstack-infra | 21:02 | |
| fungi | so if we reimplement the loopback interface over a 10-baset hub this would be like network 30% utilization! | 21:02 |
| *** patrickeast has quit IRC | 21:02 | |
| *** patrickeast_ is now known as patrickeast | 21:02 | |
| *** teran has quit IRC | 21:02 | |
| *** melwitt has quit IRC | 21:03 | |
| *** melwitt has joined #openstack-infra | 21:03 | |
| *** miqui_ has quit IRC | 21:03 | |
| *** bswartz has quit IRC | 21:04 | |
| clarkb | to address vxlan and why we use gre, I think we were looking for a least common denominator that would work across different cloud implementations. Correct me if I am wrong but vxlan requires multicast which we do not have working in our cloud envs | 21:05 |
| *** sarob has quit IRC | 21:05 | |
| clarkb | fungi: ^ you likely know | 21:05 |
| *** miqui has joined #openstack-infra | 21:05 | |
| *** annegentle has joined #openstack-infra | 21:05 | |
| *** hodos has quit IRC | 21:06 | |
| *** otter768 has joined #openstack-infra | 21:06 | |
| *** hodos has joined #openstack-infra | 21:07 | |
| fungi | i honestly have only used vxlan to communicate between hosts and switches (we needed way, way, way more than 1k vlan ids), but i don't recall what the frame looks like | 21:07 |
| *** ajmiller has quit IRC | 21:07 | |
| fungi | er, well we had vxlan going between the switches themselves too, but yea no recollection. i'll look | 21:08 |
| *** achanda has quit IRC | 21:08 | |
| *** andreykurilin_ has quit IRC | 21:08 | |
| clarkb | ya my understanding is it is required for network broadcasts across logical l2 | 21:09 |
| clarkb | which you need for dhcp so we can't have it not work | 21:09 |
| *** andreykurilin_ has joined #openstack-infra | 21:09 | |
| fungi | i know i've definitely only used it when i had control of the hardware it was traversing | 21:09 |
| fungi | or over vpcs where i controlled all the endpoints | 21:09 |
| *** wicope has quit IRC | 21:11 | |
| *** otter768 has quit IRC | 21:11 | |
| fungi | clarkb: yep, broadcast communication maps to multicast ip on the outer header https://tools.ietf.org/html/rfc7348#section-4.2 | 21:14 |
| *** sarob has joined #openstack-infra | 21:15 | |
| *** ajmiller has joined #openstack-infra | 21:17 | |
| fungi | so we wouldn't, e.g., be able to arp over a vxlan broadcast domain without multicast ip support in our service providers | 21:17 |
| *** tonytan4ever has quit IRC | 21:17 | |
| *** ZZelle_ has joined #openstack-infra | 21:17 | |
| clarkb | or dhcp | 21:18 |
| clarkb | so no vxlan for us | 21:18 |
| asselin_ | clarkb, sounds good, thanks | 21:18 |
| *** JayJ_ has quit IRC | 21:18 | |
| fungi | clarkb: yep, well, with no arp dhcp is going nowhere anyway (literally) | 21:21 |
| *** sarob has quit IRC | 21:24 | |
| *** samueldmq_ has joined #openstack-infra | 21:24 | |
| fungi | er, yeah i guess the initial broadcast request and subsequent reply don't rely on arp, and the later refreshes which did would in most cases fall back to discovery again | 21:25 |
| clarkb | ya | 21:26 |
| fungi | so i guess dhcp would in theory work in a network where arp was broken, but regardless, not where ethernet broadcast is broken | 21:26 |
| *** dkliban is now known as dkliban_afk | 21:29 | |
| *** sabeen1 has quit IRC | 21:29 | |
| *** _nadya_ has joined #openstack-infra | 21:33 | |
| *** pc_m has quit IRC | 21:33 | |
| fungi | greghaynes: ironically, your coverage report for 165682 http://logs.openstack.org/82/165682/8/check/nodepool-coverage/08c6340/cover/ comes in at 1% worse than the most recently merged change http://logs.openstack.org/89/170289/1/check/nodepool-coverage/8cff29a/cover/ | 21:35 |
| jeblair | clarkb, fungi: back | 21:35 |
| *** hashar has quit IRC | 21:35 | |
| clarkb | jeblair I +2'd but didnt approve those other zuul changes | 21:35 |
| jeblair | clarkb: yeah, i'll go ahead and aprv them | 21:36 |
| fungi | greghaynes: i'm okay with 165682 in theory, but since its result differs in practice from the goal stated in the commit message, i want to make sure that's making sense to you first | 21:36 |
| *** _nadya_ has quit IRC | 21:38 | |
| jeblair | clarkb: oh, 160050 fixes the same thing that 170319 does, but in a different way | 21:39 |
| openstackgerrit | Merged openstack-infra/zuul: Delay Gerrit events by 5s https://review.openstack.org/170300 | 21:40 |
| *** yamahata has quit IRC | 21:40 | |
| clarkb | jeblair is that the same race? | 21:40 |
| openstackgerrit | Merged openstack-infra/zuul: Fix checking all builds are waiting in tests https://review.openstack.org/170319 | 21:40 |
| *** yamahata has joined #openstack-infra | 21:40 | |
| clarkb | Yours addresses the issue in production so better fix | 21:41 |
| jeblair | clarkb: yeah... actually looking at it, i don't think i want to apply 160050 because then i think if it ever were called, it would blindly submit the job "stop:None" which will never be picked up... | 21:41 |
| jeblair | clarkb: we shouldn't actually call a method that would hit the attributeerror anymore with 170319. i think it would be okay to clean up the code so there's no possible path in the future... | 21:42 |
| jeblair | clarkb: but we should probably protect against the use of "None" explicitly in that case | 21:42 |
| clarkb | ok I can abandon my change | 21:42 |
| jeblair | clarkb: (essentially, the attributeerror protected us from submitting a nonsense gearman job) | 21:42 |
| jeblair | clarkb: either way; i put a comment on your change about adding the None check for posterity | 21:43 |
| *** ryanpetrello_ has joined #openstack-infra | 21:44 | |
| *** mriedem has quit IRC | 21:44 | |
| clarkb | oh I can update it to checj that instead. | 21:46 |
| *** sputnik13 has quit IRC | 21:46 | |
| *** vhosakot has joined #openstack-infra | 21:50 | |
| openstackgerrit | Steve Varnau proposed openstack-infra/zuul: Fix timer jobs getting aborted from abandoning a change https://review.openstack.org/170614 | 21:50 |
| *** mjordan has quit IRC | 21:50 | |
| clarkb | jeblair: do you think we hvae mitigated this race with your change such that my change is unnecessary? or should I update it for extra belts and suspenders? | 21:53 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Create a log server like http://logs.openstack.org https://review.openstack.org/167425 | 21:53 |
| *** Swami has quit IRC | 21:53 | |
| *** sarob has joined #openstack-infra | 21:55 | |
| jeblair | clarkb: i think it's currently unecessary, but your change will make the code better so that future changes are less likely to mess something up. so good belts and suspenders. | 21:55 |
| jeblair | and i think we have pants this time too. which is good. | 21:55 |
| *** gulic has quit IRC | 21:55 | |
| *** gulic has joined #openstack-infra | 21:56 | |
| *** samueldmq_ has quit IRC | 21:57 | |
| clarkb | I do recommend remembering pants before going outside | 21:58 |
| openstackgerrit | Merged openstack-infra/zuul: Fix race in CRD duplicates test https://review.openstack.org/160491 | 21:59 |
| *** teran has joined #openstack-infra | 21:59 | |
| *** sarob has quit IRC | 21:59 | |
| *** unicell has joined #openstack-infra | 22:01 | |
| openstackgerrit | Steve Varnau proposed openstack-infra/zuul: Fix timer jobs getting aborted from abandoning a change https://review.openstack.org/170614 | 22:01 |
| mordred | jeblair: we got pants? | 22:02 |
| *** teran has quit IRC | 22:03 | |
| *** sarob has joined #openstack-infra | 22:04 | |
| *** amitgandhinz has quit IRC | 22:05 | |
| *** doude has quit IRC | 22:06 | |
| *** achanda has joined #openstack-infra | 22:08 | |
| *** sarob has quit IRC | 22:09 | |
| *** Krinkle is now known as Krinkle|detached | 22:10 | |
| *** sarob has joined #openstack-infra | 22:11 | |
| *** trey has joined #openstack-infra | 22:12 | |
| *** david-lyle has joined #openstack-infra | 22:12 | |
| clarkb | jeblair: fwiw making that change makes tests unhappy so its possible this is tickling something new :) | 22:12 |
| clarkb | I am currently trying to sort that out now | 22:13 |
| *** david-lyle_ has joined #openstack-infra | 22:13 | |
| *** Bsony has quit IRC | 22:13 | |
| jeblair | yeesh! | 22:14 |
| *** achanda has quit IRC | 22:14 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Simple script to setup and test log server https://review.openstack.org/169117 | 22:15 |
| *** sarob has quit IRC | 22:15 | |
| clarkb | jeblair: I have updated the check in setBuildDescription and it explodes on an attribute error | 22:17 |
| clarkb | jeblair: but the FakeBuild in tests/base.pp has a manager attribute and we should set __gearman_manager to None at the very least | 22:17 |
| asselin_ | clarkb, changes updated. found a bug in the common log server too using the new test script! | 22:17 |
| clarkb | so unsure how we are getting to setBuildDescription before the launch happens | 22:18 |
| asselin_ | clarkb, have a good weekend | 22:18 |
| clarkb | asselin_: you too, will look when I am done fighting zuul :) | 22:18 |
| *** nelsnelson has quit IRC | 22:18 | |
| *** Swami has joined #openstack-infra | 22:18 | |
| *** yamahata has quit IRC | 22:21 | |
| *** hdd has quit IRC | 22:21 | |
| *** vhosakot has quit IRC | 22:26 | |
| *** achanda has joined #openstack-infra | 22:26 | |
| *** sarob has joined #openstack-infra | 22:26 | |
| *** smccully has quit IRC | 22:28 | |
| *** annegentle has quit IRC | 22:28 | |
| *** annegentle has joined #openstack-infra | 22:28 | |
| clarkb | jeblair: appears to be related to merge jobs that fail | 22:29 |
| clarkb | oh its for noop jobs | 22:31 |
| *** thedodd has quit IRC | 22:32 | |
| *** melwitt has quit IRC | 22:33 | |
| *** melwitt has joined #openstack-infra | 22:33 | |
| *** rbradfor has quit IRC | 22:36 | |
| *** harlowja has joined #openstack-infra | 22:40 | |
| *** akshai has quit IRC | 22:40 | |
| clarkb | oh and dunderscores are biting us | 22:40 |
| clarkb | jeblair: ^ was there a reason for making it __gearman_manager? | 22:40 |
| *** baoli has quit IRC | 22:42 | |
| clarkb | hrm same error if I remove the mangling __s | 22:42 |
| *** andreykurilin_ has quit IRC | 22:42 | |
| *** annegentle has quit IRC | 22:43 | |
| jeblair | clarkb: __ because it's modifying an object that it doesn't own.. the Job class from the gear lib | 22:43 |
| clarkb | in this case we are adding a _Gearman__gearman_manager attribute to Build objects | 22:47 |
| *** amotoki has joined #openstack-infra | 22:48 | |
| clarkb | I'm not sure how thats tied to the Job class from gear | 22:49 |
| clarkb | anyways I think the issue is that we have some Build objects that are fully launched and others that are not. Changing to use build.gearman_manager and setting that in __init__ of build works | 22:51 |
| openstackgerrit | Stephanie Miller proposed openstack-infra/puppet-zanata: Add apache configuration for zanata https://review.openstack.org/164011 | 22:52 |
| *** amotoki has quit IRC | 22:52 | |
| *** Krinkle|detached is now known as Krinkle | 22:52 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Use gate64 cpu type during multinode testing https://review.openstack.org/168408 | 22:53 |
| clarkb | flashgordon: ^ that is ready now | 22:53 |
| *** achanda has quit IRC | 22:53 | |
| timrc | Soooo... I need to do some work with lining up tasks and what not and add data binding to select different providers/regions/thingies but.... http://15.125.65.125/ | 22:55 |
| *** gaelL has quit IRC | 22:55 | |
| timrc | There's actually a lot of styling that needs to be done... but a start, I suppose. | 22:56 |
| *** gaelL has joined #openstack-infra | 22:56 | |
| *** sputnik13 has joined #openstack-infra | 22:56 | |
| *** sarob has quit IRC | 22:57 | |
| zigo | Do I need special ACLs to be able to push a tag to a stackforge project? | 22:57 |
| zigo | I'd like to tag 1.0.2.2 in xstatic-angular-lrdragndrop. | 22:57 |
| clarkb | zigo: you need to have the push signed tag permission | 22:58 |
| zigo | clarkb: Can you do it for me then? :) | 22:58 |
| zigo | Or who should I ask? | 22:59 |
| clarkb | zigo: no, we shoud make sure the ACLs are correct | 22:59 |
| clarkb | let me take a look | 22:59 |
| *** teran has joined #openstack-infra | 22:59 | |
| clarkb | zigo: https://review.openstack.org/#/admin/groups/386,members that is the group that can push tags. Either get a member of that group to push the tag or have them add you to the group | 23:00 |
| zigo | clarkb: I don't think I'll ever touch that package again, it's just that I happened to commit what was uploaded to PyPi and which never went into the project. | 23:00 |
| clarkb | (once you are added ou can push the tag yourself0 | 23:00 |
| clarkb | timrc: I am not sure what I am looking at :) | 23:02 |
| clarkb | timrc: the values seem to be in increments of 100 and have a unit of 'm' | 23:02 |
| fungi | clarkb: clearly this is metric | 23:04 |
| *** teran has quit IRC | 23:04 | |
| *** bhunter71 has quit IRC | 23:04 | |
| *** otter768 has joined #openstack-infra | 23:07 | |
| *** otter768 has quit IRC | 23:12 | |
| mordred | I think "m" is "meters" | 23:13 |
| mordred | which is neat | 23:13 |
| mordred | I didn't know nodepool had such good distance | 23:13 |
| *** sarob has joined #openstack-infra | 23:13 | |
| clarkb | jeblair: so not quite sure how to address this in zuul, and probably won't get it done before I call it a day | 23:14 |
| *** samueldmq_ has joined #openstack-infra | 23:14 | |
| clarkb | jeblair: but likely keeping the attribute error checks is ismplest | 23:14 |
| *** nibalizer has quit IRC | 23:15 | |
| *** signed8bit has quit IRC | 23:17 | |
| timrc | mordred, Haha. | 23:17 |
| timrc | I'm trying to figure out what I'm missing / is going on with that. | 23:17 |
| *** dimtruck is now known as zz_dimtruck | 23:30 | |
| *** zz_dimtruck is now known as dimtruck | 23:31 | |
| *** Viswanath has joined #openstack-infra | 23:31 | |
| *** dimtruck is now known as zz_dimtruck | 23:32 | |
| *** achanda has joined #openstack-infra | 23:32 | |
| *** hodos has quit IRC | 23:33 | |
| *** melwitt has quit IRC | 23:34 | |
| *** tkelsey has joined #openstack-infra | 23:38 | |
| Krinkle | Hm.. I'm trying to run Zuul tests locally, but not getting anywhere. | 23:39 |
| *** ZZelle_ has quit IRC | 23:40 | |
| Krinkle | I was hoping it'd be as simple as 'tox' or 'tox -epy27' | 23:40 |
| jeblair | Krinkle: "tox -e py27" should be all that's needed | 23:40 |
| jeblair | or that. space is not important | 23:40 |
| Krinkle | (copied from your jenkins output :) ) | 23:40 |
| * Krinkle is not a python developer | 23:40 | |
| Krinkle | https://gist.github.com/Krinkle/22dc0996ae91ef8febf4/raw | 23:40 |
| *** Viswanath has quit IRC | 23:40 | |
| Krinkle | I've got latest python 2.79 from Homebrew | 23:41 |
| timrc | mordred, Is this data fraction of an hour? http://graphite.openstack.org//render?format=raw&target=alias%28stats.nodepool.task.hpcloud-b1.CreateServerTask%2C%27%27%29 | 23:41 |
| jeblair | Krinkle: so the good news is it ran 51 tests and only 16 failed! | 23:41 |
| jeblair | Krinkle: oh, i have no idea if they work on OSX | 23:41 |
| Krinkle | rrrright | 23:42 |
| clarkb | jeblair: they won't due to epoll iirc | 23:42 |
| clarkb | in gear | 23:42 |
| timrc | Man that URL gives me a headache. | 23:42 |
| jeblair | yeah, that'll probably do it | 23:42 |
| Krinkle | the .poll fs thing doesn't work on OSX's system. | 23:42 |
| Krinkle | Yeah, I ran into this with gearman | 23:42 |
| Krinkle | I guess it uses that too | 23:42 |
| *** ajmiller has quit IRC | 23:42 | |
| * Krinkle boots vagrant | 23:42 | |
| *** tkelsey has quit IRC | 23:43 | |
| *** annegentle has joined #openstack-infra | 23:43 | |
| openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: web: Allow clients (browsers, proxies) to cache status.json https://review.openstack.org/170081 | 23:44 |
| jeblair | okay, i'm going to restart zuul now | 23:47 |
| mordred | timrc: its seconds | 23:48 |
| timrc | mordred, Hm. That's what I thought, but that's not what that url is giving me AFAICT. | 23:48 |
| mordred | timrc: try "averageSeries(stats.timers.nodepool.task.hpcloud-b*.CreateServerTask.mean)" | 23:49 |
| *** annegentle has quit IRC | 23:49 | |
| timrc | Ah I need to be using timers? | 23:49 |
| mordred | yeah - the other thing is, I think, giving you counts of numbers of calls | 23:50 |
| Krinkle | jeblair: Should I install gearman manually before running the test? | 23:52 |
| jeblair | (and actually, when you do want counts, i find stats_counts.... to usually be what i want to work with, as it's the raw numbers) | 23:52 |
| Krinkle | 2015-04-03 23:50:02,990 gear.Connection.unknown DEBUG Connecting to 127.0.0.1 port 55319 | 23:52 |
| Krinkle | 2015-04-03 23:50:02,990 gear.Connection.unknown DEBUG Error connecting to 127.0.0.1 port 55319 | 23:52 |
| Krinkle | 2015-04-03 23:50:02,990 gear.Client.unknown DEBUG Unable to connect to <gear.Connection 0x7fc7e0983790 host: 127.0.0.1 port: 55319> | 23:52 |
| jeblair | Krinkle: no, tox will do that | 23:52 |
| *** sarob has quit IRC | 23:52 | |
| mordred | jeblair: ++ | 23:52 |
| Krinkle | OK | 23:52 |
| *** Swami has quit IRC | 23:52 | |
| Krinkle | Woah, all tests pass. | 23:53 |
| jeblair | neato! | 23:53 |
| jeblair | Krinkle: fyi, there may occasionally be race conditions in them. we try to squash them as we find them (we just fixed two). but still may happen from time to time. | 23:54 |
| Krinkle | Yeah, I noticed in my stack of a dozen changes a few randomly failed | 23:54 |
| Krinkle | jeblair: btw, I recall a conference talk about openstack elasticsearch and something about smart 'recheck' neediness based on common errors or something like that. | 23:54 |
| Krinkle | Did that work out in the end? | 23:55 |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| jeblair | Krinkle: for openstack itself we have this: http://status.openstack.org/elastic-recheck/ | 23:55 |
| jeblair | Krinkle: and there's an associated bot which will leave a message saying "this failed because of a known bug" | 23:55 |
| *** bswartz has joined #openstack-infra | 23:55 | |
| jeblair | Krinkle: so it helps people decide whether to just recheck or perhaps they are introducing something | 23:55 |
| Krinkle | jeblair: specifically a bug that would likely not happen again if doing recheck, as opposed to a deterministic known issue? | 23:56 |
| jeblair | Krinkle: we don't use it for anything other than openstack itself though (so it won't apply to zuul changes) | 23:56 |
| Krinkle | ah, right. | 23:56 |
| Krinkle | It's not generic yet. | 23:56 |
| Krinkle | :D | 23:56 |
| Krinkle | still mighty impressive though | 23:56 |
| *** samueldmq_ has quit IRC | 23:56 | |
| jeblair | well... it's pretty generic... | 23:56 |
| jeblair | Krinkle: the only things that aren't generic are what we decide to put into logstash | 23:56 |
| jeblair | Krinkle: (so the logstash grok filters are written for openstack logs) | 23:57 |
| *** dannywilson has quit IRC | 23:57 | |
| jeblair | Krinkle: and the specific logstash queries that drive that, which are written by hand | 23:57 |
| jeblair | Krinkle: so the _system_ is generic; our use of it is just openstack because those two parts are maintained by hand | 23:57 |
| *** carl_baldwin has quit IRC | 23:58 | |
| clarkb | Krinkle: https://review.openstack.org/#/c/169285/ has an example comment from the system | 23:58 |
| Krinkle | The thing I'm personally most happy about is the visualisation and integration. Gathering the data and making it say 'recheck' is one thing. But making it easy to configure and maintain is a separate skillset. | 23:58 |
| clarkb | it identified the test failure as being https://bugs.launchpad.net/nova/+bug/1249065 | 23:58 |
| openstack | Launchpad bug 1249065 in OpenStack Compute (nova) havana "Nova throws 400 when attempting to add floating ip (No nw_info cache associated with instance)" [Medium,Confirmed] - Assigned to Aaron Rosen (arosen) | 23:58 |
| Krinkle | Ah, I see. It suggests recheck, doesn't say it itself. | 23:59 |
| clarkb | correct | 23:59 |
| jeblair | Krinkle: yeah, to avoid infinite loops and also masking test failures from developers | 23:59 |
| Krinkle | yeah, if there's no pain, bugs don't get fixed. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!