| anteaya | jhesketh: oh thanks, well that is new syntax for me, I've never seen that before | 00:00 |
|---|---|---|
| jeblair | jhesketh: i think so. the only merging function we'll really need to do, aside from launching, is the config file retrieving. i don't think i built that into the ansible-merger-launcher, which is why i didn't remove standalone mergers yet. | 00:00 |
| anteaya | I'm sure otehr projects will want to use that too | 00:00 |
| *** rbrndt_ has quit IRC | 00:01 | |
| jhesketh | jeblair: correct, I was looking at that and that was going to be my next question... should we drop the separate merger daemon and have the server clone the repos it needs to pull config from itself | 00:01 |
| jhesketh | perhaps in threads | 00:01 |
| jhesketh | granted ,that doesn't scale as well now we have the cat function | 00:01 |
| *** slagle has joined #openstack-infra | 00:02 | |
| *** sarob has quit IRC | 00:03 | |
| *** asettle has joined #openstack-infra | 00:03 | |
| *** salv-orl_ has quit IRC | 00:04 | |
| *** sarob has joined #openstack-infra | 00:04 | |
| *** bpokorny has quit IRC | 00:05 | |
| *** annegentle has joined #openstack-infra | 00:05 | |
| xarses | fungi: FYI, I'm still getting NOT_REGISTERED https://review.openstack.org/#/c/288820 | 00:06 |
| EmilienM | fungi: btw, here is the puppet-octavia thing we discussed last week -> https://review.openstack.org/289024 | 00:06 |
| *** thorst has joined #openstack-infra | 00:06 | |
| *** rfolco has joined #openstack-infra | 00:06 | |
| EmilienM | xarses: there is some maintainance now on jenkins nodes | 00:07 |
| xarses | EmilienM: thanks | 00:07 |
| *** piet has quit IRC | 00:08 | |
| *** Qiming has quit IRC | 00:10 | |
| *** Sukhdev_ has quit IRC | 00:10 | |
| fungi | xarses: oh, this is for devstack-centos7 workers, not devstack-trusty-2-node | 00:11 |
| fungi | xarses: let me see why we're not registering any of those | 00:11 |
| *** sdake has joined #openstack-infra | 00:12 | |
| fungi | also only building in bluebox | 00:12 |
| fungi | it must be "first" in the round-robin distribution | 00:12 |
| *** angdraug has quit IRC | 00:13 | |
| fungi | lovely. it looks like we're timing out testing ssh to these | 00:14 |
| *** r1chardj0n3s has joined #openstack-infra | 00:14 | |
| *** rfolco has quit IRC | 00:14 | |
| fungi | Exception: Timeout waiting for ssh access | 00:15 |
| *** jaypipes has quit IRC | 00:16 | |
| *** recheckbot is now known as bgmccollum | 00:17 | |
| fungi | we're actually not successfully using any nodes in bluebox | 00:17 |
| anteaya | :( | 00:18 |
| fungi | 10 in a building state, 1 in a delete state | 00:18 |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: WIP: Do not silently drop markers that fail to evaluate https://review.openstack.org/289628 | 00:18 |
| *** r1chardj0n3s has left #openstack-infra | 00:18 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: [WIP] Add image size report https://review.openstack.org/289629 | 00:19 |
| pleia2 | pabelanger: ^5 https://www.openstack.org/summit/austin-2016/summit-schedule/events/7337 | 00:19 |
| ianw | fungi: just to be clear, devstack-centos7 not getting network just on bluebox? | 00:20 |
| EmilienM | congrats pleia2 ! | 00:20 |
| pleia2 | EmilienM: thanks | 00:20 |
| fungi | ianw: i think evidence points to no nodes of any kind getting network on bluebox right now | 00:20 |
| ianw | fungi: ok, sorry, spidey-sense tingling with some of the changes we've been making :) if i can help ... i don't have bluebox account though | 00:21 |
| fungi | ianw: but since we restarted zuul recently any jobs for workers which have not successfully build yet fail immediately as not_registered, so there's no zuul demand for them, which means nodepool only tries to build the min-ready for them, and if it's only 1 that means it only tries in the "first" cloud which is apparently bluebox these days | 00:22 |
| *** bpokorny has joined #openstack-infra | 00:22 | |
| *** kevinsho_ has quit IRC | 00:22 | |
| fungi | so with bluebox seemingly broken we'll never register jobs for workers which have a min-ready of 1 | 00:22 |
| ianw | oh. so SNAFU | 00:22 |
| openstackgerrit | James Polley proposed openstack-dev/pbr: Clarifications around tags and version numbers https://review.openstack.org/289630 | 00:23 |
| ianw | i would never have even considered that failure path | 00:23 |
| fungi | ianw: we are the definition of "subtle interaction" | 00:23 |
| EmilienM | ianw: hey! can you consider reviewing https://review.openstack.org/#/c/288244/ when you have some time today? fungi already +2'ed | 00:24 |
| fungi | jlk: any chance you could help us figure out why we can't ssh to any nova instances we're booting in bluebox-sjc1 now? | 00:24 |
| *** bnemec has quit IRC | 00:24 | |
| jlk | fungi: do you have access to our support system? | 00:25 |
| ianw | EmilienM: ok, i can only +1 in system-config ... i guess it seems ok now i understand it a bit more; i can't think of anything better, anyway. | 00:25 |
| jlk | fungi: I'm not at my desk, so it'll be a bit difficult. | 00:25 |
| *** kevinsho_ has joined #openstack-infra | 00:25 | |
| fungi | jlk: i didn't even know you had a support system, but i'll go look | 00:25 |
| *** mriedem_away is now known as mriedem | 00:25 | |
| fungi | in the meantime i'm going to propose a patch to disable nodepool on bluebox | 00:25 |
| jlk | ah. Yeah, we do, even a web chat thing. But. | 00:25 |
| jlk | what's the symptoms ? | 00:25 |
| jlk | are the instances getting a floating IP attached, or are they stuck waiting again like a few days ago? | 00:26 |
| *** annegentle has quit IRC | 00:26 | |
| fungi | jlk: nova says the instance is ready, but it's never reachable. this is across a variety of images for different linux distros too | 00:26 |
| fungi | jlk: i'm still gathering details | 00:26 |
| fungi | just sort of fell into this problem in the middle of dealing with other unrelated issues, so still trying to sort it out myself | 00:27 |
| *** annegentle has joined #openstack-infra | 00:27 | |
| fungi | jlk: unreachable in this case means not even icmp ping gets a response | 00:28 |
| *** sdake has quit IRC | 00:29 | |
| jlk | right, but that's through floating IP | 00:29 |
| jlk | there are a number of things that can go wrong for that | 00:29 |
| *** Swami has quit IRC | 00:29 | |
| fungi | yep. tcp connection to port 22 also times out (no banner) | 00:29 |
| *** asettle has quit IRC | 00:29 | |
| jlk | what we're looking for is whether the instance fully came up (check console-log) or wether it got suck waiting for cloud-init or something. | 00:30 |
| jlk | nova can consider it "booted" if the OS comes up, but the OS may be hung trying to reach the metadata service for cloud-iniit to do its thing | 00:30 |
| fungi | will do once i remind myself how to pull a nova console log | 00:30 |
| jlk | a floating IP address may be allocated, and put onto the router, but if the instance OS doesn't bring up its private IP network then nothing will get through. | 00:30 |
| *** asettle has joined #openstack-infra | 00:31 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily disable booting in bluebox https://review.openstack.org/289632 | 00:32 |
| fungi | ianw: jeblair: ^ | 00:32 |
| fungi | jlk: thanks, i should have checked the nova log earlier | 00:34 |
| fungi | mordred: what are the chances we've recently done strange things in glean to make it incompatible with bluebox? | 00:35 |
| jlk | alright I'm stepping away for a moment. I won't likely see the pings for a bit. | 00:35 |
| fungi | jlk: no problem, thanks for your help! | 00:35 |
| fungi | looks like no new glean commits or tags for at least a couple weeks | 00:36 |
| fungi | mordred: the last line of the nova console-log for a sample host had "Waiting up to 60 more seconds for network configuration..." as the last line before nodepool gave up on it becoming reachable and deleted it | 00:37 |
| fungi | are we using configdrive for network info there, or metadata server, or something else? | 00:38 |
| fungi | dhcp? | 00:38 |
| *** annegentle has quit IRC | 00:38 | |
| *** sarob has quit IRC | 00:38 | |
| fungi | aha, this looks like it's actually an error message output by the dhcp client when not getting an ip address assigned | 00:39 |
| anteaya | fungi: https://review.openstack.org/#/c/289632/ couldn't get a devstack-centos7 node | 00:40 |
| *** sarob has joined #openstack-infra | 00:40 | |
| fungi | anteaya: yep, we don't have any | 00:40 |
| anteaya | :( | 00:40 |
| anteaya | tonight is not our night | 00:40 |
| *** sarob has quit IRC | 00:40 | |
| fungi | anteaya: i'm going to manually submit that change | 00:40 |
| anteaya | fungi: yup | 00:40 |
| *** asettle has quit IRC | 00:40 | |
| anteaya | you have support for it | 00:41 |
| *** sarob has joined #openstack-infra | 00:41 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily disable booting in bluebox https://review.openstack.org/289632 | 00:41 |
| *** asselin_ has quit IRC | 00:42 | |
| *** hdd has joined #openstack-infra | 00:42 | |
| jlk | fungi: hrm, that might be an "us" problem | 00:42 |
| jlk | fungi: that's what monty saw last week, that magically went away after an hour or so | 00:42 |
| *** nikhil has quit IRC | 00:42 | |
| fungi | while i wait for puppet to run with that nodepool config update, i'll get jehkins01 and 02 configs updated | 00:43 |
| fungi | jlk: thanks, i must have missed that incident, so good to know maybe we have a second data point now | 00:43 |
| *** mriedem has quit IRC | 00:43 | |
| *** xarses has quit IRC | 00:46 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 00:46 |
| *** armax has quit IRC | 00:47 | |
| fungi | zaro: zxiiro: electrofelix: is there anything special i need to do to perform offline configuration updates with jjb? if i just ignore the jenkins.BadHTTPException is that enough, or should i call something other than jenkins-jobs update to do it? | 00:49 |
| *** Jeffrey4l has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:52 | |
| *** asselin has joined #openstack-infra | 00:53 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Add experimental job to test xstatic packaging https://review.openstack.org/289125 | 00:53 |
| zxiiro | fungi: can you clarify what you mean by offline? | 00:53 |
| zxiiro | I thought as long as you can reach the server it should just work | 00:54 |
| jlk | fungi: my guys dug through the logs, and found a number of times where the network 8699d309-3767-42c1-92cc-d6dae91500bd became exhausted of IPs, but that would usually result in a different kind of error | 00:54 |
| fungi | zxiiro: updating the xml configs of jobs while i have the jenkins service stopped, so that it doesn't have to hit the api for each one and will let jenkins merely load them from disk at its next start | 00:55 |
| *** Qiming has joined #openstack-infra | 00:55 | |
| *** gildub has quit IRC | 00:55 | |
| *** nikhil has joined #openstack-infra | 00:55 | |
| *** HeOS has joined #openstack-infra | 00:55 | |
| zxiiro | fungi ah... I wasn't aware that was possible but would love to see that documented if it works. | 00:56 |
| fungi | jlk: yes, we've been fighting a floating ip leak of some kind there, and periodically deleting all disassociated floating ips. i deleted them all again a little while ago, confirmed there were 0 disassociated floating ips reported by nova, and new nodes booting there were getting addresses assigned but the dhcp clients on them seemed to be getting no response to their requests | 00:56 |
| *** jamesmcarthur has joined #openstack-infra | 00:56 | |
| jlk | nod | 00:56 |
| *** asselin has quit IRC | 00:58 | |
| fungi | so nova had a floating ip address reported for the instance, but dhcp on the instance was getting nothing | 00:58 |
| jlk | yup, I'm familiar with that case | 00:58 |
| jlk | could be something stupid like network connectivity lost between the HV and the controller where dhcp is running, or it could be more serious like the agent itself having a bad time | 00:58 |
| Sam-I-Am | fungi: do you know why this isn't pointing to v6? | 01:00 |
| Sam-I-Am | fungi: http://docs.openstack.org/releasenotes/heat/unreleased.html | 01:00 |
| fungi | jlk: the fip leak is something else i'd love to dig into sometime soon though, since i've been spending some time manually deleting them. i need to rule out unusual corner cases in our code failing to attempt to delete fips for instances which never booted or something, but in the past we've seen nova (at least elsewhere) accept the delete but then never actually act on it | 01:00 |
| *** ajmiller has quit IRC | 01:00 | |
| *** HeOS has quit IRC | 01:00 | |
| jlk | are you having nova try to do the delete, or neutron itself? | 01:00 |
| fungi | jlk: that's a great question. i have to check the library which is handling that | 01:01 |
| *** armax has joined #openstack-infra | 01:01 | |
| *** jamesmcarthur has quit IRC | 01:01 | |
| jlk | I don't really trust Nova to do anything right proxying to other services :) | 01:01 |
| fungi | Sam-I-Am: AJaeger said something about docs publishing jobs being broken for a while and just fixed recently, but i don't know the exact details | 01:01 |
| Sam-I-Am | fungi: awesome. i recall a problem with devref and tags causing it to sometimes publish an old ver, but i didnt know it impacted relnotes too | 01:01 |
| fungi | Sam-I-Am: i don't know whether it did, but that's my first suspicion anyway | 01:02 |
| Sam-I-Am | i'll ask AJaeger | 01:02 |
| fungi | jlk: yeah, there have definitely been bugs in the past where nova did not-so-great things if asked to proxy some action to neutron, then the neutron request it made timed out | 01:03 |
| fungi | like, leave instances in permanently undeletable error states that you had to edit the database to clean up | 01:04 |
| fungi | i think that one was port deletes | 01:05 |
| *** binbincong has quit IRC | 01:05 | |
| jlk | oh good times | 01:05 |
| anteaya | Sam-I-Am: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-07.log.html#t2016-03-07T20:31:13 | 01:05 |
| anteaya | that may help | 01:05 |
| *** binbincong has joined #openstack-infra | 01:05 | |
| *** flip214 has quit IRC | 01:05 | |
| *** flip214 has joined #openstack-infra | 01:05 | |
| *** flip214 has quit IRC | 01:05 | |
| *** flip214 has joined #openstack-infra | 01:05 | |
| *** kevinsho_ has quit IRC | 01:06 | |
| Sam-I-Am | anteaya: ahh thx | 01:07 |
| *** ZZelle_ has quit IRC | 01:08 | |
| anteaya | welcome | 01:08 |
| *** gildub has joined #openstack-infra | 01:09 | |
| *** hdd has quit IRC | 01:10 | |
| *** gyee has quit IRC | 01:11 | |
| *** keedya has joined #openstack-infra | 01:13 | |
| jeblair | fungi: mordred may want to press on with shade before we dig too deeply into that | 01:15 |
| *** sdague has joined #openstack-infra | 01:15 | |
| fungi | jeblair: yeah, i recall him saying he had something to solve something there | 01:16 |
| fungi | something something something | 01:16 |
| jeblair | yeah, possibly even something about leaking something | 01:16 |
| clarkb | so the issue aiui today is that when we say "fip attach to node foo" and that either "fails" or timesout | 01:17 |
| clarkb | then we have no remaining record of that fip for where it should be and as a result don't have a good way to delete it | 01:17 |
| clarkb | but something about how shade does the whole attach an fip makes this better | 01:17 |
| mordred | jeblair: aroo? | 01:19 |
| *** gildub_ has joined #openstack-infra | 01:19 | |
| *** gildub has quit IRC | 01:19 | |
| mordred | yes | 01:19 |
| fungi | mordred: though separately, looks like we're in another situation like you saw last week with dhcp clients in bluebox getting no replies from the dhcp server | 01:20 |
| mordred | the way shade does the ip is it says "create server ; server_port = get_port_from_server ; add_floating_ip_to(server_port) | 01:20 |
| mordred | fungi: awesome! | 01:20 |
| mordred | so if add_floating_ip_to(server_port) fails - there is no floating ip | 01:20 |
| mordred | and if it succeeds, it's attached to the server | 01:20 |
| fungi | i've temporarily set max-servers there to 0 so we can build some of our less common node types again and get jobs registered, since nodepool sees bluebox as the "first" cloud at the moment | 01:20 |
| mordred | fungi: ossum | 01:21 |
| mordred | fungi: did we ever land my extra logging patch to nodepool? | 01:21 |
| fungi | mordred: i believe both those landed | 01:21 |
| mordred | k | 01:21 |
| mordred | I think I was also going to write a patch to try to capture thelast N lines from nova console-log in the case of a wait-for-ssh timeout | 01:21 |
| mordred | also - have we considered/tried increasing the wait-for-ssh timeout for bluebox? | 01:22 |
| mordred | it's possible that the dhcp service is just behind? | 01:22 |
| *** stevebaker has joined #openstack-infra | 01:22 | |
| mordred | jlk: we're currently having nova proxy to neutron | 01:22 |
| mordred | jklare: when we finish the shade migration. we'll be calling neutron directly | 01:22 |
| mordred | gah | 01:22 |
| mordred | sorry jklare | 01:23 |
| jlk | So we bounced the neutron agents | 01:23 |
| jlk | any of you want to give a build a try and see if it'll come up? | 01:23 |
| *** harlowja has quit IRC | 01:23 | |
| *** _amrith_ is now known as amrith | 01:25 | |
| *** yamamoto_ has joined #openstack-infra | 01:25 | |
| fungi | mordred: well, it waits for something like 2 minutes | 01:25 |
| fungi | jlk: in a bit, once i'm sure we're working okay, i'll try upping the max-servers back to 1 and see if anything ever goes live there | 01:26 |
| clarkb | fungi: jeblair now that I am back to a computer anything I can do to help with the jjbs? | 01:27 |
| fungi | clarkb: well, i stopped 01 and 02, ran jjb update on them, then started them back up. spot checks suggest they have the updated job configs. i've got 3,4,5 in prepare for shutdown now | 01:28 |
| fungi | i didn't think to get a control test first to confirm that actually works, so i found a newly added job which jenkins05 doesn't know about and will confirm after an offline update it does appear when starting back up | 01:28 |
| clarkb | fungi: you stopped them or put them in shutdown mode? | 01:29 |
| clarkb | jjb talks to the api so it needs a running jenkins service I think | 01:29 |
| fungi | clarkb: stopped them completely. i thought jenkins would read updated xml from disk when starting? | 01:30 |
| fungi | was trying to avoid slowly hammering zuul with gearman registrations | 01:31 |
| clarkb | fungi: it does but jjb doesn't update jenkins off of disk, it does it through the api iirc | 01:31 |
| clarkb | looks like the swiftclient docs job still needs running so I will do that now | 01:31 |
| anteaya | so as a note as I was looking at https://www.openstack.org/summit/austin-2016/summit-schedule/#day=2016-04-29 | 01:31 |
| fungi | oh, so it doesn't update the config on disk and then tell jenkins via api to reread the file | 01:31 |
| clarkb | fungi: no I don't think so, we happen to run it on localhost but it cna run anyhwere | 01:31 |
| anteaya | if you click learn more about the summit if forces you to sign in using openid/foundation sign in | 01:31 |
| EmilienM | mordred, nibalizer: just fyi, you might like that one day https://github.com/ansible/ansible-modules-extras/pull/1807 | 01:32 |
| fungi | anteaya: that seems odd that information would be made to require an account | 01:32 |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Do not silently drop markers that fail to evaluate https://review.openstack.org/289628 | 01:32 |
| anteaya | fungi: that's what I thought | 01:32 |
| *** sputnik13 has quit IRC | 01:33 | |
| anteaya | btw I didn't sign in | 01:33 |
| clarkb | anteaya: where is the link? | 01:33 |
| *** aspiers has joined #openstack-infra | 01:33 | |
| anteaya | top of the page in blue | 01:33 |
| anteaya | Learn more about the Austin Summit 2016 Tracks | 01:33 |
| anteaya | is the text | 01:33 |
| openstackgerrit | Craige McWhirter proposed openstack-infra/system-config: Initial commit of the Phabricator puppetry. https://review.openstack.org/289648 | 01:33 |
| anteaya | am I wrong? | 01:33 |
| clarkb | oh there | 01:34 |
| *** sputnik13 has joined #openstack-infra | 01:34 | |
| fungi | looks like it links to https://www.openstack.org/summit/austin-2016/call-for-speakers/track-list | 01:34 |
| clarkb | sorry I kept looking along at the headers | 01:34 |
| clarkb | this is between headers and content | 01:34 |
| anteaya | yeah sorry | 01:34 |
| anteaya | yes | 01:34 |
| anteaya | and odd place for a link | 01:34 |
| mordred | EmilienM: neat! | 01:34 |
| fungi | also, it takes me to an empty page with a nonexistent account logged in, and has a button for "log in as someone else" | 01:34 |
| anteaya | fungi: i'm forced to sign in to view the link you posted | 01:34 |
| mordred | EmilienM: btw - have you seen our "use-hiera-with-ansible-with-puppet-apply" stuff? | 01:34 |
| clarkb | fungi: yup so probably just a bad link dest | 01:34 |
| mordred | EmilienM: I've been trying to figure out how much of ut wants to be upstreamed and how much is specific to us | 01:35 |
| clarkb | mordred: nibalizer any progress on getting failed puppet runs to post to puppetboard? | 01:35 |
| EmilienM | mordred: I like it, I also implemented something similar with https://github.com/EmilienM/ansible-marionette | 01:35 |
| clarkb | notmyname: http://docs.openstack.org/developer/python-swiftclient/ that looks better now | 01:36 |
| mordred | EmilienM: nice | 01:36 |
| anteaya | http://imgur.com/qJLokny | 01:36 |
| anteaya | that is what I see | 01:36 |
| clarkb | anteaya: yes beacuse the target url is actually a thing you probably need auth for | 01:36 |
| EmilienM | mordred: my stuff is PoC, really. Not production like you. But I've read a lot of your code and *please* upstream all you can because I see a lot of value | 01:36 |
| clarkb | anteaya: so its a broken link that isn't 100% broken in that it leads somewhere | 01:36 |
| anteaya | that excludes me | 01:36 |
| anteaya | awesome | 01:37 |
| mordred | EmilienM: cool. I will | 01:37 |
| mordred | I'll also review that module | 01:37 |
| clarkb | anteaya: the target likely just needs updating, I am sure that a bug would get that fixed soon | 01:37 |
| anteaya | do you want me to file? | 01:37 |
| clarkb | sure | 01:37 |
| anteaya | I can file | 01:37 |
| anteaya | okay | 01:37 |
| fungi | anteaya: yeah, even once you log in, you just get an empty page, so it's obviously not what they meant to link there | 01:38 |
| *** annegentle has joined #openstack-infra | 01:39 | |
| fungi | they probably meant to link to a page with real content, and which doesn't require authentication to view | 01:39 |
| fungi | it gives me "That page is secured. Enter your credentials below and we will send you right along." | 01:39 |
| anteaya | found this one, good for a laugh: https://bugs.launchpad.net/openstack-org/+bug/1315240 | 01:39 |
| openstack | Launchpad bug 1315240 in openstack-org "http://www.openstack.org/community/ congratulates jenkins frequently" [Low,Confirmed] | 01:39 |
| anteaya | fungi: wonderful | 01:40 |
| mordred | wow. I got 3 talks approved for the summit - I did not konw I had submitted two of them | 01:40 |
| fungi | mordred: are they all at the same time but in separate rooms? | 01:40 |
| mordred | fungi: I hope so | 01:41 |
| clarkb | mordred: nibalizer anything I cna help with to get puppet runs reporting to puppetboard on fials? | 01:42 |
| *** asselin has joined #openstack-infra | 01:42 | |
| mordred | clarkb: I have not thought about that problem at all yet. nibalizer, you? | 01:42 |
| mordred | SamYaple: sorry I dropped off when we were talking earlier ... | 01:43 |
| mordred | SamYaple: turns out - debootstrap is at the base of how I'm suggesting you use dib too ... so it's probably not too much of a shift for you | 01:44 |
| anteaya | https://bugs.launchpad.net/openstack-org/+bug/1554297 | 01:44 |
| openstack | Launchpad bug 1554297 in openstack-org "Austin Conf schdule link to summit requires authentication" [Undecided,New] | 01:44 |
| *** annegentle has quit IRC | 01:44 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 01:44 |
| *** sputnik13 has quit IRC | 01:44 | |
| anteaya | mordred: interesting skill you have aquired submitting talks without being aware of it | 01:44 |
| mordred | fungi: if you happen to speak to any of the lovely people who worked on the summit site thing | 01:45 |
| openstackgerrit | Clark Boylan proposed openstack/diskimage-builder: Zerofree the image if possible https://review.openstack.org/289054 | 01:45 |
| clarkb | greghaynes: ianw ^ that should be good and happy now | 01:45 |
| mordred | fungi: my current feedback is that if I go to my speaker page: https://www.openstack.org/summit/austin-2016/summit-schedule/speakers/1604 ... it shows me times of each talk (yay!) but does not show me DAYS | 01:45 |
| anteaya | ha ha ha | 01:45 |
| anteaya | everyday | 01:45 |
| mordred | anteaya: you speak words that do not sound pleasant :) | 01:46 |
| mordred | fungi: however, if I click to even tdetails, it shows me days | 01:46 |
| mordred | so I'm guessing that the information is there | 01:46 |
| anteaya | I see the secret to having a talk accepting | 01:47 |
| anteaya | name monty as co-presenter | 01:47 |
| anteaya | the first two talks have quite the crowd onstage | 01:47 |
| * clarkb happens to speak to those individuals and will let them know now | 01:47 | |
| *** MarkAtwood has quit IRC | 01:47 | |
| *** mtanino has quit IRC | 01:47 | |
| mordred | clarkb: woot! | 01:47 |
| mordred | clarkb: in general, my experience is mostly positive with the site | 01:48 |
| *** baoli has joined #openstack-infra | 01:48 | |
| mordred | but printing the day would be helpful... also, thank them for printing the day on the event details page with the Day of the Week | 01:48 |
| mordred | because I have recently looked at a different conference site which just listed days as dates, and those are usually meaningless to me when looking at athing :) | 01:48 |
| *** annegentle has joined #openstack-infra | 01:49 | |
| *** kaisers1 has joined #openstack-infra | 01:55 | |
| anteaya | so if I understand this bug correctly: https://bugs.launchpad.net/openstack-org/+bug/1315240 if git provided an atom feed the tom could get what he needs for the openstack site | 01:55 |
| openstack | Launchpad bug 1315240 in openstack-org "http://www.openstack.org/community/ congratulates jenkins frequently" [Low,Confirmed] | 01:55 |
| anteaya | and it appears cgit has the ability to provide atom feeds of comments | 01:56 |
| *** kaisers has quit IRC | 01:56 | |
| *** HeOS has joined #openstack-infra | 01:57 | |
| *** gildub_ has quit IRC | 01:58 | |
| *** thorst has quit IRC | 01:58 | |
| *** gildub_ has joined #openstack-infra | 01:58 | |
| *** thorst has joined #openstack-infra | 01:58 | |
| anteaya | pleia2: you still about? | 02:00 |
| *** jamesmcarthur has joined #openstack-infra | 02:01 | |
| *** HeOS has quit IRC | 02:01 | |
| *** annegentle has quit IRC | 02:01 | |
| *** pvaneck has quit IRC | 02:02 | |
| *** jamesmcarthur has quit IRC | 02:06 | |
| *** jamesmcarthur has joined #openstack-infra | 02:07 | |
| *** thorst has quit IRC | 02:07 | |
| clarkb | fungi: for 03-05 can we go ahead and start updating them one at a time? | 02:08 |
| *** baoli has quit IRC | 02:08 | |
| fungi | clarkb: i've got fresh updates going on 01 and 02 since you indicated my earlier methodology was flawed | 02:08 |
| *** baoli has joined #openstack-infra | 02:08 | |
| clarkb | fungi: ya I just double checked on that and jjb uses python-jenkins which does an http request | 02:10 |
| mordred | fungi, clarkb did we restart nodepool with new shade yet? | 02:10 |
| fungi | yep, just thought it updated the on-disk copies of the xml as well and didn't solely rely on the jenkins master being reachable, but maybe it only did that a long, long time ago (or perhaps never) | 02:10 |
| clarkb | mordred: no, and image builds are done so now would be a good time | 02:10 |
| mordred | clarkb: awesome | 02:10 |
| clarkb | mordred: are you doing that? | 02:11 |
| mordred | clarkb: I can | 02:11 |
| clarkb | mordred: why don't you go for it | 02:11 |
| anteaya | woooooo: http://git.openstack.org/cgit/openstack/cinder/atom | 02:11 |
| clarkb | I am mostly around if it goes south | 02:11 |
| *** nwkarsten has quit IRC | 02:12 | |
| fungi | i'm around as well | 02:12 |
| anteaya | the feed isn't including author names which is what he wanted | 02:12 |
| mordred | clarkb, fungi: restarted | 02:12 |
| *** yamahata has joined #openstack-infra | 02:13 | |
| clarkb | mordred: you may want to ask it to upload an image now to see that it owrks | 02:13 |
| mordred | I do not immediatley see tracebacks in the log | 02:13 |
| mordred | so that's good | 02:13 |
| clarkb | otherwise its wait until tomorrow morning | 02:13 |
| *** baoli has quit IRC | 02:14 | |
| *** Sukhdev has joined #openstack-infra | 02:15 | |
| *** Sukhdev has quit IRC | 02:15 | |
| *** kevinsho_ has joined #openstack-infra | 02:16 | |
| *** baoli has joined #openstack-infra | 02:17 | |
| mordred | clarkb: should I just do a nodepool image-update? | 02:17 |
| *** annegentle has joined #openstack-infra | 02:17 | |
| SamYaple | mordred: alrighty. ill give dib a go when i have time | 02:17 |
| clarkb | mordred: no, do a sudo -H -u nodepool nodepool image-upload $provider $image | 02:17 |
| SamYaple | any starter docs? | 02:18 |
| clarkb | mordred: since this is related to swifty things I would do image-upload rax-iad ubuntu-trusty or similar | 02:18 |
| mordred | clarkb: kk | 02:18 |
| clarkb | SamYaple: dib itself comes with fairly comprehensive docs | 02:18 |
| clarkb | SamYaple: http://docs.openstack.org/developer/diskimage-builder/ | 02:18 |
| mordred | clarkb: I have done that | 02:18 |
| SamYaple | thanks for the links | 02:18 |
| clarkb | mordred: cool, watching this is like watching paint dry | 02:19 |
| clarkb | mordred: so uh ya, hang out, grab a drink and enjoy the paint | 02:19 |
| mordred | clarkb: I love paint | 02:19 |
| clarkb | mordred: I think rax uploads are ~30-45 minute seach | 02:19 |
| clarkb | this is one reason I have been looking at the size of our images again | 02:20 |
| mordred | clarkb: oh joy | 02:20 |
| fungi | i recommend not sniffing the paint too much though | 02:20 |
| lifeless | fungi: whats this about pbr wedging? | 02:20 |
| clarkb | our git caches should be slightly smaller now that the change to not cache stackforge or attic merged | 02:20 |
| *** xarses has joined #openstack-infra | 02:21 | |
| clarkb | mordred: right now 6.8GB is git cache | 02:21 |
| clarkb | mordred: then you have the base OS install for raw/vhd is not cheap | 02:21 |
| fungi | lifeless: we continuously deploy zuul with a pip install . process and ended up with a pbr afflicted with bug 1375048 which prevented us updating zuul without working around it by first diong a pip install -U pbr | 02:22 |
| openstack | bug 1375048 in PBR liberty "setup.py --help-commands fails in env without testrepository" [Undecided,Fix released] https://launchpad.net/bugs/1375048 - Assigned to Akihiro Motoki (amotoki) | 02:22 |
| *** bpokorny_ has joined #openstack-infra | 02:22 | |
| fungi | lifeless: after a manual pbr upgrade, we were able to upgrade zuul from source again | 02:22 |
| *** asettle has joined #openstack-infra | 02:23 | |
| *** baoli has quit IRC | 02:23 | |
| *** baoli has joined #openstack-infra | 02:24 | |
| *** bpokorny has quit IRC | 02:24 | |
| *** edmondsw has quit IRC | 02:25 | |
| ianw | clarkb: so my before & after (old is with zeroing) -> http://paste.openstack.org/show/489620/ | 02:25 |
| fungi | lifeless: looking at the zuul commit history and comparing to the last version successfully installed in the environment, i think https://review.openstack.org/262579 is what did us in | 02:25 |
| ianw | clarkb: that's fedora-minimal f23 | 02:25 |
| clarkb | ianw: oh neat you did observe a difference | 02:26 |
| ianw | clarkb: yeah, not gb but doesn't hurt | 02:26 |
| clarkb | ya I am still not sure why occasionally our qcow2's balloon | 02:27 |
| kzaitsev_mb | we're having wierd issues with py34 job for murano-dashboard | 02:27 |
| clarkb | but local testing of the ballooned image shows that zeroing makes it better | 02:27 |
| fungi | lifeless: the commit immediately before that one merged was the last version to upgrade successfully in the environment | 02:27 |
| kzaitsev_mb | here's the link http://logs.openstack.org/95/289195/4/check/gate-murano-dashboard-python34/0002747/console.html | 02:27 |
| *** sdake has joined #openstack-infra | 02:27 | |
| kzaitsev_mb | it says smth about 'plugin html-output = htmloutput.htmloutput:HtmlOutput: No module named 'version'' | 02:28 |
| SamYaple | clarkb: you can also do discard/trim ops to shrink the image | 02:28 |
| SamYaple | clarkb: i typically do that to finialize any qcow2 i have | 02:28 |
| ianw | clarkb: a screenful of hhttps://review.openstack.org/289629 -> http://paste.openstack.org/show/489621/ | 02:28 |
| clarkb | SamYaple: want to add that as a follow up to https://review.openstack.org/#/c/289054/3 once you dig into dib? | 02:29 |
| fungi | kzaitsev_mb: /home/jenkins/workspace/gate-murano-dashboard-python34/.tox/py34/lib/python3.4/site-packages/nose/plugins/manager.py:395: RuntimeWarning: Unable to load plugin html-output = htmloutput.htmloutput:HtmlOutput: No module named 'version' | 02:29 |
| SamYaple | clarkb: roger | 02:29 |
| *** pahuang has joined #openstack-infra | 02:29 | |
| kzaitsev_mb | fungi: yep, but that doesn't make any sense to me =/ | 02:29 |
| ianw | /opt/dib/tmp/image.DR4IZg25/built/var/cache/dnf immediately stands out ... must be a logic fail in when we're cleaning that | 02:29 |
| clarkb | I don't know that the nose html output plugin is py34 compat | 02:30 |
| clarkb | and I think we juts kill that thing | 02:30 |
| kzaitsev_mb | fungi: we have nosehtmloutput>=0.0.3 under test-requirements | 02:30 |
| clarkb | ianw: isn't that where we cache the packages for devstack and bindep? | 02:30 |
| clarkb | kzaitsev_mb: ya I don't know that it works in py34 | 02:30 |
| ianw | clarkb: ahhh, yes ... we just started doing that, so yeah | 02:30 |
| *** nikhil_ has joined #openstack-infra | 02:30 | |
| clarkb | ianw: the apt cache is similarly large on ubuntu images and intentionally so | 02:31 |
| *** nikhil_ is now known as Guest36618 | 02:31 | |
| fungi | kzaitsev_mb: wow, last release of that was almost 3 years ago | 02:31 |
| ianw | yeah, i forgot we just turned that back on, dnf didn't have --downloadonly in previous revisions | 02:31 |
| clarkb | fungi: kzaitsev_mb yes don't remind me about that code :) | 02:31 |
| fungi | kzaitsev_mb: clarkb: that's well before our push to py3k all the things | 02:31 |
| *** Guest36618 is now known as nikhil_k | 02:31 | |
| clarkb | fungi: yup as the "maintainer" if you can call me that I really don't expect it to work with py3k | 02:31 |
| kzaitsev_mb | funny thing is that it broke only recently | 02:31 |
| mordred | clarkb: crappit - I got a fail on the image - but I need to AFK for a sec | 02:32 |
| kzaitsev_mb | we have like 3 different things, that broke for dashboard on m-3 tagging day =/ | 02:32 |
| fungi | kzaitsev_mb: clarkb: is it possible this is related to the switch from bare-trusty to ubuntu-trusty? | 02:32 |
| clarkb | OpenStackCloudException: Image creation failed: __exit__ | 02:32 |
| mordred | clarkb: 2016-03-08 02:22:17,621 INFO gear.Client.nodepool: Received packet from <gear.Connection 0x7f0837d3c750 host: zuul.openstack.org port: 4730>: <gear.Packet 0x7f0837dac4d0 type: WORK_EXCEPTION handle: H:104.130.155.213:10233> | 02:32 |
| mordred | 2016-03-08 02:22:17,621 ERROR jobs.ImageUploadJob: Image 357291 failed to upload | 02:32 |
| mordred | clarkb: oh, I should look at the debug log shouldn't I? | 02:32 |
| clarkb | mordred: if you look in /var/log/nodepool/builer-debug.log you get more | 02:32 |
| clarkb | mordred: ya | 02:32 |
| mordred | clarkb: k. I'll look at that when I'm not afk anymore | 02:32 |
| clarkb | fungi: maybe? | 02:33 |
| fungi | kzaitsev_mb: clarkb: just wondering whether we were previously installing some package or setting some envvar special courtesy of rackspace on bare-trusty that was making it work | 02:33 |
| *** baoli has quit IRC | 02:33 | |
| *** tphummel has quit IRC | 02:34 | |
| kzaitsev_mb | fungi: tox -e py34 locally just works for me. I wonder if removing it would unblock the py34 job though | 02:35 |
| clarkb | fungi: maybe? | 02:36 |
| clarkb | heres the deal though, that code is dead dead dead | 02:36 |
| clarkb | and the idea of debugging it makes me sad | 02:36 |
| clarkb | I am guessing it is used because django | 02:37 |
| clarkb | which amkes me extra sadder that django unittesting stuff can't use an arbitrary test runner | 02:37 |
| kzaitsev_mb | I don't really think anyone was really using it in murano tbh | 02:38 |
| kzaitsev_mb | I might be wrong though | 02:38 |
| fungi | the exception seems to be getting raised by `python setup.py egg_info` under the murano-dashboard checkout with python 3.4 | 02:38 |
| fungi | but yeah, i'm not inclined to try and figure out why | 02:39 |
| fungi | an amusingly large cargo cult of dashboard repos seems to have inherited that requirement from horizon according to http://codesearch.openstack.org/?q=nosehtmloutput&i=nope&files=&repos= | 02:41 |
| clarkb | yup iirc horizon/django unittests must use nose | 02:41 |
| kzaitsev_mb | shouldn't it be https://pypi.python.org/pypi/nose-htmloutput seems more recent | 02:42 |
| clarkb | or at the time they required it | 02:42 |
| kzaitsev_mb | and log says it's installing nosehtmloutput==0.0.5 | 02:42 |
| clarkb | oh nice did someone take it over thats awesome | 02:42 |
| kzaitsev_mb | which is probably not py34 compliant | 02:42 |
| clarkb | there was at least one person that reached out to me aobut it and I said go for it and never heard back I guess the didn't want to preserve the pypi name | 02:43 |
| clarkb | thats unfortunate but nice to see someone else do the work | 02:43 |
| fungi | clarkb: if someone took it over, they abandoned it too. 0.0.5 was uploaded to pypi almost 3 years ago | 02:43 |
| openstackgerrit | Takashi Kajinami proposed openstack-infra/project-config: [WIP] Fix storlets gatejob problem https://review.openstack.org/289666 | 02:44 |
| clarkb | fungi: they are using a different name | 02:44 |
| fungi | also we never imported this thing into gerrit i guess | 02:44 |
| fungi | or if we did, we used a different name for it too | 02:44 |
| clarkb | ours is older | 02:44 |
| clarkb | kzaitsev_mb: so ya I would suggest trying to use that upstream instead (change the name so it matches) | 02:44 |
| fungi | yep, looks like they took it over at the beginning of 2014 | 02:45 |
| fungi | and have a release as recently as september of last year | 02:45 |
| clarkb | I no longer think this was someone that reached out | 02:45 |
| clarkb | but ya use that if it owrks | 02:45 |
| *** jamesmcarthur has quit IRC | 02:46 | |
| kzaitsev_mb | clarkb: don't think that would work, since requirement's job is voting | 02:46 |
| *** esikachev has joined #openstack-infra | 02:46 | |
| fungi | clarkb: git history looks like nose-htmloutput is rewritten from scratch | 02:46 |
| fungi | so may not be compatible with existing nosehtmloutput use? no idea how much that matters | 02:47 |
| clarkb | it might not be | 02:47 |
| clarkb | but I would add it to requirements and use it since they are active | 02:47 |
| clarkb | or someone else can take over our version if incomatible is hard | 02:47 |
| clarkb | I put my effort into not using nose anymore so that I could stop using nose :) | 02:47 |
| fungi | clarkb: yeah, browsing around https://github.com/ionelmc/nose-htmloutput i don't think it's derived from your project, though i guess it's possible they plagarized you | 02:48 |
| kzaitsev_mb | removing it didn't help either. but. I found the error. we have non-ascii symbols inside README.rst, that is marked as description-file | 02:48 |
| kzaitsev_mb | and pbr is failing under py34 because of that | 02:49 |
| kzaitsev_mb | I believe, that those M-dashes break it | 02:49 |
| fungi | kzaitsev_mb: that explains the other traceback which i thought was a secondary cascade effect but i guess was actually a reraise? | 02:49 |
| clarkb | its ok I copy pastad under the terms of a bsd license for a unitest to html outputter | 02:50 |
| kzaitsev_mb | I'll try amending those —'s and would tell you about the results | 02:50 |
| *** esikachev has quit IRC | 02:50 | |
| fungi | kzaitsev_mb: so i guess the real issue was "error in setup command: Error parsing /home/jenkins/workspace/gate-murano-dashboard-python34/setup.cfg: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 218: ordinal not in range(128)" | 02:51 |
| kzaitsev_mb | yep, looks like it | 02:51 |
| fungi | and the nosehtmloutput error may be secondary or nonfatal/benign | 02:52 |
| *** sbalukoff has quit IRC | 02:52 | |
| kzaitsev_mb | and still this is weird, cause we have py34 job voting for quite some time. and those —'s should've gotten there after it got voting. | 02:53 |
| *** nwkarsten has joined #openstack-infra | 02:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** jamielennox is now known as jamielennox|away | 02:56 | |
| *** sridhar_ram has quit IRC | 02:56 | |
| *** ajmiller has joined #openstack-infra | 02:56 | |
| *** woodster_ has quit IRC | 02:57 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add image size report https://review.openstack.org/289629 | 02:58 |
| *** nwkarsten has quit IRC | 02:58 | |
| *** nwkarsten has joined #openstack-infra | 02:59 | |
| fungi | kzaitsev_mb: yeah, i still wonder if rackspace was setting some strange locale/lang in the default environment on their custom images and this issue appeared coincident with switching to a minimal fresh image build from scratch | 02:59 |
| kzaitsev_mb | fungi: clarkb: thanks for helping debug the problem. =) removing those — did work | 02:59 |
| *** jamielennox|away is now known as jamielennox | 03:01 | |
| fungi | kzaitsev_mb: glad you finally identified the issue | 03:03 |
| *** nwkarsten has quit IRC | 03:03 | |
| *** mtreinish has quit IRC | 03:04 | |
| *** psanchez has joined #openstack-infra | 03:04 | |
| openstackgerrit | Takashi Kajinami proposed openstack-infra/project-config: Fix storlets gatejob problem https://review.openstack.org/289666 | 03:04 |
| *** sarob has quit IRC | 03:04 | |
| *** mtreinish has joined #openstack-infra | 03:04 | |
| *** jamielennox is now known as jamielennox|away | 03:05 | |
| *** thorst has joined #openstack-infra | 03:05 | |
| *** sdague has quit IRC | 03:05 | |
| *** stevebaker has quit IRC | 03:06 | |
| *** stevebaker has joined #openstack-infra | 03:06 | |
| *** sarob has joined #openstack-infra | 03:06 | |
| *** gnuoy_ has joined #openstack-infra | 03:07 | |
| *** bpokorny_ has quit IRC | 03:08 | |
| *** apoorvad has quit IRC | 03:09 | |
| *** takashi has joined #openstack-infra | 03:09 | |
| *** baoli has joined #openstack-infra | 03:10 | |
| fungi | huh, jenkins05 had a corrupt jjb cache. i had to blow it away to get the update to work | 03:11 |
| fungi | looked probably truncated | 03:12 |
| fungi | puppet may have timed out an update in the middle of flushing its cache i guess | 03:12 |
| *** sarob has quit IRC | 03:12 | |
| *** thorst has quit IRC | 03:12 | |
| *** gnuoy_ has quit IRC | 03:12 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add some tracking for image sizes https://review.openstack.org/289678 | 03:12 |
| clarkb | fungi: ya I think that is what happens | 03:13 |
| clarkb | fungi: we should probaby make jjb do a write and mv rather than a direct write | 03:13 |
| *** annegentle has quit IRC | 03:13 | |
| fungi | that would reduce the race to an impossibly small duration, yeah | 03:14 |
| fungi | since an in-place rename should be a single atomic fs call | 03:14 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add some tracking for image sizes https://review.openstack.org/289678 | 03:15 |
| fungi | as long as it's on the same filesystem (so probably create it in the same directory) | 03:15 |
| *** jamielennox|away is now known as jamielennox | 03:15 | |
| *** cfarquhar has quit IRC | 03:15 | |
| *** yuanying has quit IRC | 03:16 | |
| *** gildub_ has quit IRC | 03:16 | |
| *** gildub has joined #openstack-infra | 03:16 | |
| takashi | Hello. Can I ask some help about gate job testing here? | 03:17 |
| *** christian_ has joined #openstack-infra | 03:17 | |
| *** binbincong has quit IRC | 03:18 | |
| clarkb | takashi: sure | 03:18 |
| clarkb | fungi: yup otherwise its a copy | 03:19 |
| fungi | takashi: i can try to help, what's the question? depending on the details it may be something one of us in here can answer, or it may be appropriate for a different channel (for example qa or a project-specific channel) | 03:19 |
| takashi | clarkb, fungi: thx | 03:19 |
| takashi | clarkb, fungi: I noticed that test jobs of our projects (storlets) always fails from this morning(in JST), and trying to fix it | 03:20 |
| fungi | takashi: a link to one of the failing logs is probably the easiest place for us to start | 03:20 |
| *** cfarquhar has joined #openstack-infra | 03:21 | |
| *** cfarquhar has quit IRC | 03:21 | |
| *** cfarquhar has joined #openstack-infra | 03:21 | |
| *** baoli has quit IRC | 03:21 | |
| takashi | clarkb, fungi: ex. patch 282931 | 03:21 |
| *** treaki_ has quit IRC | 03:21 | |
| *** baoli has joined #openstack-infra | 03:21 | |
| takashi | I think the cause of that failure is that our install scripts fails to sudo, because revoke-sudo is already executed | 03:22 |
| clarkb | ansible-playbook is not installed | 03:23 |
| *** treaki_ has joined #openstack-infra | 03:23 | |
| takashi | clarkb: We use the following install script https://github.com/openstack/storlets/blob/master/tests/setup_functional_test.sh | 03:24 |
| fungi | though also it does look like it wants to sudo things after it has removed the sudoers extension | 03:24 |
| fungi | i wonder if the template this job was using got a revoke-sudo macro added during the switch to ubuntu-trusty workers | 03:25 |
| fungi | i'm checking our recent config history to confirm | 03:25 |
| clarkb | oh its not installed because it apt get installs it with sudo | 03:25 |
| takashi | clarkb: yes | 03:25 |
| takashi | I already committed a patch for this https://review.openstack.org/#/c/289666/ | 03:26 |
| takashi | but I'm not so familiar with project-config, so want to ask some comments about this | 03:26 |
| *** christian_ has quit IRC | 03:27 | |
| takashi | and I have one question. Is there any way to test that project-config? I mean I would like to run our test job using that new config. | 03:27 |
| fungi | it looks like this was unintentionally broken by https://review.openstack.org/288860 | 03:27 |
| fungi | that patch should not have added a revoke-sudo macro | 03:27 |
| fungi | i'll get a fix up for this real quick | 03:28 |
| clarkb | I can approve it | 03:28 |
| takashi | fungi: I just notice that DID add revoke-sudo macro. | 03:28 |
| fungi | there were a couple others in that change which added revoke-sudo when they shouldn't have | 03:29 |
| *** baoli has quit IRC | 03:30 | |
| fungi | looks like trove is also broken by this | 03:30 |
| takashi | fungi, clarkb: Should I wait for your working about this? | 03:31 |
| fungi | the dev-zuul-coverage job gained a revoke-sudo too, but i suspect that should have had one to begin with | 03:31 |
| fungi | takashi: yes, it'll be just a few minutes | 03:31 |
| takashi | fungi: thx! | 03:32 |
| *** annegentle has joined #openstack-infra | 03:35 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Remove revoke-sudo from storlets, trove functional https://review.openstack.org/289684 | 03:35 |
| fungi | takashi: clarkb: ^ there's the fix | 03:35 |
| *** binbincong has joined #openstack-infra | 03:36 | |
| takashi | fungi: thanks a lot! :-) | 03:36 |
| fungi | AJaeger: jeblair: pabelanger: yolanda: ^ that got overlooked when 288860 was reviewed | 03:36 |
| fungi | wondering if there are other recent ubuntu-trusty conversion patches that need rereviewing to make sure we didn't add revoke-sudo to jobs which previously lacked it | 03:37 |
| *** rguillebert has quit IRC | 03:37 | |
| *** yuanying has joined #openstack-infra | 03:38 | |
| *** kzaitsev_mb has quit IRC | 03:39 | |
| *** yuanying has quit IRC | 03:40 | |
| *** yuanying has joined #openstack-infra | 03:41 | |
| *** amrith is now known as _amrith_ | 03:42 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 03:43 |
| mordred | clarkb: this error is the best error ever | 03:44 |
| clarkb | mordred: yes "__exit__" | 03:45 |
| mordred | Shrews: when you wake up - I want you to glory in the amazingness of this traceback: http://paste.openstack.org/show/489622/ | 03:45 |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** madorn has joined #openstack-infra | 03:45 | |
| fungi | the underlying exception raised was "__exit__"? | 03:47 |
| clarkb | fungi: that was the error message | 03:47 |
| fungi | er | 03:47 |
| fungi | yeah | 03:47 |
| mordred | which does imply a context manager bailed out | 03:47 |
| mordred | or something | 03:47 |
| mordred | I think | 03:47 |
| fungi | is that like python saying, "just go away"? | 03:48 |
| fungi | i agree, it's amusing to have made it into the exception message | 03:48 |
| lifeless | Nakato: Build-Depends: cython, debhelper (>= 8.9.7~), gfortran (>= 4:4.2), libblas-dev [!arm !m68k], liblapack-dev [!arm !m68k], python-all-dbg, python-all-dev, python-nose, python-tz, python3-all-dbg, python3-all-dev, python3-nose, python3-tz | 03:48 |
| madorn | fungi: is www.openstack.org code kept in a public github repo? | 03:48 |
| ianw | mordred: that was very similar to why i was digging for so long on that token timeout thing ... the path to where the error message just doesn't show up in the traceback | 03:49 |
| madorn | like the foundation website | 03:49 |
| ianw | where the error message is created, i mean | 03:49 |
| lifeless | Nakato: Build-Depends: debhelper (>= 9), python-all-dev (>= 2.6.5-13~), python-all-dbg, python3-all-dev (>= 3.1.2-10~), python3-all-dbg, libyaml-dev (>= 0.1.2~), cython, cython-dbg, quilt | 03:49 |
| fungi | madorn: yes, but it's not maintained by the community. i understand it can be found at https://github.com/OpenStackweb/openstack-org | 03:50 |
| madorn | fungi: thanks for the help | 03:51 |
| *** hdd has joined #openstack-infra | 03:51 | |
| fungi | madorn: it's maintained by some web devs at the foundation, though much (most?) of the content is in a cms database not something easily patched via git | 03:51 |
| madorn | i see | 03:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: clear our rpm/dnf/yum data in chroot https://review.openstack.org/281596 | 03:54 |
| *** ccrouch has quit IRC | 03:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove revoke-sudo from storlets, trove functional https://review.openstack.org/289684 | 03:54 |
| fungi | clarkb: jenkins03-05 are updated now, 01-02 are almost done reupdating and then i'll do 06-07 | 03:56 |
| *** ccrouch has joined #openstack-infra | 03:56 | |
| clarkb | fungi: cool | 03:57 |
| fungi | with jenkins in prepare for shutdown, job updates/additions go quickly but deletions are still somewhat slow for some reason | 03:57 |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Use apt-cache generated packages to provide build deps https://review.openstack.org/289689 | 03:59 |
| *** treaki_ has quit IRC | 03:59 | |
| *** yuanying has quit IRC | 04:00 | |
| *** yuanying has joined #openstack-infra | 04:01 | |
| mordred | clarkb: so - I put in a logging debug statement and reran | 04:01 |
| mordred | clarkb: with self._swift_service_lock: | 04:01 |
| *** ccrouch has quit IRC | 04:02 | |
| *** treaki_ has joined #openstack-infra | 04:02 | |
| mordred | hahaha | 04:02 |
| mordred | I see it | 04:03 |
| *** jamesmcarthur has joined #openstack-infra | 04:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: In the service lock, reset the service, not the lock https://review.openstack.org/289692 | 04:04 |
| mordred | clarkb, fungi: enjoy | 04:04 |
| *** ccrouch has joined #openstack-infra | 04:05 | |
| *** yuanying has quit IRC | 04:05 | |
| mordred | Shrews: ^^ you'll also like that | 04:07 |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** salv-orlando has joined #openstack-infra | 04:08 | |
| ianw | :( | 04:09 |
| fungi | oh wow | 04:09 |
| *** thorst has joined #openstack-infra | 04:09 | |
| *** bdemers has quit IRC | 04:11 | |
| * mordred revels in everyone's collective pain | 04:12 | |
| *** bapalm has quit IRC | 04:13 | |
| fungi | that's a subtle bug indeed | 04:13 |
| fungi | cody-somerville: wow, so even though we gave them the okay to pack up and ship out the west hardware asap, they still didn't get around to it before the deadline and now it's wrapped up in the decommissioning project? | 04:15 |
| *** salv-orlando has quit IRC | 04:15 | |
| *** bdemers has joined #openstack-infra | 04:16 | |
| cody-somerville | fungi: Not sure why it didn't get shipped in time but the HW is not delayed due to the decommission. I'll work with HPE to see if we can get a ship date and/or expedite it. | 04:16 |
| cody-somerville | *now | 04:16 |
| clarkb | can we turn it back on? | 04:16 |
| clarkb | and use it until they get their act together? | 04:17 |
| fungi | unfortunate | 04:17 |
| *** thorst has quit IRC | 04:17 | |
| cody-somerville | clarkb: I'll ask if that's possible but I imagine if yes they could disappear at any time as they move through the DC performing the decommission. | 04:17 |
| *** ajmiller has quit IRC | 04:18 | |
| cody-somerville | It is unfortunate they didn't ship west instead of east. We all agreed to prioritize the SL390s in west if possible. | 04:18 |
| ianw | fungi / clarkb: pymysql.err.OperationalError: (1044, u"Access denied for user 'openstack_citest'@'%' to database 'bhbvuZWG_6881'") <- do we know about this issue for nodepool testing? | 04:18 |
| ianw | i'm guessing we switched node types | 04:18 |
| clarkb | fungi: ^ | 04:19 |
| fungi | ianw: that's a new one on me. i know fuel-web had a postgres issue, but... | 04:20 |
| fungi | first i've seen of mysql behavior changing with this | 04:20 |
| ianw | fungi: from https://review.openstack.org/#/c/289678/ | 04:21 |
| fungi | what is it trying to do to that database? | 04:21 |
| *** ccrouch has quit IRC | 04:22 | |
| *** ccrouch has joined #openstack-infra | 04:22 | |
| fungi | it's using the openstack_citest user to interact with a different database than the one it's granted access to (as opposed to using the root account or creating one)? | 04:23 |
| ianw | where's the output of mysql-prep stuff? | 04:25 |
| clarkb | thats the superuser account | 04:25 |
| *** sarob has joined #openstack-infra | 04:26 | |
| ianw | less jenkins/jobs/python-jobs.yaml | 04:26 |
| clarkb | fungi: openstack_citest is the acocunt with create adn delete perms on old bare-* | 04:26 |
| fungi | ianw: the output of that should be earlier in the log | 04:26 |
| fungi | it's all set -x so echoes to the console log | 04:27 |
| ianw | fungi: yeah ... http://logs.openstack.org/78/289678/2/check/gate-nodepool-python27/501da9d/console.html | 04:27 |
| fungi | clarkb: yeah, i believe we just flat out grant openstack_citest access to do everything | 04:27 |
| clarkb | ya that sounds right | 04:27 |
| fungi | though curious why not just use the root user in the db in that case | 04:27 |
| clarkb | fungi: because you may have access to a db without root | 04:28 |
| clarkb | they are separate concerns | 04:28 |
| fungi | ahh, makes sense | 04:28 |
| fungi | anyway, i guess it's possible there's something esoteric we missed in the grant for openstack_citest that the puppet mysql access class was setting? | 04:29 |
| mordred | clarkb: I'm running an image upload with that patch applied | 04:29 |
| *** bapalm has joined #openstack-infra | 04:30 | |
| *** sarob has quit IRC | 04:31 | |
| *** asselin_ has joined #openstack-infra | 04:31 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 04:32 |
| *** harlowja_at_home has joined #openstack-infra | 04:32 | |
| ianw | fungi: http://logs.openstack.org/78/289678/2/check/gate-nodepool-python27/501da9d/console.html#_2016-03-08_03_17_59_595 | 04:32 |
| fungi | okay, jenkins01 and 02 reupdates are complete, and shutdown has been cancelled on them | 04:33 |
| *** ccrouch has quit IRC | 04:33 | |
| *** ccrouch has joined #openstack-infra | 04:33 | |
| *** links has joined #openstack-infra | 04:33 | |
| fungi | ianw: starts up at .457 | 04:34 |
| *** salv-orlando has joined #openstack-infra | 04:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:35 | |
| fungi | does GRANT ALL PRIVILEGES ON *.* | 04:35 |
| fungi | TO 'openstack_citest'@'%' | 04:36 |
| clarkb | my first guess would be its connecting via local socket and we only allowed via tcp or vice versa | 04:37 |
| clarkb | but % should match all right? | 04:37 |
| *** bdemers has quit IRC | 04:37 | |
| fungi | that's what i thought | 04:37 |
| clarkb | does it not need to be @'*.*' ? | 04:37 |
| clarkb | mysql so weird | 04:37 |
| fungi | this also worked fine with nova's unit tests; they ran their db interactions successfully | 04:37 |
| fungi | though no idea if they have a much simpler use case | 04:38 |
| clarkb | I'm not sure if they use other dbs in addition to the one test one | 04:38 |
| *** Sukhdev has joined #openstack-infra | 04:39 | |
| *** treaki__ has joined #openstack-infra | 04:39 | |
| ianw | oh, i wonder if the bind address was previously 127.0.0.1 and now isn't ... | 04:39 |
| mordred | clarkb: upload to rax completed in about 30 minutes | 04:40 |
| *** annegentle has quit IRC | 04:40 | |
| *** ccrouch has quit IRC | 04:40 | |
| clarkb | mordred: great | 04:40 |
| fungi | pretty sure i compared the grants and user tables after that script on ubuntu-trusty to the result of our puppet setup on bare-trusty, but also may have been prior to the mysql module upgrade | 04:40 |
| *** ccrouch has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:43 | |
| *** treaki_ has quit IRC | 04:43 | |
| ianw | fungi: what about the old anonymous user issue ... maybe the puppet cleared them out? | 04:44 |
| *** salv-orlando has quit IRC | 04:45 | |
| *** bdemers has joined #openstack-infra | 04:45 | |
| fungi | i've been going long today and starting to get a little fuzzy-brained, so need to look up the details of the old anonymous user issue | 04:45 |
| *** jamesmcarthur has joined #openstack-infra | 04:46 | |
| *** annegentle has joined #openstack-infra | 04:46 | |
| fungi | ahh, empty username | 04:46 |
| fungi | right, there by default normally | 04:46 |
| *** jaosorior has joined #openstack-infra | 04:46 | |
| *** esikachev has joined #openstack-infra | 04:47 | |
| fungi | so maybe nodepool is ending up using the anonymous user instead of the actual openstack_citest user we authorized? | 04:47 |
| fungi | so may need to add "DELETE FROM mysql.user WHERE User='';" | 04:48 |
| fungi | i'll jump on one of these nodes and see if the server package is leaving it there | 04:48 |
| mordred | fungi: the old puppet cleared out the anonymous user | 04:49 |
| mordred | fungi: to my knowledge, the debian/ubuntu packages STILL persist in leaving it in | 04:49 |
| fungi | oh, right, i need a used node | 04:49 |
| mordred | even though it's a terrible idea to have and a only serves to break things | 04:49 |
| fungi | mordred: lovely, so we should almost certainly add that line to the mysql-prep job macro | 04:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add mysql 'localhost' user https://review.openstack.org/289704 | 04:49 |
| mordred | yes | 04:49 |
| *** kzaitsev_mb has quit IRC | 04:49 | |
| mordred | followed by a flush privileges; | 04:49 |
| fungi | and then rerun jjb updates by hand, because that's probably some thousands of jobs it will update | 04:50 |
| *** esikachev has quit IRC | 04:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Remove anonymous mysql users https://review.openstack.org/289704 | 04:52 |
| *** annegentle has quit IRC | 04:52 | |
| ianw | fungi / mordred : ^ | 04:52 |
| fungi | ianw: you beat me to hitting git review | 04:53 |
| *** annegentle has joined #openstack-infra | 04:53 | |
| openstackgerrit | Merged openstack/diskimage-builder: Zerofree the image if possible https://review.openstack.org/289054 | 04:53 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 04:54 |
| fungi | ianw: was there a reason that needed a separate mysqlclient invocation rather than just adding more lines to the existing one right below it? | 04:55 |
| ianw | fungi: just so if it failed, it stopped there, but no, i can quickly move it | 04:55 |
| fungi | i think if any of those fail and throw an error it'll be pretty obvious | 04:55 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Remove anonymous mysql users https://review.openstack.org/289704 | 04:56 |
| fungi | also wondering if flush privileges should happen after the new account creation, or if it's only necessary on changes and deletions | 04:56 |
| fungi | rather, after the grant technically | 04:57 |
| fungi | mordred: ^? | 04:57 |
| fungi | anyway, i'm in favor but this will need another round of manual jjb updates | 04:58 |
| *** jamesmcarthur has quit IRC | 04:58 | |
| ianw | that i don't know (exactly where to flush). every time i use mysql i end up hitting this issue in some form :( | 04:59 |
| *** sdake_ has joined #openstack-infra | 05:00 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/jenkins-job-builder: Allow to omit list of projects in Gerrit trigger config https://review.openstack.org/272392 | 05:00 |
| fungi | refraining from reciting rude rhymes about when to flush | 05:00 |
| *** sdake has quit IRC | 05:01 | |
| *** annegentle has quit IRC | 05:01 | |
| *** annegentle has joined #openstack-infra | 05:02 | |
| *** jamesmcarthur has joined #openstack-infra | 05:02 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Do not silently drop markers that fail to evaluate https://review.openstack.org/289628 | 05:05 |
| *** markvoelker_ has quit IRC | 05:06 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/infra-specs: Create elections.openstack.org https://review.openstack.org/287577 | 05:08 |
| *** annegentle has quit IRC | 05:10 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Do not silently drop markers that fail to evaluate https://review.openstack.org/289628 | 05:11 |
| fungi | tonyb: i should follow up in gerrit but i'm falling asleep. wouldn't election info make sense on the governance.openstack.org site? we can always publish a different repo to a subtree of that so the tc doesn't have to approve changes for it | 05:11 |
| tonyb | fungi: That might work but I thought it would be hard. | 05:12 |
| tonyb | fungi: ahppy to talk about it tomorrow. | 05:13 |
| fungi | tonyb: yeah, i don't feel strongly about it, just knee-jerk reaction to creating more and more subdomains | 05:13 |
| *** thorst has joined #openstack-infra | 05:15 | |
| jhesketh | fungi: that was listed as one of the alternatives :-) | 05:15 |
| craige | For anyone woh wants to help resolve the last blocker with Maniphest, at the bottom of this page: https://etherpad.openstack.org/p/maniphest-migration I describe the current blocker. | 05:16 |
| * craige sends out the bat signal for krotscheck ^^ | 05:16 | |
| clarkb | craige: Zara may also be helpful though is in europe so timezones may be hard | 05:17 |
| * craige nods | 05:18 | |
| *** Sukhdev has quit IRC | 05:19 | |
| fungi | i am fading fast. i intentionally didn't put jenkins06/07 in prepare for shutdown since they tend to jjb quickly and i'm afraid i won't be awake to cancel shutdown on them | 05:20 |
| fungi | they're already on processing deletes, so should be wrapping up shortly anyway | 05:21 |
| clarkb | ok, are they running in screen? | 05:21 |
| *** fawadkhaliq has quit IRC | 05:22 | |
| *** thorst has quit IRC | 05:22 | |
| clarkb | I can shepherd them through the rest of the night | 05:22 |
| tchaypo | Mikal: and lo, a miracle has occurred, I'll see you at pothong | 05:23 |
| craige | \o/ | 05:23 |
| jhesketh | fungi, clarkb: is there something I can help with? | 05:23 |
| clarkb | jhesketh: just finishing up large JJB applies to the servers | 05:23 |
| *** zul has quit IRC | 05:23 | |
| craige | Perhaps the wrong window tchaypo but \o/ see you post bug smash there! | 05:23 |
| clarkb | jhesketh: sounds like 06 and 07 are the last jenkins servers getting updated and they are almost done? | 05:23 |
| fungi | jhesketh: though if 289704 is approved we need to do it all over again | 05:24 |
| fungi | also i didn't get around to trying putting bluebox back into service | 05:24 |
| tchaypo | Craige: nope, I was picking a channel mikal is in | 05:24 |
| *** rlandy has quit IRC | 05:24 | |
| fungi | it was indicated that the admins restarted some backend services and so it may be working again | 05:24 |
| jhesketh | fungi: why wouldn't we wait for the usual rebuilds with 289704? | 05:25 |
| openstackgerrit | Victor Morales proposed openstack-infra/elastic-recheck: Add query for neutron bug 1553595 https://review.openstack.org/289712 | 05:25 |
| openstack | bug 1553595 in neutron "test_external_network_visibility intermittent failure" [Critical,Fix released] https://launchpad.net/bugs/1553595 - Assigned to Kevin Benton (kevinbenton) | 05:25 |
| jhesketh | fungi: is there a way to tell if bluebox is working, or is that just adding it back into the pool | 05:26 |
| clarkb | jhesketh: I think if you manually boot an instance there and it works then that is a sufficient test | 05:26 |
| clarkb | apparently hte problem earlier was dhcp wasn't working | 05:26 |
| jhesketh | clarkb: /me will try a thing | 05:27 |
| fungi | yeah, and then revert my change which zeroed the max-servers | 05:27 |
| jhesketh | yep | 05:27 |
| *** megm has joined #openstack-infra | 05:27 | |
| fungi | also, now that jjb updates are wrapping up, i don't know whether zuul will benefit from another restart so it ceases seeing phantom demand for bare-trusty nodes | 05:28 |
| clarkb | SamYaple: reading up on fstrim, it looks like it may not work if the underlying device doesn't support TRIM | 05:30 |
| clarkb | SamYaple: however it says that support for loopback mounted images is present which is what dib does so I think it will work | 05:30 |
| fungi | oh, good call. less hacky than the zerofree trick | 05:31 |
| clarkb | the trick is it requires a newish kernel which I guess dib likeyl assumed wasn't a thing way back when | 05:31 |
| clarkb | and by newish I mean trusty is fine | 05:31 |
| *** abregman has joined #openstack-infra | 05:32 | |
| *** mrmartin has joined #openstack-infra | 05:33 | |
| *** abregman has quit IRC | 05:33 | |
| *** abregman has joined #openstack-infra | 05:34 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: [WIP] Add support for building images capable of UEFI https://review.openstack.org/287784 | 05:34 |
| *** gmann has quit IRC | 05:35 | |
| fungi | well, i'm out. it's like 1am-ish here. tomorrow is a day o' meetings | 05:35 |
| fungi | night all! | 05:35 |
| fungi | jenkins06 did finish, 07 is really close | 05:36 |
| *** hdd has quit IRC | 05:36 | |
| jhesketh | night fungi | 05:38 |
| *** mrmartin has quit IRC | 05:39 | |
| *** sdake_ has quit IRC | 05:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/ansible-puppet: Always puppetboard upload if puppetboard defined https://review.openstack.org/289718 | 05:40 |
| *** gmann has joined #openstack-infra | 05:40 | |
| *** christian_ has joined #openstack-infra | 05:40 | |
| clarkb | mordred: nibalizer ^ assuming when is valid in an always block that should fix our not uploading to puppetboard on fails | 05:40 |
| clarkb | this is new in ansible 2.0 though | 05:40 |
| clarkb | puppetmaster runs 2.0.1.0 | 05:41 |
| jhesketh | clarkb: I booted a node in bluebox but it didn't attach a public ip.. do I need to figure out a way to do that or is the fact that it booted enough | 05:41 |
| *** fawadkhaliq has joined #openstack-infra | 05:42 | |
| clarkb | jhesketh: they use floating IPs so you need to attach a floating ip | 05:42 |
| fungi | 07 finished too now | 05:43 |
| fungi | /reallysleeping | 05:44 |
| nibalizer | clarkb: there are a couple ways it can fail | 05:45 |
| *** tdasilva has quit IRC | 05:45 | |
| nibalizer | if puppet fails to apply, we get a log | 05:45 |
| nibalizer | if it fails to compile, we don't | 05:45 |
| *** kzaitsev_mb has joined #openstack-infra | 05:45 | |
| *** christian_ has quit IRC | 05:46 | |
| jhesketh | clarkb: ah thanks: ERROR (NotFound): Floating ip pool not found. (HTTP 404) (Request-ID: req-554a3e77-5932-4c1a-9284-65cca64d5416) | 05:46 |
| *** frickler has quit IRC | 05:46 | |
| clarkb | nibalizer: thats independent though | 05:46 |
| clarkb | nibalizer: all this does is run log posting if puppet apply exits failure | 05:46 |
| nibalizer | right at sometimes we won't have logs to post | 05:46 |
| nibalizer | s/at/and/ | 05:47 |
| nibalizer | so this is better, but not a complete solution | 05:47 |
| *** Mmike has quit IRC | 05:48 | |
| openstackgerrit | Anusha Ramineni proposed openstack-infra/project-config: Enable docs jobs for congress https://review.openstack.org/289719 | 05:48 |
| *** kzaitsev_mb has quit IRC | 05:50 | |
| *** jamesmcarthur has quit IRC | 05:51 | |
| *** boris-42 has quit IRC | 05:54 | |
| *** abregman_ has joined #openstack-infra | 05:55 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Do not silently drop markers that fail to evaluate https://review.openstack.org/289628 | 05:56 |
| *** darvon has quit IRC | 05:56 | |
| *** abregman has quit IRC | 05:57 | |
| *** darvon has joined #openstack-infra | 05:57 | |
| jhesketh | clarkb: I have to go offline for a bit, but still around on irc... as mentioned I booted a node but I can't create a floating ip so it may not be ready to add back into nodepool | 05:57 |
| clarkb | jhesketh: ya I am not sure if you have to specify the pool | 06:00 |
| *** sarob has joined #openstack-infra | 06:01 | |
| *** harlowja_at_home has quit IRC | 06:01 | |
| *** rcernin has joined #openstack-infra | 06:02 | |
| *** abregman_ is now known as abregman | 06:03 | |
| *** Mmike has joined #openstack-infra | 06:05 | |
| *** sarob has quit IRC | 06:05 | |
| *** roxanagh_ has joined #openstack-infra | 06:06 | |
| *** markvoelker has joined #openstack-infra | 06:07 | |
| *** jamielennox is now known as jamielennox|away | 06:08 | |
| *** sarob has joined #openstack-infra | 06:11 | |
| *** _sarob has joined #openstack-infra | 06:14 | |
| *** sarob has quit IRC | 06:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/289731 | 06:17 |
| *** asettle has quit IRC | 06:18 | |
| *** Mmike has quit IRC | 06:19 | |
| *** Mmike has joined #openstack-infra | 06:19 | |
| *** tdasilva has joined #openstack-infra | 06:19 | |
| *** thorst has joined #openstack-infra | 06:20 | |
| jhesketh | clarkb: ah, that was it | 06:23 |
| jhesketh | I can boot and attach an IP to it | 06:23 |
| *** salv-orlando has joined #openstack-infra | 06:26 | |
| *** thorst has quit IRC | 06:27 | |
| *** vgridnev has joined #openstack-infra | 06:30 | |
| *** asettle has joined #openstack-infra | 06:30 | |
| *** [1]Thelo has joined #openstack-infra | 06:30 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/288849 | 06:30 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: [WIP] Add support for building images capable of UEFI https://review.openstack.org/287784 | 06:32 |
| *** Thelo has quit IRC | 06:32 | |
| *** [1]Thelo is now known as Thelo | 06:32 | |
| *** salv-orlando has quit IRC | 06:33 | |
| *** jcoufal has joined #openstack-infra | 06:34 | |
| jhesketh | clarkb: okay that appears to work so I'll revert the limit in nodepool | 06:34 |
| *** asettle has quit IRC | 06:35 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Revert "Temporarily disable booting in bluebox" https://review.openstack.org/289738 | 06:36 |
| *** roxanagh_ has quit IRC | 06:37 | |
| *** jcoufal has quit IRC | 06:39 | |
| jhesketh | clarkb: ^ that should be safe, but since things are quiet I'll wait for your (or another infra-core) okay instead of self-approving | 06:41 |
| *** asettle has joined #openstack-infra | 06:42 | |
| *** gildub has quit IRC | 06:42 | |
| *** zzzeek has quit IRC | 06:42 | |
| *** david-lyle_ has joined #openstack-infra | 06:44 | |
| *** david-lyle has quit IRC | 06:44 | |
| *** fawadkhaliq has quit IRC | 06:44 | |
| *** gildub has joined #openstack-infra | 06:45 | |
| *** hichihara has quit IRC | 06:46 | |
| *** esikachev has joined #openstack-infra | 06:48 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 06:48 |
| *** jtomasek has joined #openstack-infra | 06:49 | |
| Jeffrey4l | rax-iad seems have two ethernets. eth0 is a publice network(104.130.3.103), and eth1 is a private network(10.208.234.87). But other node only has one network. anyone know why/how it happend? | 06:51 |
| *** vgridnev has quit IRC | 06:52 | |
| *** esikachev has quit IRC | 06:52 | |
| openstackgerrit | Cody A.W. Somerville proposed openstack/gertty: Make 'title' attribute available on dialog widgets https://review.openstack.org/289748 | 06:53 |
| *** kushal has joined #openstack-infra | 06:55 | |
| *** pcaruana has quit IRC | 06:57 | |
| *** gildub has quit IRC | 06:59 | |
| *** asettle has quit IRC | 07:01 | |
| *** oanson has quit IRC | 07:02 | |
| *** asettle has joined #openstack-infra | 07:09 | |
| cody-somerville | jeblair: So I've looked at short title stuff further and I don't think it is actually necessary if we put the ChangeList query description at the front of the title. | 07:14 |
| *** armax has quit IRC | 07:15 | |
| cody-somerville | jeblair: also see above change ^^ to make title attribute available on dialog widgets so they show up in breadcrumb nicely instead of as classname for overlay widget. | 07:15 |
| *** _nadya_ has joined #openstack-infra | 07:16 | |
| *** _nadya_ has quit IRC | 07:16 | |
| *** gildub has joined #openstack-infra | 07:16 | |
| *** sarob has joined #openstack-infra | 07:22 | |
| *** thorst has joined #openstack-infra | 07:25 | |
| *** _sarob has quit IRC | 07:25 | |
| *** gildub has quit IRC | 07:26 | |
| *** sshnaidm has quit IRC | 07:26 | |
| *** gildub has joined #openstack-infra | 07:26 | |
| *** sarob has quit IRC | 07:26 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/289731 | 07:27 |
| *** sbalukoff has joined #openstack-infra | 07:28 | |
| *** thorst has quit IRC | 07:32 | |
| *** mrmartin has joined #openstack-infra | 07:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove experimental-bindep-jobs from repos https://review.openstack.org/288694 | 07:35 |
| *** nikhil_k_ has joined #openstack-infra | 07:36 | |
| *** nikhil has quit IRC | 07:38 | |
| *** kushal has quit IRC | 07:40 | |
| *** salv-orlando has joined #openstack-infra | 07:42 | |
| *** takashi has left #openstack-infra | 07:42 | |
| *** nikhil_k_ has quit IRC | 07:44 | |
| *** fhubik has joined #openstack-infra | 07:45 | |
| *** fhubik is now known as fhubik_brb | 07:45 | |
| *** flepied1 has quit IRC | 07:45 | |
| *** nikhil has joined #openstack-infra | 07:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:48 | |
| *** denisra_ has joined #openstack-infra | 07:48 | |
| *** e0ne has quit IRC | 07:49 | |
| *** salv-orlando has quit IRC | 07:50 | |
| *** cody-somerville has quit IRC | 07:51 | |
| *** cody-somerville has joined #openstack-infra | 07:51 | |
| *** yaume has joined #openstack-infra | 07:51 | |
| *** denisra has quit IRC | 07:52 | |
| *** fhubik_brb is now known as fhubik | 07:52 | |
| *** frickler has joined #openstack-infra | 07:52 | |
| *** frickler has quit IRC | 07:52 | |
| *** kzaitsev_mb has quit IRC | 07:52 | |
| *** frickler has joined #openstack-infra | 07:53 | |
| *** ihrachys has joined #openstack-infra | 07:56 | |
| *** scheuran has joined #openstack-infra | 07:58 | |
| *** vgridnev has joined #openstack-infra | 08:00 | |
| *** ihrachys has quit IRC | 08:02 | |
| *** asettle has quit IRC | 08:08 | |
| *** jogo has quit IRC | 08:08 | |
| *** jogo has joined #openstack-infra | 08:12 | |
| *** ihrachys has joined #openstack-infra | 08:13 | |
| AJaeger | fungi, Sam-I-Am : heat 6.0.0 is Mitaka, everything is correct for release notes AFAIU | 08:13 |
| *** vgridnev has quit IRC | 08:14 | |
| *** andreykurilin__ has joined #openstack-infra | 08:15 | |
| *** matrohon has joined #openstack-infra | 08:16 | |
| *** asettle has joined #openstack-infra | 08:18 | |
| *** shardy has joined #openstack-infra | 08:18 | |
| *** acabot has joined #openstack-infra | 08:21 | |
| *** ildikov has joined #openstack-infra | 08:22 | |
| *** asettle has quit IRC | 08:23 | |
| *** sarob has joined #openstack-infra | 08:23 | |
| *** cody-somerville has quit IRC | 08:24 | |
| *** flepied has joined #openstack-infra | 08:25 | |
| *** mikelk has joined #openstack-infra | 08:30 | |
| *** thorst has joined #openstack-infra | 08:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for various tox tests not using database https://review.openstack.org/289785 | 08:32 |
| *** yaume has quit IRC | 08:32 | |
| *** dizquierdo has joined #openstack-infra | 08:36 | |
| anteaya | jhesketh: have you a moment to join the third party meeting in -meeting? | 08:36 |
| *** thorst has quit IRC | 08:37 | |
| *** wei-wang has joined #openstack-infra | 08:37 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for various tox tests not using database https://review.openstack.org/289785 | 08:39 |
| *** asettle has joined #openstack-infra | 08:40 | |
| *** cody-somerville has joined #openstack-infra | 08:40 | |
| *** cody-somerville has quit IRC | 08:40 | |
| *** cody-somerville has joined #openstack-infra | 08:40 | |
| *** christian_ has joined #openstack-infra | 08:40 | |
| *** acruz has joined #openstack-infra | 08:42 | |
| anteaya | jhesketh: we were just talking about the zuul memory leak a bit, the meeting is over now, I'm off to bed | 08:45 |
| anteaya | g'night again | 08:45 |
| *** nmagnezi has joined #openstack-infra | 08:46 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:47 | |
| AJaeger | good night, anteaya ! | 08:47 |
| *** esikachev has joined #openstack-infra | 08:49 | |
| *** salv-orlando has joined #openstack-infra | 08:49 | |
| *** jordanP has joined #openstack-infra | 08:50 | |
| *** jcoufal has joined #openstack-infra | 08:50 | |
| *** dingyichen has quit IRC | 08:52 | |
| *** asettle has quit IRC | 08:53 | |
| *** jlanoux has joined #openstack-infra | 08:53 | |
| *** esikachev has quit IRC | 08:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Add bandit gate job for Octavia https://review.openstack.org/286912 | 08:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for python-bitrot-jobs https://review.openstack.org/289793 | 08:56 |
| *** hashar has joined #openstack-infra | 09:00 | |
| *** acruz has quit IRC | 09:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert sahara to ubuntu-trusty https://review.openstack.org/289797 | 09:01 |
| *** fhubik has quit IRC | 09:02 | |
| *** christian_ has quit IRC | 09:02 | |
| *** arxcruz has joined #openstack-infra | 09:02 | |
| *** yaume has joined #openstack-infra | 09:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove unused selenium.yaml https://review.openstack.org/289799 | 09:03 |
| *** kzaitsev_mb has quit IRC | 09:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove unused selenium.yaml https://review.openstack.org/289799 | 09:04 |
| *** wei-wang has quit IRC | 09:05 | |
| *** yaume has quit IRC | 09:07 | |
| *** jistr has joined #openstack-infra | 09:11 | |
| *** ifarkas has joined #openstack-infra | 09:13 | |
| *** ildikov has quit IRC | 09:16 | |
| *** dizquierdo has quit IRC | 09:16 | |
| yolanda | good morning | 09:19 |
| *** lucas-dinner is now known as lucasagomes | 09:19 | |
| *** salv-orlando has quit IRC | 09:19 | |
| *** cznewt has quit IRC | 09:19 | |
| *** ihrachys has quit IRC | 09:20 | |
| openstackgerrit | Paul Bourke proposed openstack-infra/project-config: Add new experimental jobs for Kolla oraclelinux https://review.openstack.org/287957 | 09:22 |
| *** cznewt has joined #openstack-infra | 09:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for various tox tests not using database https://review.openstack.org/289785 | 09:27 |
| *** ildikov has joined #openstack-infra | 09:28 | |
| *** cznewt has quit IRC | 09:29 | |
| *** Jeffrey4l has quit IRC | 09:33 | |
| *** thorst has joined #openstack-infra | 09:35 | |
| *** cznewt has joined #openstack-infra | 09:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert storyboard-docs to ubuntu-trusty https://review.openstack.org/289816 | 09:36 |
| *** tpsilva has joined #openstack-infra | 09:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert storyboard-docs, -announce-release to ubuntu-trusty https://review.openstack.org/289816 | 09:37 |
| AJaeger | good morning, yolanda ! | 09:38 |
| *** fhubik has joined #openstack-infra | 09:38 | |
| *** emmenemoi has left #openstack-infra | 09:42 | |
| *** thorst has quit IRC | 09:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for various tox tests not using database https://review.openstack.org/289785 | 09:43 |
| *** sfinucan has joined #openstack-infra | 09:43 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: horizon: Use ubuntu-trusty https://review.openstack.org/289819 | 09:47 |
| *** esikachev has joined #openstack-infra | 09:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix node name in gate-infra-docs-index https://review.openstack.org/289820 | 09:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for releasenotes jobs https://review.openstack.org/289824 | 09:51 |
| *** esikachev has quit IRC | 09:51 | |
| *** flepied has quit IRC | 09:54 | |
| *** ociuhandu has quit IRC | 09:54 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Always puppetboard upload if puppetboard defined https://review.openstack.org/289718 | 09:57 |
| *** flepied has joined #openstack-infra | 10:00 | |
| *** asettle has joined #openstack-infra | 10:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config: Enable docs jobs for congress https://review.openstack.org/289719 | 10:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 10:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert linters to use ubuntu-trusty https://review.openstack.org/289832 | 10:07 |
| *** zul has joined #openstack-infra | 10:07 | |
| *** yamamoto_ has quit IRC | 10:09 | |
| *** dtantsur|afk is now known as dtantsur | 10:11 | |
| *** fhubik is now known as fhubik_brb | 10:13 | |
| *** esikachev has joined #openstack-infra | 10:13 | |
| *** Qiming has quit IRC | 10:14 | |
| *** sambetts|afk is now known as sambetts | 10:16 | |
| openstackgerrit | Merged openstack-infra/system-config: Allow the review history panel to resize https://review.openstack.org/259560 | 10:16 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 10:19 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Fix unrecognized escape puppet warning https://review.openstack.org/279672 | 10:19 |
| *** mikelk has quit IRC | 10:20 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Use VPN devstack plugin in heat jobs https://review.openstack.org/259463 | 10:20 |
| *** salv-orlando has joined #openstack-infra | 10:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Add puppet-octavia in Puppet OpenStack project https://review.openstack.org/289024 | 10:20 |
| *** rguillebert has joined #openstack-infra | 10:21 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for check-osc-plugins https://review.openstack.org/289843 | 10:21 |
| *** ociuhandu has joined #openstack-infra | 10:23 | |
| *** cznewt has quit IRC | 10:24 | |
| *** sarob has quit IRC | 10:29 | |
| *** fhubik_brb is now known as fhubik | 10:30 | |
| *** salv-orlando has quit IRC | 10:31 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert tox jobs with DB to use ubuntu-trusty https://review.openstack.org/289848 | 10:31 |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** openstackgerrit has joined #openstack-infra | 10:33 | |
| *** ihrachys has joined #openstack-infra | 10:33 | |
| *** cznewt has joined #openstack-infra | 10:35 | |
| *** fhubik is now known as fhubik_brb | 10:36 | |
| stevemar | AJaeger: poke | 10:38 |
| AJaeger | stevemar: hi! | 10:38 |
| AJaeger | curious why I added you to that review? | 10:38 |
| stevemar | AJaeger: so i don't think we need the legacy_drivers change in https://review.openstack.org/#/c/289848/1/zuul/layout.yaml | 10:38 |
| stevemar | AJaeger: a little :) | 10:38 |
| AJaeger | stevemar: we will switch from bare-trusty to ubuntu-trusty. | 10:39 |
| *** salv-orlando has joined #openstack-infra | 10:39 | |
| AJaeger | Question is : Does legacy_driver need a database or not? | 10:39 |
| stevemar | AJaeger: it has the same database requirement as our unit tests | 10:39 |
| stevemar | so yes | 10:39 |
| AJaeger | then the change is fine since it moves to the template using database access | 10:40 |
| stevemar | AJaeger: are the py27 jobs being moved to ubuntu-trusty ? | 10:40 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Hide 'quote' button for non-logged-in users https://review.openstack.org/289851 | 10:40 |
| AJaeger | stevemar: already moved on Friday ;) | 10:40 |
| AJaeger | glad that you didn't notice ;) | 10:40 |
| stevemar | ohhh | 10:40 |
| stevemar | AJaeger: caught me ! | 10:40 |
| *** thorst has joined #openstack-infra | 10:40 | |
| AJaeger | http://logs.openstack.org/68/289768/1/check/gate-keystone-python27/54ce571/console.html | 10:41 |
| stevemar | so ubuntu-trusty will install mysql on the machine? | 10:41 |
| AJaeger | stevemar: we didn't do a large announcement yet, need to solve a few problems first... | 10:41 |
| stevemar | AJaeger: better to ask for forgiveness and deal with the fallout instead of asking for permission and having a bikeshed argument :) | 10:41 |
| AJaeger | stevemar: ubuntu-trusty does not come with database - so your docs jobs will not have one - but we add database for specific jobs like python27 or this tox-db one | 10:42 |
| stevemar | i see | 10:42 |
| AJaeger | stevemar: we now have ubuntu-trusty on all clouds - instead of bare-trusty only on rackspace | 10:42 |
| stevemar | this is a good first step to running our tests with mysql instead of sqlite | 10:42 |
| AJaeger | let's tell fungi that keystone wants to test with mysql... | 10:43 |
| stevemar | AJaeger: fungi: we definitely want to test with mysql!!! | 10:43 |
| AJaeger | We don't want to setup database for those projects and jobs that don't need it | 10:43 |
| AJaeger | stevemar: that's perfectly fine ;) | 10:43 |
| *** fhubik_brb is now known as fhubik | 10:43 | |
| jhesketh | anteaya: sorry I missed you | 10:44 |
| stevemar | AJaeger: yeah, that makes sense, no need to set it up for docs and such | 10:44 |
| jhesketh | AJaeger: I'm back.. thinking I should merge in the nodepool revert before the load increases with people waking up | 10:44 |
| jhesketh | any objections? | 10:44 |
| AJaeger | stevemar: so, would be nice if you comment on the change and say that you need database for your jobs | 10:44 |
| AJaeger | jhesketh: go ahead... | 10:44 |
| *** dims has joined #openstack-infra | 10:44 | |
| stevemar | AJaeger: yes, i will | 10:44 |
| AJaeger | yolanda: what do you think? | 10:44 |
| AJaeger | stevemar: but first I need to fix the build failure ;( | 10:45 |
| *** christian_ has joined #openstack-infra | 10:45 | |
| jhesketh | AJaeger: +w'd | 10:45 |
| *** ihrachys has quit IRC | 10:46 | |
| yolanda | AJaeger ++ | 10:46 |
| *** gildub has quit IRC | 10:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert tox jobs with DB to use ubuntu-trusty https://review.openstack.org/289848 | 10:47 |
| *** thorst has quit IRC | 10:47 | |
| AJaeger | yolanda, jhesketh: Could you review https://review.openstack.org/289820 , please? That's a bug I introduced ;( | 10:48 |
| jhesketh | AJaeger: looking | 10:48 |
| *** esikachev has quit IRC | 10:48 | |
| jhesketh | yolanda: oh you're around. I just self-approved a revert to bring bluebox back into nodepool fyi | 10:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for check-osc-plugins https://review.openstack.org/289843 | 10:49 |
| yolanda | jhesketh yes around, i'm concentrated on some stuff so i wasn't reading the channel so much | 10:51 |
| AJaeger | yolanda, jhesketh: If it blows up again, we revert the revert ;) | 10:51 |
| yolanda | i love that chains :) | 10:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for various tox tests not using database https://review.openstack.org/289785 | 10:53 |
| yolanda | jhesketh, what's going on with bluebox? looks as we have outages quite often | 10:55 |
| AJaeger | jhesketh: if you want to create a tiny bit more of load, reviews for the other ubuntu-trusty changes would be nice - as time permits... https://review.openstack.org/#/q/status:open++branch:master+topic:bindep | 10:55 |
| openstackgerrit | Francesco Longo proposed openstack-infra/project-config: Added IoTronic project. https://review.openstack.org/286113 | 10:56 |
| *** claudiub has joined #openstack-infra | 10:56 | |
| *** asettle has quit IRC | 10:58 | |
| *** pblaho has joined #openstack-infra | 10:58 | |
| jhesketh | yolanda: there were network issues earlier today.. fungi (I think) was dealing with them, but they are back up now so we can revert the nodepool removal | 10:58 |
| jhesketh | yolanda: I've confirmed you can boot and ssh into a node | 10:58 |
| jhesketh | and will watch to see if a job runs on bluebox successfully | 10:58 |
| jhesketh | yolanda: if it plays up we can take them out of the pool again, just wanted to increase the capacity | 10:58 |
| jhesketh | AJaeger: will try and get to those tomorrow (it's getting latish sorry) | 10:59 |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| yolanda | jhesketh, thanks for that, i can continue with that if is too late for you | 10:59 |
| jhesketh | nah it's not too late, just limited room for extra brain cycles ;-) | 10:59 |
| *** mizar has joined #openstack-infra | 11:00 | |
| *** oanson has joined #openstack-infra | 11:01 | |
| *** Qiming has joined #openstack-infra | 11:04 | |
| *** yamamoto has quit IRC | 11:05 | |
| *** yamamoto has joined #openstack-infra | 11:06 | |
| *** yamamoto has quit IRC | 11:10 | |
| *** amotoki has joined #openstack-infra | 11:10 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Temporarily disable booting in bluebox" https://review.openstack.org/289738 | 11:12 |
| openstackgerrit | Merged openstack-infra/project-config: Fix node name in gate-infra-docs-index https://review.openstack.org/289820 | 11:13 |
| *** christian_ has quit IRC | 11:17 | |
| *** fhubik is now known as fhubik_brb | 11:23 | |
| *** fhubik_brb is now known as fhubik | 11:26 | |
| jhesketh | yolanda: looks like bluebox is running jobs successfully: https://jenkins04.openstack.org/job/gate-tempest-dsvm-full-drbd-devstack-nv/1724/console | 11:30 |
| yolanda | nice | 11:31 |
| AJaeger | jhesketh: no worries, thanks | 11:31 |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| * jhesketh is still around for a little bit more if needed and will keep watching | 11:32 | |
| *** jlanoux_ has joined #openstack-infra | 11:32 | |
| *** jlanoux has quit IRC | 11:33 | |
| *** rossella_s has joined #openstack-infra | 11:33 | |
| *** mattt has joined #openstack-infra | 11:37 | |
| *** yamamoto has joined #openstack-infra | 11:37 | |
| mattt | hi all, are there any issues with whatever servers up CI logs? trying to browse http://logs.openstack.org/56/289856/3/check/gate-openstack-ansible-os_cinder-ansible-func-ubuntu-trusty/7081a47/console.html for example and it seems to hang mid-way through | 11:38 |
| mattt | seen same thing trying to wget it | 11:38 |
| mattt | (from a different location, to rule that out also :) | 11:39 |
| *** pblaho has quit IRC | 11:39 | |
| *** mikelk has joined #openstack-infra | 11:42 | |
| *** |-paul-| has joined #openstack-infra | 11:42 | |
| *** rfolco has joined #openstack-infra | 11:44 | |
| *** thorst has joined #openstack-infra | 11:45 | |
| *** gildub has joined #openstack-infra | 11:45 | |
| *** jlanoux has joined #openstack-infra | 11:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: In the service lock, reset the service, not the lock https://review.openstack.org/289692 | 11:47 |
| *** asselin_ has quit IRC | 11:47 | |
| *** asselin_ has joined #openstack-infra | 11:48 | |
| *** jlanoux_ has quit IRC | 11:48 | |
| *** thorst has quit IRC | 11:52 | |
| *** fhubik is now known as fhubik_brb | 11:52 | |
| *** salv-orl_ has joined #openstack-infra | 11:52 | |
| *** baoli has joined #openstack-infra | 11:55 | |
| *** salv-orlando has quit IRC | 11:56 | |
| *** baoli has quit IRC | 11:56 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Mention Comments in Heading for Events Timeline https://review.openstack.org/289879 | 11:56 |
| *** andymccr has joined #openstack-infra | 11:57 | |
| *** sshnaidm has joined #openstack-infra | 11:57 | |
| AJaeger | mattt: let me check... | 11:58 |
| *** rhallisey has joined #openstack-infra | 11:59 | |
| *** _amrith_ is now known as amrith | 11:59 | |
| AJaeger | yolanda: any idea whether logs.openstack.org is broken? See mattt's URL above, the 3.7 MB cannot be downloaded | 12:00 |
| yolanda | AJaeger i 'll take a look | 12:00 |
| mattt | thank you AJaeger, yolanda | 12:02 |
| *** jaosorior has quit IRC | 12:02 | |
| *** sdague has joined #openstack-infra | 12:02 | |
| *** jaosorior has joined #openstack-infra | 12:02 | |
| *** davideagnello has quit IRC | 12:03 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on Null exception https://review.openstack.org/289883 | 12:03 |
| yolanda | AJaeger, mattt, i'm browsing that correctly | 12:04 |
| *** aysyd has joined #openstack-infra | 12:04 | |
| yolanda | mm, actually it hangs at some point but it still says loaded | 12:05 |
| yolanda | going to check with an wget | 12:05 |
| *** ihrachys has joined #openstack-infra | 12:05 | |
| *** aysyd has quit IRC | 12:06 | |
| mattt | yolanda: i noticed my wget eventually finished | 12:06 |
| mattt | but i've tried again and it's hanging | 12:06 |
| *** gildub has quit IRC | 12:06 | |
| *** aysyd has joined #openstack-infra | 12:06 | |
| *** sfinucan_ has joined #openstack-infra | 12:07 | |
| *** jaypipes has joined #openstack-infra | 12:07 | |
| yolanda | i'm seeing I/O errors on static | 12:08 |
| rcarrillocruz | :/ | 12:09 |
| rcarrillocruz | yolanda: does static have a cinder volume ? | 12:09 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on Null exception https://review.openstack.org/289883 | 12:09 |
| *** sfinucan has quit IRC | 12:10 | |
| yolanda | yes, we use volumes for each content | 12:10 |
| yolanda | logs, tarballs, etc | 12:11 |
| *** weshay has joined #openstack-infra | 12:15 | |
| *** sfinucan_ has quit IRC | 12:16 | |
| *** sfinucan has joined #openstack-infra | 12:16 | |
| yolanda | i see apache using all cpu, and hanging at a different point all the time | 12:17 |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 12:17 |
| *** lucasagomes is now known as lucas-hungry | 12:19 | |
| *** mizar has quit IRC | 12:20 | |
| AJaeger | mattt, yolanda now downloaded for me as well - took some time... | 12:21 |
| AJaeger | I/O errors ? That's bad ;( | 12:21 |
| *** jcoufal has quit IRC | 12:22 | |
| jhesketh | (bluebox looks good fyi) | 12:22 |
| *** jcoufal has joined #openstack-infra | 12:22 | |
| openstackgerrit | Purandhar Sairam Mannidi proposed openstack/diskimage-builder: Add support for building images capable of UEFI https://review.openstack.org/287784 | 12:23 |
| *** yamamoto has quit IRC | 12:23 | |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 12:25 |
| *** baoli has joined #openstack-infra | 12:27 | |
| openstackgerrit | Dougal Matthews proposed openstack-dev/hacking: Update the assertRaises example to be more modern https://review.openstack.org/284097 | 12:27 |
| *** |-paul-| has quit IRC | 12:27 | |
| *** christian_ has joined #openstack-infra | 12:29 | |
| *** christian_ is now known as christiana_ | 12:32 | |
| *** amrith has quit IRC | 12:33 | |
| *** fhubik_brb is now known as fhubik | 12:33 | |
| *** amrith has joined #openstack-infra | 12:34 | |
| *** yamamoto has quit IRC | 12:34 | |
| *** gordc has joined #openstack-infra | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:35 | |
| *** zhurong has joined #openstack-infra | 12:35 | |
| *** rfolco has quit IRC | 12:36 | |
| *** christiana_ has quit IRC | 12:37 | |
| yolanda | same here, worked for me after a long time | 12:39 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Don't override the private_net settings for the tenant https://review.openstack.org/289495 | 12:39 |
| *** kaisers1 has quit IRC | 12:42 | |
| *** thorst has joined #openstack-infra | 12:43 | |
| *** thorst has quit IRC | 12:43 | |
| *** thorst_ has joined #openstack-infra | 12:44 | |
| *** kaisers has joined #openstack-infra | 12:44 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Generate a log with deprecation warnings https://review.openstack.org/289901 | 12:45 |
| openstackgerrit | Merged openstack-infra/shade: In the service lock, reset the service, not the lock https://review.openstack.org/289692 | 12:50 |
| *** baoli_ has joined #openstack-infra | 12:52 | |
| *** baoli has quit IRC | 12:55 | |
| frickler | is there a known issue with neutron checks being flaky? see https://review.openstack.org/289767 the set of failures changed after a recheck and now seems to consist of timeouts mostly | 12:55 |
| *** markvoelker has quit IRC | 12:57 | |
| *** bdemers has quit IRC | 12:57 | |
| jaypipes | dstufft: good morning :) any ideas on why this would be happening? all py27 tests pass just fine... http://paste.openstack.org/show/489679/ | 12:57 |
| *** markvoelker has joined #openstack-infra | 12:57 | |
| *** edmondsw has joined #openstack-infra | 12:57 | |
| dstufft | jaypipes: yea, pip 8.1 doesn't have that function anymore, maybe pip 8.0 as well | 12:58 |
| *** christiana_ has joined #openstack-infra | 12:58 | |
| jaypipes | dstufft: k. would this be the pip that tox or virtualenv is installing? | 12:58 |
| mordred | infra-root: I have tagged shade 1.5.1 to fix the swift upload mutex reset bug | 12:59 |
| *** rfolco has joined #openstack-infra | 12:59 | |
| dstufft | jaypipes: well tox creates the virtualenv which installs pip | 12:59 |
| *** markvoelker_ has joined #openstack-infra | 12:59 | |
| dstufft | pip-missing-reqs should use packaging instead | 12:59 |
| *** dprince has joined #openstack-infra | 13:00 | |
| dstufft | pip install packaging and then from packaging.utils import cannoniclize_name or however it's spelled | 13:00 |
| dstufft | https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name | 13:00 |
| *** markvoelker has quit IRC | 13:01 | |
| jaypipes | dstufft: I guess I'm just wondering how to fix this (or even if I *should* fix this?) locally. I don't really want to install another local version of pip because last time I did that, everything broke with my py27 envs... perhaps I should just leave this be? | 13:02 |
| dstufft | jaypipes: dunno, I don't know anything about pip-missing-reqs except that it is apparently using private APIs ;) | 13:03 |
| mordred | jaypipes: is that using https://github.com/r1chardj0n3s/pip-check-reqs or a local script? | 13:03 |
| jaypipes | mordred: this is me running $ tox | 13:04 |
| jaypipes | mordred: locally, that's all. | 13:04 |
| jaypipes | mordred: in nova. no args, no nothin. | 13:04 |
| mordred | yah. looks like a bug in upstream pip-missing-reqs | 13:04 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: WIP: Add an IPv6 gate job https://review.openstack.org/289445 | 13:04 |
| mordred | https://github.com/r1chardj0n3s/pip-check-reqs/blob/master/pip_check_reqs/find_missing_reqs.py#L10 | 13:04 |
| mordred | jaypipes: correct fix would be to patch pip-missing-reqs, send a PR and get richard jones to cut a new release | 13:05 |
| jaypipes | mordred: k, thx | 13:05 |
| jaypipes | mordred: will do. | 13:05 |
| mordred | jaypipes: for you for now ... you could comment out the tox envlist that does that for the moment | 13:06 |
| mordred | jaypipes: during feature freeze I'd hope we don't have too many missing reqs slip in to nova :) | 13:06 |
| *** doug-fish has joined #openstack-infra | 13:07 | |
| *** trown has joined #openstack-infra | 13:07 | |
| *** annegentle has joined #openstack-infra | 13:08 | |
| jaypipes | mordred: damn it. dims already beat me to it :) https://github.com/r1chardj0n3s/pip-check-reqs/pull/8 | 13:09 |
| dims | lol, you are the 2nd person to say that jaypipes :) | 13:09 |
| jaypipes | hehe | 13:09 |
| mordred | dims: why not use packaging.canonicalize_name? (curious) | 13:10 |
| mordred | infra-root: I've got to AFK for a bit - but wiki.openstack.org seems to be unhappy | 13:11 |
| ihrachys | AJaeger: hey | 13:13 |
| ihrachys | AJaeger: I spotted an issue since latest constraints changes for neutron liberty | 13:13 |
| ihrachys | AJaeger: periodic jobs do not have the constraints file prepared for them | 13:14 |
| ihrachys | probably because periodic-{name}-python27{branch-name} uses branch-git-prep | 13:14 |
| *** pblaho has joined #openstack-infra | 13:15 | |
| *** yamamoto has quit IRC | 13:15 | |
| dims | mordred : jaypipes : +1 to the suggestion, i probably did not spot that | 13:15 |
| *** thiagop has joined #openstack-infra | 13:15 | |
| jaypipes | dims: I merely pass on the wisdom of the dstufft :) | 13:15 |
| *** keedya has quit IRC | 13:16 | |
| *** fhubik has quit IRC | 13:16 | |
| * dims bows to dstufft :) | 13:17 | |
| *** yamamoto has joined #openstack-infra | 13:18 | |
| nibalizer | ohai | 13:20 |
| nibalizer | wiki seems to be wikining | 13:21 |
| *** yamamoto has quit IRC | 13:21 | |
| *** kushal has joined #openstack-infra | 13:22 | |
| *** lucas-hungry is now known as lucasagomes | 13:23 | |
| *** kgiusti has joined #openstack-infra | 13:24 | |
| *** cznewt has quit IRC | 13:25 | |
| *** amrith is now known as _amrith_ | 13:29 | |
| dims | sdague : mordred : may i have your support for logging deprecations (https://review.openstack.org/#/c/289901/) example log - http://logs.openstack.org/01/289901/1/check/gate-tempest-dsvm-full/b7a56e7/logs/deprecations.txt.gz | 13:29 |
| dims | s/may i/may i please/ | 13:29 |
| *** nikhil has quit IRC | 13:29 | |
| annegentle | nibalizer: it's down for me, but not for everyone http://www.downforeveryoneorjustme.com/wiki.openstack.org | 13:30 |
| annegentle | nibalizer: annd now it's back | 13:30 |
| * nibalizer looks at logs | 13:30 | |
| nibalizer | what error were you getting? | 13:30 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Prepare constraints file for periodic bitrot jobs https://review.openstack.org/289918 | 13:34 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Prepare constraints file for periodic bitrot jobs https://review.openstack.org/289918 | 13:35 |
| *** pblaho has quit IRC | 13:35 | |
| nibalizer | eep | 13:36 |
| nibalizer | http://paste.openstack.org/show/489684/ | 13:36 |
| *** armax has joined #openstack-infra | 13:37 | |
| nibalizer | there are a ton of those messages | 13:37 |
| nibalizer | in the error.log | 13:37 |
| *** rlandy has joined #openstack-infra | 13:37 | |
| sdague | dims: what do you need? | 13:38 |
| *** cbader has joined #openstack-infra | 13:38 | |
| nibalizer | seems to be the start | 13:39 |
| nibalizer | http://paste.openstack.org/show/489685/ | 13:39 |
| dims | sdague : am a bit rusty about the devstack-gate. wondering if that's the right spot to find and log deprecation(s) or if there's a better way, i do seem to get some good output | 13:39 |
| sdague | that's probably a way to do it | 13:40 |
| *** annegentle has quit IRC | 13:41 | |
| dims | sdague : will it work on both sides of grenade too? | 13:42 |
| *** annegentle has joined #openstack-infra | 13:42 | |
| nibalizer | the access and error logs seem to be happier now | 13:42 |
| sdague | the logs will be in a different place | 13:42 |
| nibalizer | only expected errors like 404 in error.log | 13:42 |
| sdague | however... deprecations in grenade are not a thing you want to track | 13:42 |
| dims | sdague : y expecting 2 log files | 13:42 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on my schedule https://review.openstack.org/289923 | 13:42 |
| sdague | because we'd expect there to be deprecations | 13:42 |
| sdague | it's kind of the point | 13:43 |
| dims | sdague : i was trying to see if there's value in seeing if something that was deprecated is still deprecated... | 13:43 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on my schedule https://review.openstack.org/289923 | 13:44 |
| *** pradk_ has joined #openstack-infra | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:47 | |
| *** zzzeek has joined #openstack-infra | 13:48 | |
| *** zhurong has quit IRC | 13:49 | |
| *** lennyb__ is now known as lennyb | 13:49 | |
| *** zhurong has joined #openstack-infra | 13:50 | |
| fungi | nibalizer: what did you end up doing to get the wiki back on track and stop crashing apache? | 13:51 |
| nibalizer | fungi: by the time i got here it had repaired itself | 13:52 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Generate fedora-atomic images using dib https://review.openstack.org/287167 | 13:52 |
| fungi | oh | 13:52 |
| fungi | i hate it when that happens | 13:52 |
| nibalizer | its possible mordred issued a service restart before dissapearing into the mist | 13:52 |
| yolanda | nibalizer, i thought you fixed, by the time i looked, it was working as well :) | 13:52 |
| SamYaple | clarkb: fungi re fstrim, it should work on any kernel for many years | 13:52 |
| fungi | so wiki has apache crashing, logs.o.o has filesystem i/o errors going on? | 13:53 |
| SamYaple | clarkb: im working on a patch for fstrim instead of zerofree | 13:53 |
| SamYaple | expect it today | 13:53 |
| fungi | SamYaple: sounds awesome | 13:53 |
| * SamYaple is new to DIB | 13:53 | |
| fungi | as for the wiki and static issues, nothing apparent on rackspace's status page but i'll check our tickets for that project | 13:54 |
| automagically | FYI all, seeing recurring issues downloading logs from gate failures. Example is http://logs.openstack.org/54/289454/3/check/gate-openstack-ansible-os_heat-ansible-func-ubuntu-trusty/ab268e0/console.html which is 3.6mb, but I can only ever seem to fetch the first ~40kb | 13:55 |
| *** hdd has joined #openstack-infra | 13:55 | |
| *** andrearosa has joined #openstack-infra | 13:56 | |
| *** krtaylor has quit IRC | 13:57 | |
| *** mizar has joined #openstack-infra | 13:57 | |
| *** mizar is now known as flongo | 13:58 | |
| *** flongo has quit IRC | 13:58 | |
| *** flongo has joined #openstack-infra | 13:58 | |
| fungi | automagically: yep, i'm suspecting problems in rackspace | 13:59 |
| fungi | just checked our account, no support tickets raised by them alerting us to an issue | 13:59 |
| *** jistr is now known as jistr|call | 13:59 | |
| automagically | Thx fungi | 13:59 |
| *** markvoelker_ has quit IRC | 14:00 | |
| fungi | ...apache2[22569]: segfault at 0 ip 00007ffb62cf1d93 sp 00007ffb4d7e9b20 error 6 in libapr-1.so.0.4.6... | 14:00 |
| fungi | from dmesg | 14:01 |
| fungi | i wonder if the apache crashes on wiki and static are related to a broader attack of some sort | 14:01 |
| fungi | yolanda: where were you seeing i/o errors on static.o.o? dmesg doesn't show any so i'm guessing it wasn't at the block layer | 14:02 |
| yolanda | fungi, in the error log for logs.openstack.org | 14:02 |
| *** dizquierdo has joined #openstack-infra | 14:03 | |
| fungi | oh, so i/o between the process and its filesystem calls | 14:04 |
| *** nelsnelson has joined #openstack-infra | 14:04 | |
| fungi | i'll see if i can spot new malicious access requests from specific addresses or whether this is something distributed | 14:05 |
| yolanda | for static, i could see successful downloads if wating a long time, but they were hanged for minutes. also cpu was at 100% | 14:05 |
| *** nwkarsten has joined #openstack-infra | 14:06 | |
| *** yamamoto has joined #openstack-infra | 14:08 | |
| fungi | issues on wiki.o.o look like the oom killer taking out memcached | 14:09 |
| *** nelsnelson has quit IRC | 14:09 | |
| *** krtaylor has joined #openstack-infra | 14:09 | |
| *** nelsnelson has joined #openstack-infra | 14:09 | |
| fungi | which again could be a secondary effect of some particular high-volume attacks | 14:09 |
| *** ccrouch has quit IRC | 14:10 | |
| sdague | dims: I would mostly focus on non grenade runs and build up a deprecations hit list | 14:10 |
| AJaeger | ihrachys: seems we haven't set up the periodic jobs the same way. sdague was their a specific reason to not setup periodic jobs with constraints? | 14:10 |
| *** jcoufal has quit IRC | 14:10 | |
| dims | sdague : +1 should i have to find a way to not generate the log for grenade runs or just leave it as is? | 14:11 |
| sdague | AJaeger: nothing intentional, I was just trying to make the stuff that worked, worked | 14:11 |
| AJaeger | ihrachys: Ah, I see your patch... | 14:11 |
| sdague | dims: there is a var in devstack-gate about whether it's grenade | 14:11 |
| sdague | you can switch on it | 14:11 |
| *** jcoufal has joined #openstack-infra | 14:11 | |
| sdague | dims: is there a patch? I only saw the log | 14:11 |
| *** yamamoto has quit IRC | 14:11 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add script to manage power state of ironic servers https://review.openstack.org/284981 | 14:12 |
| *** krtaylor has quit IRC | 14:15 | |
| fungi | if anything's flooding the logs.o.o vhost with requests, it's our logstash workers | 14:15 |
| fungi | i'll check the other vhosts | 14:16 |
| *** nwkarsten has quit IRC | 14:17 | |
| *** nwkarsten has joined #openstack-infra | 14:17 | |
| *** ajmiller has joined #openstack-infra | 14:18 | |
| *** yamamoto has joined #openstack-infra | 14:19 | |
| *** ajmiller_ has joined #openstack-infra | 14:20 | |
| fungi | none of the other vhosts on static.o.o is especially heavily hit with requests either | 14:21 |
| *** nwkarsten has quit IRC | 14:22 | |
| AJaeger | pabelanger, fungi, sdague I did some more changes converting to ubuntu-trusty. Reviews are welcome: https://review.openstack.org/#/q/topic:bindep+status:open | 14:22 |
| *** ajmiller has quit IRC | 14:22 | |
| AJaeger | pabelanger: once you're around, I need your help with missing pieces, please ping me... | 14:23 |
| *** _ari_ has joined #openstack-infra | 14:26 | |
| *** sarob has joined #openstack-infra | 14:26 | |
| *** links has quit IRC | 14:27 | |
| *** krtaylor has joined #openstack-infra | 14:29 | |
| fungi | still not finding any smoking gun for the issues on static and wiki, but i need to step away and get ready for some meetings if anyone wants to continue digging on them | 14:29 |
| *** zhurong has quit IRC | 14:29 | |
| *** bnemec has joined #openstack-infra | 14:29 | |
| *** mriedem has joined #openstack-infra | 14:30 | |
| *** armax has quit IRC | 14:30 | |
| *** _amrith_ is now known as amrith | 14:30 | |
| *** markvoelker has joined #openstack-infra | 14:31 | |
| *** sarob has quit IRC | 14:31 | |
| AJaeger | sdague: ihrachys has a patch up for the periodic jobs, could you review it, please? https://review.openstack.org/289918 | 14:32 |
| *** doug-fish has quit IRC | 14:32 | |
| *** ddieterly has joined #openstack-infra | 14:33 | |
| *** doug-fish has joined #openstack-infra | 14:33 | |
| dims | AJaeger : this is failure a fallout of the bindep problem too? https://jenkins07.openstack.org/job/gate-devstack-gate-unittests/657/consoleFull | 14:34 |
| *** ccrouch has joined #openstack-infra | 14:35 | |
| *** flongo has quit IRC | 14:35 | |
| *** pblaho has joined #openstack-infra | 14:35 | |
| *** doug-fish has quit IRC | 14:37 | |
| *** ajmiller_ has quit IRC | 14:40 | |
| *** jistr|call is now known as jistr | 14:40 | |
| AJaeger | dims, let me check | 14:40 |
| AJaeger | dims: runs on ubuntu-trusty indeed | 14:40 |
| AJaeger | " ImportError: No module named yaml" | 14:41 |
| AJaeger | we might need to add this to the package list. | 14:41 |
| dims | AJaeger : +1 thanks! | 14:41 |
| *** kushal has quit IRC | 14:42 | |
| AJaeger | let me dig a bit deeper | 14:42 |
| *** sbelous_ has joined #openstack-infra | 14:43 | |
| *** nwkarsten has joined #openstack-infra | 14:44 | |
| *** sbelous_ has left #openstack-infra | 14:44 | |
| *** nwkarsten has quit IRC | 14:44 | |
| *** nwkarsten has joined #openstack-infra | 14:45 | |
| *** eharney has joined #openstack-infra | 14:46 | |
| *** doug-fish has joined #openstack-infra | 14:46 | |
| *** dalgaaf has joined #openstack-infra | 14:48 | |
| *** flongo has joined #openstack-infra | 14:48 | |
| openstackgerrit | Deklan Dieterly proposed openstack-infra/project-config: Modify freezer job for tempest tests https://review.openstack.org/289442 | 14:48 |
| *** armax has joined #openstack-infra | 14:49 | |
| AJaeger | dims: which package should provide the yaml python module? | 14:49 |
| *** nwkarsten has quit IRC | 14:49 | |
| *** markvoelker has quit IRC | 14:50 | |
| dims | AJaeger : PyYAML? | 14:50 |
| *** nwkarsten has joined #openstack-infra | 14:50 | |
| dims | AJaeger at least that's what we have in u-c and g-r | 14:50 |
| AJaeger | thanks | 14:51 |
| AJaeger | too blind ;( | 14:51 |
| AJaeger | I wonder what's the right thing to do here, dims, pabelanger, fungi: Should we install pyyaml in bindep-fallback list? | 14:52 |
| AJaeger | Or should we change the gate-devstack-gate-unittests job to work in a venv that installs pyyaml? | 14:52 |
| openstackgerrit | Devananda van der Veen proposed openstack/diskimage-builder: [WIP] switch enable-serial-console element to ttyS2 https://review.openstack.org/289953 | 14:52 |
| *** regXboi has joined #openstack-infra | 14:53 | |
| *** arxcruz has quit IRC | 14:54 | |
| *** nwkarsten has quit IRC | 14:54 | |
| AJaeger | dims, how severe is the breakage? Did we break all of devstack - or even more? | 14:54 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:55 | |
| dims | AJaeger : not that severe, we can't make any change to devstack-gate itself | 14:55 |
| fungi | AJaeger: that job only runs for changes to openstack-infra/devstack-gate itself | 14:55 |
| dims | y | 14:55 |
| fungi | i wonder if the bare-trusty nodes have it installed globally? | 14:55 |
| AJaeger | fungi, seems to be the case - otherwise it wouldn't have worked | 14:56 |
| *** Daisy has joined #openstack-infra | 14:56 | |
| fungi | yeah, i'll jump on one in a sec and check | 14:56 |
| *** yamamoto has quit IRC | 14:56 | |
| *** dmsimard has joined #openstack-infra | 14:56 | |
| AJaeger | we have in system-config modules/openstack_project/manifests/proposal_slave.pp an install for it - but that's all I could find there | 14:56 |
| *** rbrndt_ has joined #openstack-infra | 14:57 | |
| fungi | AJaeger: odds are it was preinstalled by rackspace on their base images as a dependency of some other package | 14:57 |
| *** xarses has quit IRC | 14:58 | |
| *** Daisy has quit IRC | 14:59 | |
| *** Daisy has joined #openstack-infra | 14:59 | |
| pabelanger | AJaeger: So, I don't think we should install python libs with bindep. Since this is devstack, it should be easy to add the missing dependency into the requirements.txt (or whatever devstack uses). | 14:59 |
| pabelanger | I'm checking puppet thick too | 14:59 |
| *** sdake has joined #openstack-infra | 15:00 | |
| *** keedya has joined #openstack-infra | 15:00 | |
| pabelanger | also, morning! | 15:00 |
| *** amitgandhinz has joined #openstack-infra | 15:00 | |
| *** lhinds has quit IRC | 15:01 | |
| AJaeger | morning, pabelanger | 15:01 |
| fungi | i've approved https://review.openstack.org/289704 and will manually start jjb updates again once it merges | 15:01 |
| *** flongo has quit IRC | 15:01 | |
| AJaeger | fungi, https://review.openstack.org/289824 might change a few projects as well - but needs two +2s first... | 15:03 |
| AJaeger | pabelanger, fungi: So, let's ask dims to change the job to run in a venv with requirements? | 15:04 |
| *** denisra_ is now known as denisra | 15:04 | |
| fungi | AJaeger: not sure if you saw https://review.openstack.org/289684 but can you think of any other recent changes where we might have missed the same error? | 15:05 |
| *** kzaitsev_mb has joined #openstack-infra | 15:05 | |
| AJaeger | fungi: saw it - thanks for fixing! | 15:05 |
| *** rfolco has quit IRC | 15:05 | |
| AJaeger | let me double check my changes - hope today's are good | 15:05 |
| *** bdemers has joined #openstack-infra | 15:05 | |
| pabelanger | AJaeger: fungi: I think we could, especially if they already install some python dependencies | 15:06 |
| *** woodster_ has joined #openstack-infra | 15:06 | |
| fungi | now that we're getting into the weeds of less common job definitions, we're going to be finding more one-offs that relied on having sudo access | 15:06 |
| openstackgerrit | Merged openstack-infra/project-config: Remove anonymous mysql users https://review.openstack.org/289704 | 15:07 |
| *** flongo has joined #openstack-infra | 15:07 | |
| *** rossella_s has quit IRC | 15:08 | |
| AJaeger | double checked my open changes - looks fine | 15:08 |
| fungi | awesome--thanks! | 15:08 |
| *** tsg has joined #openstack-infra | 15:09 | |
| *** salv-orl_ has quit IRC | 15:09 | |
| * AJaeger also double check the merged ones - you found all the problems... | 15:10 | |
| AJaeger | fungi, could you review https://review.openstack.org/289918, please? - that one is a regression for neutron switching from constraints jobs... | 15:11 |
| AJaeger | this one also changes a lot of jobs: https://review.openstack.org/289816 | 15:11 |
| openstackgerrit | Jay_Clark proposed openstack-infra/irc-meetings: Updating the meeting chair to Ales Komarek for the 3/8/16 meeting. https://review.openstack.org/289964 | 15:12 |
| *** automagically_ has joined #openstack-infra | 15:12 | |
| *** kzaitsev_mb has quit IRC | 15:13 | |
| fungi | AJaeger: looks like it should do what we want, though i'm not certain we've tried using zuul-cloner in the periodic pipeline so this might be a surprise | 15:13 |
| fungi | on 289918 i mean | 15:13 |
| *** infra-red has joined #openstack-infra | 15:14 | |
| *** infra-red has quit IRC | 15:14 | |
| *** infra-red has joined #openstack-infra | 15:14 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 15:14 |
| jeblair | fungi, AJaeger: zuul-cloner has all of the devstack-gate unit tests and should work anywhere devstack-gate is run, including periodic | 15:15 |
| fungi | ahh, perfect | 15:15 |
| AJaeger | fungi, we had it running in the periodic constraints job - which we just removed. | 15:15 |
| openstackgerrit | Merged openstack-infra/system-config: Add vexxhost mirror to documentation and cacti https://review.openstack.org/287310 | 15:15 |
| AJaeger | so, we need to add it back to the normal one | 15:16 |
| fungi | AJaeger: pabelanger: so the short story as to how we were able to use pyyaml on bare-trusty without installing it is that the python-yaml deb was indeed preinstalled as part of cloud-init's dependency chain | 15:16 |
| AJaeger | Argh ;( | 15:16 |
| AJaeger | So, what's the best approach? Change devstack-gate to use tox - or change job defnition to run in venv? | 15:18 |
| fungi | i'm looking at the d-g unit tests now to see how we can work around this in the least disgusting way | 15:18 |
| AJaeger | thanks, fungi | 15:18 |
| openstackgerrit | Merged openstack-infra/project-config: Prepare constraints file for periodic bitrot jobs https://review.openstack.org/289918 | 15:21 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: [WIP] Run tests in venv https://review.openstack.org/289969 | 15:21 |
| dims | fungi : something like that? ^^ | 15:22 |
| fungi | dims: that's the answer i was arriving at. running it under tox is not a stretch given that one of the test scripts is written in python | 15:23 |
| *** ajmiller has joined #openstack-infra | 15:23 | |
| fungi | so in the end i don't feel back treating it like a python-based job | 15:23 |
| dims | y feels odd | 15:24 |
| fungi | dims: are you also working on the project-config change for that or shall i? | 15:24 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for python-bitrot-jobs https://review.openstack.org/289793 | 15:24 |
| dims | fungi : please go ahead | 15:24 |
| dims | let me fix up the commit message on that one | 15:24 |
| AJaeger | dims, thanks for fixing this! | 15:25 |
| fungi | manual testing of 289969 looks good | 15:25 |
| AJaeger | clarkb, fungi, could you review again https://review.openstack.org/#/c/289404 , please? clarkb gave a -1 for the node change | 15:27 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Only reduce the node vm's when using introspection https://review.openstack.org/289973 | 15:27 |
| *** zul has quit IRC | 15:28 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Run tests in venv https://review.openstack.org/289969 | 15:28 |
| jeblair | AJaeger: why do you want to remove the variable? | 15:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Use tox for devstack-gate unittest job https://review.openstack.org/289975 | 15:29 |
| fungi | dims: ^ | 15:29 |
| jeblair | AJaeger: i noticed you wrote another change that removes it too | 15:29 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/devstack-gate: Generate a log with deprecation warnings https://review.openstack.org/289901 | 15:29 |
| AJaeger | pabelanger: there're a few jobs not converted to ubuntu-trusty that use diskimage-builder. Can you have a look at them and convert them? I wonder whether they need database access or not | 15:29 |
| *** zul has joined #openstack-infra | 15:29 | |
| *** dizquierdo_ has joined #openstack-infra | 15:29 | |
| dims | fungi : that simple? :) nice | 15:29 |
| AJaeger | jeblair: we need to change this since not all jobs are updated and thus we cannot update just projects.yaml - we have to revert these changes afterwards... | 15:30 |
| fungi | jeblair: AJaeger: clarkb: yeah, if we want those jobs to be able to run under ubuntu-trusty for some projects but not for others, we need to track down which ones they are and manually instantiate them in the project.yaml with the correct node preference | 15:30 |
| jeblair | fungi: i don't think you want the depends on. i think we want to land the config change first so that the d-g change can pass. | 15:30 |
| *** armax has quit IRC | 15:30 | |
| fungi | jeblair: oh, good point, it's broken now anyway | 15:31 |
| pabelanger | AJaeger: sure, in general you shouldn't need DB support for DIBs | 15:31 |
| pabelanger | but will look | 15:31 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Use tox for devstack-gate unittest job https://review.openstack.org/289975 | 15:31 |
| jeblair | fungi: i don't think we want them to run on ubuntu-trusty for some and not others at the moment. i think we want to be able to change to ubuntu-xenial project by project | 15:31 |
| openstackgerrit | Ryan Hallisey proposed openstack-infra/tripleo-ci: Allow the continer job to run again https://review.openstack.org/288915 | 15:31 |
| AJaeger | pabelanger: that was my assumption as well but then decided not to convert them all and instead wait for somebody to help with it ;) | 15:31 |
| clarkb | AJaeger: fungi pabelanger I am fine with hardcoding the node if it makes it eaiser to transition, do you want me to remove my -1s? | 15:32 |
| fungi | jeblair: that i agree with, but is there a good way to do that without blowing out the projects.yaml file to 10x its size while we switch on a job-by-job basis? | 15:32 |
| pabelanger | clarkb: AJaeger I was actually about to fix that now | 15:32 |
| *** sbelous_ has joined #openstack-infra | 15:32 | |
| pabelanger | assuming we are talking about 289404 | 15:33 |
| fungi | jeblair: the problem is that right now most of those jobs are inheriting per-project node defaults, and changing the node defaults in those projects means switching over jobs which haven't had their configuration updated to work on the new nodes | 15:33 |
| clarkb | fungi: the extra size being in job specifc node parameters right? | 15:33 |
| *** dizquierdo has quit IRC | 15:33 | |
| jeblair | fungi: maybe not, but it would be good to make sure we agree on a plan, rather than just ignoring clarkb's -1. | 15:33 |
| jbernard_ | clarkb: heya, any update on the ceph package mirror for the plugin job? | 15:33 |
| AJaeger | fungi, or add for each job a node: ubuntu-trusty line in projects.yaml... | 15:33 |
| *** Daisy has quit IRC | 15:34 | |
| clarkb | jbernard_: I havent started on it yet, focus has been adding cloud resources and reconfiguring jobs to better utilize cloud resources | 15:34 |
| fungi | AJaeger: that's what i meant by expanding the projects.yaml file to 10x its current size | 15:34 |
| AJaeger | what about adding comments: # TODO() change ubuntu-trusty back to {node} later | 15:34 |
| jbernard_ | clarkb: cool, if there's anything i can do to help out, let me know | 15:34 |
| AJaeger | fungi: ok | 15:34 |
| jeblair | AJaeger: that would at least help us keep track of it. ++ | 15:34 |
| jbernard_ | sdague: when we swith the plugin to vote, should i remove the '-nv' from the name? | 15:34 |
| AJaeger | jeblair: then let's do this. pabelanger can you update your change? | 15:35 |
| clarkb | jbernard_: not unless you want to write puppet. I think we need a slight reorg in order to get multple apt repos going | 15:35 |
| * AJaeger will update open changes. | 15:35 | |
| sdague | jbernard_: yes | 15:35 |
| jbernard_ | sdague: ok, ill get that in the queue | 15:35 |
| AJaeger | pabelanger: I'll update https://etherpad.openstack.org/p/bindep so that we use the same wording... | 15:35 |
| jbernard_ | clarkb: okie, ill let you have at it ;) | 15:35 |
| andrearosa | I have a commit with the DOCIMPACT flag which generates a bug. I work on it adding a comment with details (hopefully) inforamtion, do I need to re-assign the bug to someone else? ref https://bugs.launchpad.net/nova/+bug/1550525 | 15:35 |
| openstack | Launchpad bug 1550525 in OpenStack Compute (nova) " Abort an ongoing live migration" [Undecided,Confirmed] - Assigned to Andrea Rosa (andrea-rosa-m) | 15:35 |
| pabelanger | AJaeger: sure | 15:36 |
| *** xarses has joined #openstack-infra | 15:36 | |
| fungi | andrearosa: normally you would let the nova bug team triage it | 15:36 |
| fungi | andrearosa: you can probably unassign it from yourself if you don't intend to work on it | 15:36 |
| andrearosa | fungi: ok, tahnks | 15:36 |
| jeblair | clarkb: i think we accommodate multiple reprerpros | 15:38 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 15:38 |
| *** zeih has joined #openstack-infra | 15:38 | |
| jeblair | clarkb: remember, cody-somerville pointed out that debian and ubuntu have the same package names for different packages, so mordred refactors to let us support both debian and ubuntu? | 15:38 |
| AJaeger | jeblair: better ^ | 15:38 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes https://review.openstack.org/252041 | 15:38 |
| jeblair | refactored | 15:39 |
| clarkb | jeblair: the refactor was insufficient because puppet. It needs to use a defined resource type not a class | 15:39 |
| *** ff has joined #openstack-infra | 15:39 | |
| jeblair | clarkb: ah ok | 15:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch gate-{name}-linters to bindep https://review.openstack.org/289404 | 15:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert gate-{name}-puppet-syntax to ubuntu-trusty https://review.openstack.org/289036 | 15:40 |
| *** armax has joined #openstack-infra | 15:40 | |
| jeblair | wait | 15:40 |
| jeblair | you can't have more than one instance of a *class* in puppet? | 15:40 |
| pabelanger | AJaeger: clarkb: Going to do 1 job at a time, since we are touching projects.yaml now too ^ | 15:40 |
| *** claudiub has quit IRC | 15:40 | |
| pabelanger | jeblair: usually not | 15:40 |
| pabelanger | class foo with multiple import foo work | 15:41 |
| pabelanger | or include foo | 15:41 |
| jeblair | that's like, computer science 201. | 15:41 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert linters to use ubuntu-trusty https://review.openstack.org/289832 | 15:41 |
| *** piet has joined #openstack-infra | 15:41 | |
| AJaeger | pabelanger: duplicate with ^ | 15:41 |
| jeblair | i mean, it's right there in the word -- it's the *very definition* of the word "class" | 15:41 |
| rcarrillocruz | :D | 15:41 |
| pabelanger | AJaeger: H ha | 15:41 |
| fungi | clarkb: jeblair: on the topic of the trusty->xenial transition, i know we discussed that using a zuul parameter function to choose a node label depending on a change's branch name was ugly... but was that because we had both bare-.* and devstack-.* nodes? does a similar implementation become much simpler when bare-trusty and devstack-trusty jobs have all moved to ubuntu-trusty? | 15:42 |
| *** ashleighfarnham has joined #openstack-infra | 15:42 | |
| AJaeger | pabelanger: please check the list of open changes... | 15:42 |
| *** sbelous_ has quit IRC | 15:42 | |
| AJaeger | pabelanger: I'm not planning to send new ones, just update existing ones for now | 15:42 |
| jeblair | fungi: i thought that's the main reason we wrote parameter functions... | 15:42 |
| clarkb | it will still be an issue | 15:43 |
| openstackgerrit | Moshe Levi proposed openstack/diskimage-builder: Add lshw package to ironic-agent https://review.openstack.org/289233 | 15:43 |
| pabelanger | AJaeger: I think you could safely do: https://review.openstack.org/#/c/289404/2/jenkins/jobs/projects.yaml for linters | 15:43 |
| clarkb | the problem is how to get a job registered with both labels and only run the job on the one you want | 15:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: horizon: Use ubuntu-trusty https://review.openstack.org/289819 | 15:43 |
| pabelanger | since we now have bindep working on all 3 dibs | 15:43 |
| fungi | jeblair: yeah, it just became a maze of twisty passages soon after but i think that's because we were trying to split jobs by branch and node variant | 15:43 |
| pabelanger | AJaeger: I can abandon my change | 15:44 |
| *** gnuoy_ has joined #openstack-infra | 15:44 | |
| *** gnuoy_ has quit IRC | 15:44 | |
| * AJaeger updated all his changes to include the TODO | 15:44 | |
| fungi | seems like a parameter function after the ubuntu-trusty transition is complete can just map original node name to new node name depending on the branch | 15:44 |
| *** esikachev has joined #openstack-infra | 15:44 | |
| fungi | rather than doing convoluted matching on patterns within job names | 15:44 |
| jeblair | fungi: "original node name" ? | 15:45 |
| fungi | oh, right, we only know the "original" node name if it's only registered for one label | 15:45 |
| fungi | hrm | 15:45 |
| jeblair | fungi: zuul doesn't know the "original" | 15:45 |
| fungi | yeah, zuul knows that it either changes the label or doesn't change the label in the request | 15:46 |
| AJaeger | pabelanger: Oh, your change was older ;( Missed that we both touched it - I was confused and apologize. I should have abandoned mine... | 15:46 |
| jeblair | i'm confused by the conversation because i think this is a solved problem... so let me back up and at the risk of being redundant, state how we solved this last time, then we can figure out why we think this is complex :) | 15:46 |
| *** pblaho has quit IRC | 15:46 | |
| jeblair | in jjb, we set "node" to "ubuntu-trusty || ubuntu-xenial" for any jobs that may need to run on different nodes for different branches. this gets them registered where they should be. | 15:47 |
| pabelanger | AJaeger: All good, already nuked. | 15:47 |
| jeblair | in zuul, we have a paremeter function to select one or the other of those based on conditions (like branch name, and also job name due to things like stable periodic jobs) | 15:48 |
| AJaeger | pabelanger: regarding 289832 - why don't we need the node change? I'm confused... | 15:48 |
| pabelanger | AJaeger: see 289404 for my thinking | 15:49 |
| fungi | we had a discussion around the time when we finally ripped out all the icehouse branch mappings for precise that it was a source of frustration because every time someone added a job that wasn't going to run on bare-trusty they would invariably forget to try to update the parameter function's massive list of regular expressions, or would update it in some nuanced way that caused a bunch of jobs to | 15:49 |
| fungi | suddenly start running on the wrong node types | 15:49 |
| pabelanger | AJaeger: It was a result of clarkb comment about allowing jobs to override it. | 15:49 |
| *** esikachev has quit IRC | 15:49 | |
| pabelanger | AJaeger: if we want to hardcode to ubuntu-trusty, I am fine with that | 15:49 |
| *** jlanoux has quit IRC | 15:49 | |
| openstackgerrit | Merged openstack-infra/project-config: Use tox for devstack-gate unittest job https://review.openstack.org/289975 | 15:49 |
| AJaeger | let's hardcode for now to ubuntu-trusty - and then change again... | 15:50 |
| fungi | and so we were frequently hunting down NOT_REGISTERED errors in zuul's gearman registrations and debug logs to figure out what was wrong | 15:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert linters to use ubuntu-trusty https://review.openstack.org/289832 | 15:50 |
| AJaeger | thanks for catching the wrong order, pabelanger | 15:50 |
| jeblair | pabelanger, AJaeger, fungi: yeah, this was why i was trying to get us to have a conversation and agree on an approach for the node changes :) | 15:50 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 15:50 |
| *** nwkarsten has joined #openstack-infra | 15:51 | |
| *** oanson has quit IRC | 15:51 | |
| pabelanger | jeblair: FWIW: I'm on board with hive mind. No real objections ATM | 15:51 |
| fungi | jeblair: anyway, i was mostly trying to figure out if taking the same approach again with the switch to xenial will still become as frustrating, or possibly be simpler because we won't have multiple trusty and xenial node types | 15:52 |
| *** abregman has quit IRC | 15:52 | |
| sdake | AJaeger any chance you can review https://review.openstack.org/#/c/287957/3 | 15:52 |
| *** links has joined #openstack-infra | 15:52 | |
| clarkb | fungi: I don't think the problem has changed due to having a single xenial and single trusty image. Since most of the selection is based on branch | 15:52 |
| *** esker has joined #openstack-infra | 15:53 | |
| AJaeger | sdake: 4 jobs for one distro? That's a lot IMHO. But syntactically fine, so I'll +2 | 15:53 |
| jeblair | fungi: maybe simpler? but still fraught with peril, so worth considering very carefully as we do it. if we can make a logic that says very simply "xenial is now the default, these are the constrained set of exceptions" it may not get too far out of hand. | 15:53 |
| fungi | clarkb: the "problem" meaning the need for the migration, or the need for maintaining a massive list of pattern matches on various job names so that things targeting centos or fedora don't wind up running on ubuntu? | 15:54 |
| *** akshai has joined #openstack-infra | 15:54 | |
| sdake | AJaeger ya we have 4 jobs for ever ydistro - centos and ubuntu and now oracle linux | 15:54 |
| sdake | we are adding 2 more jobs per distro as well shortly | 15:54 |
| clarkb | fungi: the need for parameter function label selection, it is possible we can get away with massive pattern matches if we limit the functions only to jobs that run against multiple branches | 15:55 |
| sdake | iirc fungi said this is ok, as long as the gates dont take a long time to run | 15:55 |
| clarkb | fungi: the trade off there is it becomes easy to run the job on the wrong host rather than getting a NOT_REGISTERED | 15:55 |
| AJaeger | jeblair: I'll have to leave now, will be back for infra meeting. Thanks for starting this discussion - please write up the way forward somewhere as reference. | 15:55 |
| *** jpr has joined #openstack-infra | 15:55 | |
| jeblair | AJaeger: i won't be writing it up, but i agree someone should :) | 15:56 |
| *** nwkarsten has quit IRC | 15:56 | |
| *** sc68cal has joined #openstack-infra | 15:56 | |
| clarkb | I think our options are to do as we did last time and be explicit which results in NOT_REGISTERED jobs or use a subset list of jobs that need selection and anything that isn't in that list risks running on the wrong label if it has >1 label | 15:57 |
| jeblair | clarkb: right, i think that's the situation we started out with until we started adding even more node types to the switches | 15:57 |
| *** yamamoto has joined #openstack-infra | 15:57 | |
| jeblair | er, that was for clarkb_msg[-2] | 15:57 |
| fungi | as far as parameterization, why does doing node: ubuntu-trusty instead of node: '{node}' make the transition to 'ubuntu-trusty || ubuntu-xenial' more complicated? | 15:57 |
| clarkb | jeblair: ya I think most of the frustration came from jobs that wanted to run on centos7 or fedora 21 but ran on ubuntu because the default was run on ubuntu | 15:57 |
| *** sbelous_ has joined #openstack-infra | 15:58 | |
| jeblair | fungi: because the jobs have to be registered in jenkins as being able to run on both node types | 15:58 |
| fungi | jeblair: perhaps i'm not being clear. why does the sed -i get more complicated regardless of where the ubuntu-trusty is hard-coded (in projects.yaml or in individual jobs and job-templates)? | 15:59 |
| *** automagically_ has left #openstack-infra | 15:59 | |
| clarkb | fungi: I think the reason is not all templates want to run on that label | 15:59 |
| *** sbelous_ has quit IRC | 15:59 | |
| clarkb | fungi: some of them get applied to centos and fedora too | 15:59 |
| *** daemontool_ has joined #openstack-infra | 15:59 | |
| clarkb | granted the old way of selecting fedora or centos was to use the job name so we would be broken in the old scheme regardless | 16:00 |
| *** harlowja_at_home has joined #openstack-infra | 16:00 | |
| *** daemontool has quit IRC | 16:01 | |
| *** harlowja_at_home has quit IRC | 16:02 | |
| *** harlowja_at_home has joined #openstack-infra | 16:03 | |
| *** bdemers has quit IRC | 16:04 | |
| *** Qiming has quit IRC | 16:04 | |
| *** sdake_ has joined #openstack-infra | 16:05 | |
| *** bdemers has joined #openstack-infra | 16:05 | |
| *** sdake has quit IRC | 16:05 | |
| *** zul has quit IRC | 16:05 | |
| *** mugsie has quit IRC | 16:06 | |
| *** zul has joined #openstack-infra | 16:06 | |
| rcarrillocruz | folks | 16:06 |
| *** mugsie has joined #openstack-infra | 16:07 | |
| rcarrillocruz | for the networks you use in the various openstack infra cloud providers | 16:07 |
| rcarrillocruz | do you shade the network between openstackjenkins and openstackci projects | 16:07 |
| rcarrillocruz | or each get one | 16:07 |
| rcarrillocruz | clarkb: ? | 16:07 |
| rcarrillocruz | erm | 16:07 |
| rcarrillocruz | s/shade/share | 16:07 |
| rcarrillocruz | too much shade coding lately :-) | 16:08 |
| *** IlyaG has joined #openstack-infra | 16:08 | |
| *** sridhar_ram has joined #openstack-infra | 16:09 | |
| *** yamamoto has quit IRC | 16:10 | |
| fungi | rcarrillocruz: they communicate "through the internet" | 16:10 |
| jeblair | clarkb, fungi: the node parameterization was mostly so that we could set per-project default nodes for unit tests, but maybe it is simpler to just drop that now in favor of focusing on the branch selection logic | 16:11 |
| fungi | rcarrillocruz: no special connecting network between them | 16:11 |
| *** kzaitsev_mb has joined #openstack-infra | 16:11 | |
| rcarrillocruz | i mean, the neutron networks | 16:11 |
| *** rfolco has joined #openstack-infra | 16:11 | |
| rcarrillocruz | i assume you just get them created for you | 16:11 |
| rcarrillocruz | i.e. you don't have cloud admin on most, so you never have to care about a shared network or not? | 16:11 |
| fungi | rcarrillocruz: in most cases we use whatever's there. i think for hpcloud, infra-cloud and osic we had to explicitly add them | 16:12 |
| jeblair | also bluebox | 16:12 |
| rcarrillocruz | i'm modeling the ansible cloud launcher role, and just noticed the 'shared' param network creation | 16:12 |
| *** salv-orlando has joined #openstack-infra | 16:12 | |
| fungi | ahh, yep | 16:12 |
| fungi | so maybe "most" was incorrect | 16:12 |
| *** zz_dimtruck is now known as dimtruck | 16:12 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Add NFS devstack plugin https://review.openstack.org/289589 | 16:12 |
| rcarrillocruz | jeblair: do you know off the top of your head if you had both nets shared between the openstackjenkins and openstackci tenants? | 16:12 |
| rcarrillocruz | i'd say not, but if there's a case where it's shared i'd need to take into account that in the role... | 16:13 |
| jeblair | rcarrillocruz: no, like fungi said, we just want our hosts to be on the internet. our tenants can be completely separate | 16:14 |
| rcarrillocruz | alright, thx | 16:14 |
| *** scheuran has quit IRC | 16:15 | |
| *** esker has quit IRC | 16:15 | |
| *** esikachev has joined #openstack-infra | 16:16 | |
| *** flongo has quit IRC | 16:17 | |
| smcginnis | Anyone here know where http://www.openstack.org/marketplace/drivers/ is generated from? | 16:17 |
| smcginnis | It has some duplicate but slightly different and out of date information when looking at the Cinder drivers. | 16:17 |
| *** harlowja_at_home has quit IRC | 16:18 | |
| *** jlanoux has joined #openstack-infra | 16:19 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:19 |
| *** pblaho has joined #openstack-infra | 16:19 | |
| *** esker has joined #openstack-infra | 16:19 | |
| *** salv-orlando has quit IRC | 16:19 | |
| *** nwkarsten has joined #openstack-infra | 16:19 | |
| *** mtanino has joined #openstack-infra | 16:20 | |
| *** ildikov has quit IRC | 16:20 | |
| fungi | smcginnis: probably https://github.com/OpenStackweb/openstack-org but i don't really know, since the www.openstack.org site is not community-maintained (and a lot of the content is in a cms database, not in git) | 16:20 |
| fungi | smcginnis: its caretakers are a group of web developers employed/contracted by the foundation | 16:20 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:20 |
| smcginnis | fungi: OK, cool. Thanks for the pointer. | 16:21 |
| clarkb | there is a bug tracker you can file bugs against | 16:21 |
| clarkb | let me get a link | 16:21 |
| fungi | smcginnis: bugs for it go to https://bugs.launchpad.net/openstack-org/ | 16:21 |
| *** jaosorior has quit IRC | 16:21 | |
| clarkb | there it | 16:21 |
| clarkb | is | 16:21 |
| smcginnis | clarkb, fungi: Thanks! | 16:21 |
| *** akshai has quit IRC | 16:21 | |
| clarkb | jeblair: ya with the incrase in distros and releases the idea of a default may not be as useful anymore | 16:22 |
| *** flongo has joined #openstack-infra | 16:23 | |
| *** thorst_ is now known as thorst_afk | 16:23 | |
| *** akshai has joined #openstack-infra | 16:23 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 16:23 |
| *** Swami has joined #openstack-infra | 16:25 | |
| jeblair | clarkb, fungi: should we just go ahead and yank the per-project node templating then? | 16:26 |
| fungi | jeblair: that was where i was leaning, but wanted to have a conversation about it first (as to whether we do it only temporarily for this transition or indefinitely until we discover that we need to reintroduce it for some reason). it does bring additional complexity for (currently) minimal benefit | 16:27 |
| *** david-lyle_ is now known as david-lyle | 16:28 | |
| *** akshai has quit IRC | 16:28 | |
| clarkb | I am on board with removing it now to aid in transition and adding it back if we need to later | 16:28 |
| *** pblaho has quit IRC | 16:29 | |
| *** zhurong has joined #openstack-infra | 16:29 | |
| *** nmagnezi has quit IRC | 16:29 | |
| *** akshai has joined #openstack-infra | 16:30 | |
| *** [1]Thelo has joined #openstack-infra | 16:30 | |
| *** annegentle has quit IRC | 16:30 | |
| *** zeih has quit IRC | 16:30 | |
| *** pblaho has joined #openstack-infra | 16:31 | |
| *** annegentle has joined #openstack-infra | 16:31 | |
| *** infra-re_ has joined #openstack-infra | 16:31 | |
| *** dmsimard has quit IRC | 16:31 | |
| jeblair | sounds like a plan. | 16:32 |
| *** ashleighfarnham has quit IRC | 16:32 | |
| *** Thelo has quit IRC | 16:32 | |
| *** [1]Thelo is now known as Thelo | 16:32 | |
| *** infra-r__ has joined #openstack-infra | 16:33 | |
| *** flongo has quit IRC | 16:34 | |
| *** gyee has joined #openstack-infra | 16:34 | |
| *** infra-red has quit IRC | 16:34 | |
| *** sdake_ is now known as sdake | 16:36 | |
| *** bdemers has quit IRC | 16:37 | |
| *** infra-re_ has quit IRC | 16:37 | |
| *** dguitarbite has quit IRC | 16:37 | |
| pleia2 | good morning | 16:38 |
| clarkb | good morning | 16:38 |
| *** bdemers has joined #openstack-infra | 16:39 | |
| clarkb | mnaser: looks like there is a group of recent node launch errors. I can dig up details if you are interested | 16:40 |
| mnaser | clarkb: if you could be kind enough | 16:41 |
| sdague | http://logs.openstack.org/36/289936/1/check/gate-devstack-unit-tests/5b94ee8/console.html - did that change node types? | 16:41 |
| *** MarkAtwood has joined #openstack-infra | 16:42 | |
| sdague | and if so, what mechanism should we use to install correct packages? | 16:42 |
| *** acabot has quit IRC | 16:43 | |
| sdague | oh, never mind, something else weird is happening on sorting | 16:43 |
| *** acabot has joined #openstack-infra | 16:43 | |
| *** hashar has quit IRC | 16:44 | |
| clarkb | mnaser: looks like most of them are timeout waiting for ssh access, I will get a representative traceback for you + server uuids | 16:44 |
| *** arxcruz has joined #openstack-infra | 16:46 | |
| *** tpsilva has quit IRC | 16:47 | |
| *** bpokorny has joined #openstack-infra | 16:47 | |
| *** vincentfrancoise has joined #openstack-infra | 16:47 | |
| *** ddieterly is now known as ddieterly[away] | 16:48 | |
| *** mikelk has quit IRC | 16:48 | |
| *** yamahata has quit IRC | 16:48 | |
| clarkb | mnaser: http://paste.openstack.org/show/489706/ that isn't a complete list but should be representative | 16:50 |
| mnaser | checking | 16:50 |
| *** ddieterly[away] is now known as ddieterly | 16:51 | |
| fungi | sdague: granted, that job _did_ change node types recently. if you think this is a behavior change because of something being subtly different about the worker, i'm happy to help dig in | 16:51 |
| sdague | fungi: well, I can't produce the fail locally | 16:51 |
| sdague | let me see if it's across the boardc | 16:52 |
| sdague | it looks like a change in how the sorting works | 16:52 |
| fungi | sdague: for example, dims just discovered that devstack-gate's unit tests were accidentally depending on the distro-packaged python-yaml which was present as an indirect dependency of cloud-init | 16:52 |
| sdague | https://review.openstack.org/#/c/289965/ | 16:52 |
| sdague | yeh, something is up | 16:52 |
| *** sc68cal has quit IRC | 16:53 | |
| fungi | sdague: it's possible the collation default in the environment was being "conveniently" set to a more user-friendly non-default value in rackspace's custom images | 16:53 |
| *** sshnaidm has quit IRC | 16:53 | |
| sdague | fungi: or that dib doesn't set it up like distros do | 16:53 |
| sdague | ./tests/test_package_ordering.sh in devstack works fine locally for me | 16:54 |
| sdague | but fails in the gate | 16:54 |
| *** ff has quit IRC | 16:55 | |
| sdague | on both 14.04 and 15.10 | 16:55 |
| fungi | sdague: i take it test_package_ordering.sh doesn't output anywhere what's actually misordered? | 16:55 |
| sdague | it sort of does | 16:55 |
| fungi | in a different file? | 16:55 |
| sdague | http://logs.openstack.org/65/289965/1/check/gate-devstack-unit-tests/1d99a0f/console.html#_2016-03-08_15_17_06_930 | 16:55 |
| sdague | it's a little obtuse | 16:55 |
| sdague | but looks like differences in casing on sort | 16:56 |
| *** ff has joined #openstack-infra | 16:56 | |
| fungi | yeah, i bet it's using C.UTF-8 | 16:56 |
| mnaser | clarkb: looking like a compute node gone nuts, should be okay but im seeing whats going on | 16:57 |
| fungi | which is my preference for server environments too, but lots of desktopy distros like to set human case ordering | 16:57 |
| mnaser | scheduler kept sending new servers to that specific one | 16:57 |
| clarkb | mnaser: ok, its not a problem for us but happy to help you out by passing info along | 16:57 |
| fungi | the test should probably set the collation it expects before comparing | 16:57 |
| mnaser | yeah it syncs up with an issue we noticed, id like to see how we can avoid this (maybe update our scheduler code) | 16:58 |
| mnaser | thank you for bringing this up | 16:58 |
| *** ff has quit IRC | 16:58 | |
| sdague | fungi: ok | 16:58 |
| sdague | fungi: remember which var to set there? | 16:58 |
| fungi | sdague: because i have a feeling that test would fail locally for me as my default environment uses LC_COLLATE=C.UTF-8 | 16:58 |
| Clint | i'd do LC_ALL for the sort line | 16:59 |
| fungi | Clint: sdague: yeah, chances are it will pass if you export LC_ALL=en_US.UTF-8 there | 16:59 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Add a non-voting iPXE job to ironic-inspector https://review.openstack.org/290016 | 17:00 |
| *** Jeffrey4l has joined #openstack-infra | 17:01 | |
| fungi | if so, definitely as case of poor test isolation from system defaults | 17:01 |
| sdague | cool, thanks | 17:01 |
| sdague | fungi: sure, though when it works in the test system for 2 years, and everyone's personal dev environment, you don't really ponder it | 17:01 |
| mriedem | mtreinish: sdague: fyi https://review.openstack.org/#/c/290021/ | 17:02 |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci: Use IPv6 on the ceph gate job https://review.openstack.org/289445 | 17:03 |
| *** esikachev has quit IRC | 17:03 | |
| sdague | mriedem: isn't that an indication that it should have been a major release? | 17:03 |
| *** mrmartin has quit IRC | 17:03 | |
| *** mrmartin has joined #openstack-infra | 17:04 | |
| mriedem | sdague: sure, but things would still be busted b/c trove unit tests aren't using constraints | 17:04 |
| sdague | mriedem: sure | 17:04 |
| sdague | I just wonder if we should force a major there as well | 17:05 |
| mriedem | so, i could revert the troveclient change and blacklist 2.1.0 | 17:05 |
| *** apoorvad has joined #openstack-infra | 17:05 | |
| mriedem | it's getting pretty late for that especially since client libs are frozen for mitaka | 17:05 |
| *** mrmartin has quit IRC | 17:05 | |
| sdague | mriedem: sure | 17:06 |
| mriedem | stable/liberty upper-constraints for troveclient wouldn't use 2.1.0, but that's assuming deployers are using it | 17:06 |
| sdague | https://review.openstack.org/#/c/290021 is a good instance of why I think semver is not a thing in the real world :) | 17:06 |
| mriedem | basically, i'm tired of the trove team not owning this so i'm using a hammer | 17:06 |
| sdague | I'm +2 on that | 17:06 |
| *** ildikov has joined #openstack-infra | 17:06 | |
| *** cloudtrainme has joined #openstack-infra | 17:07 | |
| *** matrohon has quit IRC | 17:08 | |
| anteaya | I find a hammer sometimes a useful tool in finding folks to take responsibility for their actions | 17:08 |
| clarkb | I am somewhat surprised that en_US vs C on package names sort different | 17:10 |
| clarkb | isn't that basically alphanum in both cases? | 17:10 |
| *** kzaitsev_mb has quit IRC | 17:11 | |
| *** yamahata has joined #openstack-infra | 17:12 | |
| tristanC | clarkb: iirc, C will sort uppercase before lowercase, see LC_COLLATE | 17:13 |
| *** pradk_ has quit IRC | 17:13 | |
| anteaya | infra-root could someone please re-enable the following four gerrit accounts: 14208, 14212, 14213, 15443 http://paste.openstack.org/show/489263/ | 17:13 |
| fungi | clarkb: C is traditional sort order (ordered by the codepoint integer value basically), while newer locale-based sorts try to treat "a" as sorting before "B" (for example) | 17:13 |
| mriedem | sdague: http://lists.openstack.org/pipermail/openstack-dev/2016-March/088696.html in case you want to comment | 17:14 |
| pleia2 | anteaya: looking now | 17:14 |
| anteaya | pleia2: thank you | 17:14 |
| clarkb | tristanC: fungi gotcha | 17:14 |
| *** moravec has quit IRC | 17:14 | |
| *** thorst_afk is now known as thorst_ | 17:14 | |
| *** moravec has joined #openstack-infra | 17:14 | |
| fungi | in the days before unicode, you could have considered C sorts as "ANSI order" | 17:14 |
| fungi | er, i mean "ASCII order" | 17:14 |
| clarkb | and ls confirms | 17:14 |
| fungi | yeah, personally i find non-c sorting annoying, but probably just because i spent decades expecting ascii ordered sorting | 17:15 |
| Sam-I-Am | AJaeger: you around? | 17:15 |
| pleia2 | anteaya: all marked as active | 17:16 |
| anteaya | pleia2: thank you | 17:16 |
| pleia2 | anteaya: you're welcome | 17:16 |
| *** daemontool_ has quit IRC | 17:16 | |
| anteaya | I'm formuating a reply to http://lists.openstack.org/pipermail/third-party-announce/2016-March/000297.html | 17:18 |
| anteaya | but as of right now I don't know the correct answer to the question about clearing zuul caches | 17:18 |
| anteaya | Also someone told me that I might need to clean local zuul cache from time to time, is that correct? | 17:19 |
| anteaya | how should I respond that that question? | 17:19 |
| clarkb | anteaya: check the zuul merger logs | 17:19 |
| anteaya | sorry that isn't enough for me to create a sentence | 17:19 |
| anteaya | what indicator does the zuul merger logs show me as a reason to clear or not clear zuul caches | 17:19 |
| clarkb | anteaya: when zuul reports that merging has failed the reason for this should be recorded in the zuul-merger process' logs | 17:19 |
| clarkb | anteaya: I don't think it will tell you whether or not to clear caches, it will tell you why merges are failing | 17:20 |
| anteaya | oh okay | 17:20 |
| anteaya | so instead of clearing caches, check the zuul merger logs | 17:20 |
| anteaya | thanks I will offer that | 17:20 |
| anteaya | unless you have a minute to draft a reply? | 17:20 |
| clarkb | I can do it in a few minutes if you prefer | 17:21 |
| *** ccrouch has quit IRC | 17:21 | |
| anteaya | that would be awesome | 17:21 |
| anteaya | thanks so much | 17:21 |
| anteaya | because I really don't know what I'm talking about here | 17:21 |
| *** daemontool has joined #openstack-infra | 17:21 | |
| *** ociuhandu has quit IRC | 17:22 | |
| jeblair | can we keep discussion off of the announce list? | 17:22 |
| *** jamesmcarthur has quit IRC | 17:22 | |
| anteaya | jeblair: we can move it to infra if you prefer | 17:22 |
| jeblair | we probably shouldn't even let people who aren't us post to it.... | 17:22 |
| *** apoorvad has quit IRC | 17:22 | |
| jeblair | (or at least moderate) | 17:23 |
| anteaya | I think this person is a third party ci operator: http://lists.openstack.org/pipermail/third-party-announce/2016-March/000297.html | 17:23 |
| *** annegentle has quit IRC | 17:23 | |
| *** apoorvad has joined #openstack-infra | 17:23 | |
| jeblair | right, but the list was so that we would have a way to get in contact with all of the third-party ci operators with important announcements | 17:23 |
| anteaya | I felt it was good that other third party ci operators were bothering to fix their systems based on other's mistakes | 17:23 |
| *** xyang1 has joined #openstack-infra | 17:23 | |
| *** salv-orlando has joined #openstack-infra | 17:23 | |
| jeblair | yeah, i'm not saying that's a bad thing | 17:23 |
| *** annegentle has joined #openstack-infra | 17:23 | |
| jeblair | i'm saying that there should be some other place for that | 17:23 |
| *** christiana_ has quit IRC | 17:24 | |
| jeblair | because we're *requiring* people to subscribe to that list | 17:24 |
| anteaya | I can reply to this post moving it to infra | 17:24 |
| jeblair | so we have a duty to keep it as low-traffic as possible | 17:24 |
| anteaya | and then clarkb can reply on the infra thread | 17:24 |
| anteaya | clarkb: does that work for you? | 17:24 |
| clarkb | anteaya: yes | 17:24 |
| anteaya | jeblair: you make a good point, sorry about that | 17:24 |
| anteaya | it is being successful being low traffic | 17:24 |
| jeblair | anteaya: np... and cool -- best keep it that way :) | 17:25 |
| *** zhurong has quit IRC | 17:26 | |
| *** d0ugal has quit IRC | 17:26 | |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| anteaya | yup, thanks | 17:27 |
| *** tpsilva has joined #openstack-infra | 17:27 | |
| *** dtantsur is now known as dtantsur|afk | 17:27 | |
| anteaya | clarkb: http://lists.openstack.org/pipermail/openstack-infra/2016-March/003972.html | 17:28 |
| *** annegentle has quit IRC | 17:28 | |
| *** Guest92187 has quit IRC | 17:29 | |
| anteaya | and the post to announce telling them I've moved the thread for discussion: http://lists.openstack.org/pipermail/third-party-announce/2016-March/000300.html | 17:29 |
| *** harlowja has joined #openstack-infra | 17:29 | |
| *** ihrachys has quit IRC | 17:30 | |
| *** ihrachys has joined #openstack-infra | 17:31 | |
| *** ihrachys has quit IRC | 17:31 | |
| *** sfinucan has quit IRC | 17:32 | |
| *** piet has quit IRC | 17:33 | |
| *** ccrouch has joined #openstack-infra | 17:33 | |
| *** rcernin has quit IRC | 17:34 | |
| *** ifarkas has quit IRC | 17:36 | |
| *** akshai has quit IRC | 17:39 | |
| *** esker has quit IRC | 17:40 | |
| *** claudiub has joined #openstack-infra | 17:41 | |
| *** davideagnello has joined #openstack-infra | 17:41 | |
| *** jistr has quit IRC | 17:43 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 17:44 |
| *** jlanoux has quit IRC | 17:44 | |
| AJaeger | Sam-I-Am: I'm here now.... | 17:44 |
| pabelanger | nibalizer: so I've been looking at RST publishing websites, and stumbled across http://blog.getpelican.com/ | 17:44 |
| pabelanger | nibalizer: it takes RST (from git) and converts it into static content | 17:45 |
| pabelanger | nibalizer: going to try using github pages to test it out | 17:45 |
| *** Daisy has joined #openstack-infra | 17:45 | |
| *** lucasagomes is now known as lucas-dinner | 17:46 | |
| AJaeger | jeblair, fungi, clarkb: Reading backscroll about the node change: Should we move it out of projects.yaml into individual jobs in all cases? Or just as neccessary? Should we add comments about the change as in https://review.openstack.org/#/c/289832/3/jenkins/jobs/misc.yaml | 17:47 |
| *** exploreshaifali has joined #openstack-infra | 17:47 | |
| fungi | AJaeger: i think moving it out of projects.yaml and not leaving comments behind makes sense. we do however need to be careful to look out for any projects which set something other than node: bare-trusty | 17:48 |
| AJaeger | dims: I think we need manual runs of JJB to update the Jenkins instances before you can recheck the devstack-gate change | 17:48 |
| AJaeger | fungi, the top-level is *always* bare-trusty, just grepped for it. It's only overwritten for individual jobs | 17:49 |
| *** Daisy has quit IRC | 17:50 | |
| AJaeger | fungi, then let me remove my #TODO comments again... | 17:50 |
| *** claudiub has quit IRC | 17:50 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 17:50 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 17:50 |
| dims | AJaeger : ah, it's been an hour since it merged, so i assumed | 17:50 |
| *** ociuhandu has joined #openstack-infra | 17:51 | |
| AJaeger | dims: we have another change in front of it that needs the manual JJB run... | 17:51 |
| *** zz_ja is now known as zz_zz_ja | 17:51 | |
| *** ihrachys has joined #openstack-infra | 17:51 | |
| Sam-I-Am | AJaeger: neutron is back to liberty - http://docs.openstack.org/releasenotes/neutron/unreleased.html | 17:53 |
| *** nwkarsten has quit IRC | 17:53 | |
| *** salv-orl_ has joined #openstack-infra | 17:53 | |
| *** akshai has joined #openstack-infra | 17:53 | |
| *** nwkarsten has joined #openstack-infra | 17:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert linters+bashate to use ubuntu-trusty https://review.openstack.org/289832 | 17:53 |
| AJaeger | dhellmann: can you check Sam-I-Am's problem with neutron and reno, please? | 17:54 |
| dhellmann | AJaeger, Sam-I-Am : sure, let's move the conversation to #openstack-release | 17:55 |
| *** akshai has quit IRC | 17:56 | |
| *** ihrachys has quit IRC | 17:56 | |
| *** salv-orlando has quit IRC | 17:56 | |
| sdague | Clint: I beat you to a fix by 60 seconds | 17:57 |
| Clint | sdague: i see that | 17:57 |
| *** _nadya_ has joined #openstack-infra | 17:57 | |
| *** akshai has joined #openstack-infra | 17:57 | |
| *** nwkarsten has quit IRC | 17:58 | |
| *** derekh has quit IRC | 17:58 | |
| sdague | I was just checking on things, then came across your fix, plus another one | 17:58 |
| Clint | abandoned mine | 17:58 |
| sdague | cool | 17:59 |
| *** jordanP has quit IRC | 17:59 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quotas support https://review.openstack.org/285110 | 17:59 |
| *** zul has quit IRC | 18:00 | |
| *** zul has joined #openstack-infra | 18:01 | |
| *** zul_ has joined #openstack-infra | 18:01 | |
| *** sc68cal has joined #openstack-infra | 18:01 | |
| clarkb | infra-root can you think of any reason I shouldn't test the fstrim idea against our large raw image then convert to qcow2 to see how small it can get? | 18:01 |
| clarkb | I would be doing this on the nodepool server to avoid large file transfers | 18:01 |
| *** zul has quit IRC | 18:01 | |
| *** zul_ is now known as zul | 18:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert gate-{name}-puppet-syntax to ubuntu-trusty https://review.openstack.org/289036 | 18:02 |
| anteaya | I'm for testing how to decrease image size | 18:02 |
| anteaya | if the test fails you aren't impacting anything, correct? | 18:02 |
| *** e0ne has joined #openstack-infra | 18:03 | |
| *** sputnik13 has joined #openstack-infra | 18:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 18:03 |
| *** arxcruz has quit IRC | 18:03 | |
| *** ddieterly is now known as ddieterly[away] | 18:03 | |
| clarkb | anteaya: I don't expect it to impact anything but the idea of running fstrim on a loopback device where a raw image is attached seems like something that could make the kernel unhappy in a very rare case | 18:03 |
| fungi | clarkb: is there room to work on a copy? if so i see no reason for concern | 18:04 |
| clarkb | fungi: there is plenty of room | 18:04 |
| *** infra-red has joined #openstack-infra | 18:04 | |
| fungi | the kernel should be sane about not doing strange things to other devices when you tell it to do something to a loopback device | 18:04 |
| AJaeger | We need some help with the release-notes publish post job | 18:05 |
| AJaeger | Log file is here http://logs.openstack.org/74/7442bfd4ea6a755dcc255dd80922c0f904fa0bde/post/neutron-releasenotes/be7bcd9/console.html | 18:05 |
| AJaeger | It shows that the job is run from stable/liberty | 18:05 |
| dhellmann | that job should only every and always run using the most current version of master, regardless of what triggers it to run | 18:05 |
| AJaeger | but we use branch-git-prep: branch: master | 18:05 |
| fungi | AJaeger: at least it's not a victim of the ubuntu-trusty migration, since it's still running on bare-trusty there | 18:05 |
| AJaeger | fungi: I have a patch up for that conversion ;) | 18:06 |
| AJaeger | fungi: but my first thought as well ;) | 18:06 |
| anteaya | is this the offending line? http://logs.openstack.org/74/7442bfd4ea6a755dcc255dd80922c0f904fa0bde/post/neutron-releasenotes/be7bcd9/console.html#_2016-03-08_17_01_55_793 | 18:06 |
| *** ihrachys has joined #openstack-infra | 18:06 | |
| clarkb | fungi: rgr I will go ahead and see what this looks like after an fstrim | 18:06 |
| dhellmann | anteaya : http://logs.openstack.org/74/7442bfd4ea6a755dcc255dd80922c0f904fa0bde/post/neutron-releasenotes/be7bcd9/console.html#_2016-03-08_17_00_56_815 | 18:07 |
| clarkb | oh! fstrim works on mounted fses | 18:07 |
| clarkb | this will make dib things simplre | 18:07 |
| *** infra-r__ has quit IRC | 18:07 | |
| fungi | clarkb: yep, which is why SamYaple's idea there was awesome | 18:08 |
| anteaya | dhellmann: I don't see that section mention stable liberty | 18:08 |
| fungi | don't know why it didn't occur to me | 18:08 |
| dhellmann | anteaya : that commit is on the stable branch | 18:08 |
| AJaeger | dhellmann: +1 | 18:08 |
| anteaya | right but I'm wondering where AJaeger's assertion "It shows that the job is run from stable/liberty" is coming from | 18:09 |
| anteaya | as that seems to be the source of the problem that you want solved | 18:09 |
| anteaya | unless I misunderstand | 18:09 |
| *** Guest40848 has quit IRC | 18:09 | |
| *** salv-orl_ has quit IRC | 18:10 | |
| *** sarob has joined #openstack-infra | 18:10 | |
| AJaeger | anteaya: echo stable/liberty three liens above - and the hash confirms that this is indeed from liberty | 18:10 |
| dhellmann | anteaya : what AJaeger said | 18:10 |
| anteaya | I'll stop because I don't understand what you want changed | 18:10 |
| clarkb | hrm fstrim returned very quickly not sure if it did much | 18:10 |
| dhellmann | anteaya : the release notes job is weird. We want it triggered when a patch lands on any branch, but we want the version of the repo checked out to run the job to always be HEAD of the master branch. | 18:11 |
| dhellmann | anteaya : as it's configured now, the script for checking out the repo is checking out the commit that lands, which is normal for most jobs but wrong for this one specific job. | 18:11 |
| dhellmann | I'm looking at builders where we can override that somehow | 18:12 |
| *** bpokorny has quit IRC | 18:12 | |
| dhellmann | AJaeger : we might have to have a custom builder for this. :-/ | 18:12 |
| *** bpokorny has joined #openstack-infra | 18:12 | |
| AJaeger | dhellmann: yeah ;( | 18:12 |
| anteaya | how can the version of the repo checked out be HEAD of master if the patch isn't to master? | 18:12 |
| anteaya | what tells zuul merger to go get HEAD of master? | 18:12 |
| *** infra-re_ has joined #openstack-infra | 18:13 | |
| *** mriedem has quit IRC | 18:13 | |
| dhellmann | anteaya : the branch-git-prep builder step of {name}-releasenotes checks it out, but that builder doesn't know to always use master | 18:13 |
| AJaeger | https://jenkins06.openstack.org/job/neutron-releasenotes/116/parameters/ is the job | 18:13 |
| dhellmann | and the branch override doesn't seem to change it | 18:13 |
| *** mriedem has joined #openstack-infra | 18:13 | |
| anteaya | dhellmann: that seems to be the case, yes | 18:13 |
| dhellmann | so I don't think this is a bug in anything that currently exists, just a missing feature | 18:14 |
| AJaeger | and ZUUL_REF is set, so jenkins/scripts/gerrit-git-prep.sh ignores $BRANCH which we set ;( | 18:14 |
| dhellmann | or "feature" | 18:14 |
| AJaeger | dhellmann: I'll give you a reference... | 18:14 |
| dhellmann | AJaeger : yeah | 18:14 |
| *** tonytan4ever has joined #openstack-infra | 18:14 | |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/gerrit-git-prep.sh#n50 | 18:15 |
| AJaeger | We never reach that line ;( | 18:15 |
| dhellmann | AJaeger : right | 18:15 |
| *** sshnaidm has joined #openstack-infra | 18:15 | |
| AJaeger | fungi, so gerrit-git-prep does not work as we expect it to work ;( | 18:15 |
| tonytan4ever | Hi folks, we are running into a unicode issue: http://logs.openstack.org/98/279798/4/gate/gate-poppy-python34/a6593d6/console.html | 18:15 |
| *** sdake_ has joined #openstack-infra | 18:15 | |
| dhellmann | AJaeger : oh, so you think this is a bug after all? | 18:16 |
| tonytan4ever | Where one file that contains unicode triggers an error. | 18:16 |
| tonytan4ever | Can anybody here points me a direction of how to fix that ? | 18:16 |
| *** infra-red has quit IRC | 18:16 | |
| fungi | dhellmann: AJaeger: the intention of branch overrides to gerrit-git-prep is to make it fall back to a particular branch if there's no change for it to check out | 18:16 |
| clarkb | tonytan4ever: you likely need to explicitly set the LC_ALL variable in your tests to allow utf8 | 18:16 |
| dhellmann | ok, so it's working as designed but not as we want for this case | 18:16 |
| fungi | dhellmann: AJaeger: but not to tell it not to check out the change being evaluated | 18:17 |
| *** e0ne has quit IRC | 18:17 | |
| clarkb | tonytan4ever: try LC_ALL=en_US.utf8 or similar in your tox.ini | 18:17 |
| tonytan4ever | Ok. | 18:17 |
| tonytan4ever | I see. | 18:17 |
| tonytan4ever | Let me try. | 18:17 |
| dhellmann | fungi, AJaeger : so I'm back to thinking we need a custom builder step to do the checkout, do you agree? | 18:17 |
| *** e0ne has joined #openstack-infra | 18:17 | |
| tonytan4ever | Thanks | 18:17 |
| fungi | tonytan4ever: or alternatively adjust the content there to be 7-bit clean | 18:17 |
| *** e0ne has quit IRC | 18:17 | |
| pabelanger | clarkb: tonytan4ever: yes, that will likely work. python-swiftclient had the same issue | 18:17 |
| *** markvoelker has joined #openstack-infra | 18:18 | |
| *** IlyaG has quit IRC | 18:19 | |
| fungi | pabelanger: there was another project last night reporting a similar issue, where they had an em-dash in their setup.cfg i think | 18:19 |
| *** sdake has quit IRC | 18:19 | |
| *** jamesmcarthur has quit IRC | 18:19 | |
| *** IlyaG has joined #openstack-infra | 18:19 | |
| fungi | pabelanger: in that case i think it was setuptools deciding not to decode the file as anything other than ascii | 18:20 |
| fungi | pabelanger: specifically under python 3.4 | 18:20 |
| clarkb | mordred: I am told that the lack of days on your speaker schedule should be addressed soon | 18:20 |
| *** sdake_ is now known as sdake | 18:20 | |
| fungi | so this is probably a setuptools py3k locale/decoding issue | 18:20 |
| *** dprince has quit IRC | 18:21 | |
| AJaeger | So, what's the best action? Enhance gerrit-git-prep to take another parameter? Just checkout git master locally? Write an extra git clone function? | 18:22 |
| dhellmann | AJaeger : I'm writing a custom builder inline in the job template. If it proves useful, we can make it more general later. | 18:22 |
| AJaeger | dhellmann: ok | 18:22 |
| *** dprince has joined #openstack-infra | 18:22 | |
| fungi | AJaeger: it might make sense to override ZUUL_REF before calling into one of the git prep scripts, though that's also pretty ugly | 18:23 |
| *** bswartz has quit IRC | 18:23 | |
| fungi | right now i don't think our checkout scripts (via ggp or z-c) support the idea of ignoring ZUUL_REF | 18:23 |
| *** links has quit IRC | 18:23 | |
| AJaeger | we use ZUUL_REF=master already in propose-devstack-plugins-list - see jenkins/jobs/devstack.yaml | 18:24 |
| *** bswartz has joined #openstack-infra | 18:24 | |
| *** ddieterly[away] is now known as ddieterly | 18:24 | |
| AJaeger | wait, that's different | 18:24 |
| *** ddieterly has quit IRC | 18:24 | |
| fungi | dhellmann: just to clarify, when release notes get updated in a stable branch, the edits get forward-ported to master and landed first, so we should expect the latest state of the master branch to always be appropriate for any release no matter what branch was released? | 18:25 |
| openstackgerrit | Merged openstack-infra/release-tools: dump a release tracking dashboard https://review.openstack.org/288739 | 18:25 |
| AJaeger | fungi: the reno script checks all active branches - as configured in the repo - for release notes | 18:25 |
| *** armax has quit IRC | 18:25 | |
| AJaeger | So, the neutron one checks master and stable/liberty itself | 18:26 |
| dhellmann | fungi : the reason this is weird is that reno does not look at the checked out files, it uses "git show" to get file contents, but *sphinx* reads the files on the filesystem and we need the branch-specific rst files for sphinx to exist properly | 18:26 |
| *** armax has joined #openstack-infra | 18:26 | |
| *** ddieterly has joined #openstack-infra | 18:26 | |
| AJaeger | Or differently: Checks current branch and stable/liberty | 18:26 |
| dhellmann | fungi : that's not quite true, reno looks at the current checked out branch history in one case (unreleased.rst) and that's where we're seeing this issue | 18:26 |
| fungi | dhellmann: so it should explicitly switch to master before it does that? | 18:26 |
| AJaeger | yes | 18:27 |
| mriedem | dims: have you seen this before? http://logs.openstack.org/21/290021/1/check/gate-requirements-tox-nodb-validate/289663c/console.html#_2016-03-08_17_06_04_810 | 18:27 |
| fungi | dhellmann: can the script just `git checkout origin/master` at some point prior to that step? | 18:27 |
| *** kzaitsev_mb has joined #openstack-infra | 18:27 | |
| openstackgerrit | Merged openstack-infra/release-tools: ignore existing default branch when setting new default branch https://review.openstack.org/289384 | 18:27 |
| mriedem | dims: 'python-openstackisdk' appears in global-requirements.txt - is not actually in g-r on stable/liberty | 18:27 |
| fungi | or `git reset --hard origin/master` or something | 18:27 |
| *** flepied has quit IRC | 18:28 | |
| dhellmann | fungi : we could do that. is that better than checking out the right thing to begin with? | 18:28 |
| dims | mriedem : typo? "i" | 18:28 |
| AJaeger | dhellmann, fungi : we could add a customer builder that does it , can't we? | 18:28 |
| *** fawadkhaliq has joined #openstack-infra | 18:28 | |
| *** sridhar_ram is now known as sridhar_ram_afk | 18:28 | |
| dims | mriedem : python-openstackisdk -> python-openstacksdk | 18:28 |
| dhellmann | AJaeger : sure, I guess it's a matter of whether we should "git checkout what-we-don't-want" then "git checkout what we do want" or just do the second step | 18:29 |
| clarkb | the fstrimmed qcow2 is slightly smaller than the zerofreed qcow2, but fstrim will work with all the filesystems so is a better tool for dib | 18:29 |
| mriedem | dims: it's not actually part of my change | 18:29 |
| clarkb | ~60MB difference | 18:29 |
| mriedem | https://review.openstack.org/#/c/290021/ | 18:29 |
| dhellmann | AJaeger : I suppose this way it's a simpler change to the project config | 18:29 |
| fungi | dhellmann: AJaeger: yeah, the difference is that we already have scripts which clone the repos you need through some fairly complex logic, but also happen to intentionally check out whatever ref zuul says is being processed. the git checkout is a relatively simple task compared to the cloneing, cache management, updating, et cetera | 18:29 |
| dhellmann | that's a good point | 18:30 |
| dhellmann | ok, I'll do that | 18:30 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Hide 'quote' button for non-logged-in users https://review.openstack.org/289851 | 18:31 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Mention Comments in Heading for Events Timeline https://review.openstack.org/289879 | 18:31 |
| fungi | we could in theory enhance those scripts to also take additional variables/parameters/something to override the checkout behavior so that jobs can intentionally ignore ZUUL_REF but i'm hesitant to see that as a justified additional complexity if it's just in support of this one job | 18:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/gearman-plugin: Register the diff of functions https://review.openstack.org/180371 | 18:31 |
| dhellmann | fungi , AJaeger : actually, we have 2 cases. During the *check* queue we do want the job to run with the patch being checked, because the "unreleased.rst" file in that branch will show the patch but the series-specific rst file won't show the notes because the patch is not yet on the branch | 18:31 |
| jeblair | clarkb: i dusted off an old change of yours ^ | 18:31 |
| dhellmann | fungi , AJaeger : is there some way to tell inside the job whether we're in the check or the post queue? | 18:32 |
| clarkb | jeblair: thank you I will have to take a look | 18:32 |
| dhellmann | ah, zuul_pipeline | 18:32 |
| dhellmann | ok, I can use that to control the logic | 18:32 |
| jeblair | clarkb, zaro: i added one line to https://review.openstack.org/180371 which i think will address zaro's comments | 18:32 |
| mriedem | dims: ha | 18:32 |
| mriedem | dims: you sir, are a master reviewer | 18:32 |
| dims | lol :) | 18:33 |
| fungi | dhellmann: yeah, you can have a look at the parameters that get set by zuul by looking in jenkins https://jenkins01.openstack.org/job/neutron-releasenotes/105/parameters/ | 18:33 |
| fungi | and ZUUL_PIPELINE is probably what you're looking for there | 18:33 |
| jeblair | dhellmann: do not switch based on ZUUL_PIPELINE | 18:33 |
| dhellmann | jeblair : oh? | 18:34 |
| AJaeger | has any project-config core time for some more reviews of moving us to ubuntu-trusty, please? https://review.openstack.org/#/q/status:open topic:bindep is current list | 18:34 |
| jeblair | dhellmann: it's a word that means things for humans, not something we should be encoding in jobs | 18:34 |
| fungi | true, if we were to rename the pipeline... | 18:34 |
| dhellmann | jeblair : ok | 18:34 |
| *** ybathia has joined #openstack-infra | 18:34 | |
| dhellmann | oh, nevermind, we have a separate gate job defined for release notes builds anyway | 18:34 |
| dhellmann | so we already have 2 job definitions | 18:34 |
| fungi | yeah, it's different job names | 18:34 |
| jeblair | yeah, that's the way to go | 18:34 |
| fungi | i was just noticing that | 18:34 |
| fungi | and far more sane that way | 18:35 |
| *** markvoelker has quit IRC | 18:35 | |
| jeblair | (and if they have similar components, make them into macros, etc) | 18:35 |
| dhellmann | yeah, the whole point of the separate jobs originally was to set up the checkout properly | 18:35 |
| dhellmann | we just misunderstood how that actually worked | 18:35 |
| *** nwkarsten has joined #openstack-infra | 18:35 | |
| dhellmann | so, git experts, what's the best way to get to master? "git checkout origin/master" or "git checkout master" or "git reset --hard origin/master" or something else? | 18:36 |
| * jeblair reads backscroll | 18:37 | |
| *** markvoelker has joined #openstack-infra | 18:37 | |
| jesusaur | dhellmann: I typically go with 'git remote update ; git reset --hard origin/master' | 18:38 |
| *** jamesmcarthur has joined #openstack-infra | 18:38 | |
| fungi | dhellmann: checkout and reset are going to work mostly the same (one changes to a different branch, while the other forces a new state on the current branch), though if you're looking for consistency we use checkout origin/master in propose_update.sh | 18:38 |
| dhellmann | consistency is good | 18:39 |
| *** ddieterly has quit IRC | 18:39 | |
| *** jamesmcarthur has quit IRC | 18:40 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: always build release notes from master https://review.openstack.org/290067 | 18:40 |
| dhellmann | fungi , AJaeger, jeblair : ^^ | 18:41 |
| *** salv-orlando has joined #openstack-infra | 18:42 | |
| AJaeger | dhellmann: why do you want to handle gate and post job differently here? | 18:43 |
| *** sdake_ has joined #openstack-infra | 18:43 | |
| dhellmann | AJaeger : in the gate job we want unreleased.rst to refer to the current branch, where the patch merged, so we can tell if the release notes build successfully | 18:43 |
| AJaeger | dhellmann: ok | 18:44 |
| dhellmann | AJaeger : if you put bad rst in your note, the job fails with an error | 18:44 |
| fungi | also that avoids you deadlocking the job on one branch by merging bad content into a different branch | 18:44 |
| AJaeger | you're right, we're on a *branch* in the gate | 18:44 |
| AJaeger | dhellmann: thanks, I'm fine now... | 18:44 |
| *** keedya has quit IRC | 18:44 | |
| clarkb | jeblair: I think your change is sufficient, we clear the functionmap, then the next time through working we update the list | 18:45 |
| dhellmann | fungi : right | 18:45 |
| jeblair | (if reno used zuul-cloner to perform the checkouts, we could actually run the same process in both places) | 18:45 |
| *** sdake has quit IRC | 18:45 | |
| *** keedya has joined #openstack-infra | 18:45 | |
| dhellmann | jeblair : reno depends on the repo already being checked out, it doesn't check it out itself | 18:45 |
| jeblair | (zuul-cloner is best looked at as "make this git repo represent the proposed future state of this project-branch) | 18:45 |
| fungi | so the goal of the version of the job in post is to be essentially "branchless" (with the caveat that there's a file it needs to obtain from the master branch of the repo) | 18:45 |
| clarkb | jeblair: previously I had done some simple testing against jenkins-dev iirc and it mostly worked except for the reconnection things zaro found. Not sure how we want to test the latest patchset | 18:45 |
| dhellmann | jeblair : the issue is that from master we want to build release notes for all branches | 18:46 |
| fungi | and yet trigger on merges to any branch | 18:46 |
| dhellmann | or at least, we want a job that does that | 18:46 |
| dhellmann | right | 18:46 |
| jeblair | dhellmann: i understand the issue, i think the disconnect is that reno does not 'check things out' but rather simply fetches git content from the repo, right? | 18:46 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add a bandit non-voting for Kolla https://review.openstack.org/290070 | 18:47 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add a bandit non-voting for Kolla https://review.openstack.org/290070 | 18:47 |
| jeblair | dhellmann: so while z-c can be used to check out a repo state, it can not be used to stream git content from a proposed future branch | 18:47 |
| jeblair | which is what reno is doing, right? | 18:47 |
| dhellmann | jeblair : so you're saying check out the branch and then read the files instead of using git commands to fetch them? | 18:48 |
| dhellmann | we're scanning the git history to figure out which file has which tag(s) to sort the notes into the right versions | 18:48 |
| dhellmann | fungi , AJaeger : jjb does not like that patch locally, let me push a new one | 18:49 |
| *** abregman has joined #openstack-infra | 18:49 | |
| *** salv-orlando has quit IRC | 18:49 | |
| anteaya | fungi: is the meeting agenda the way you want it for the meeting? | 18:49 |
| rcarrillocruz | pabelanger: just noticed the trusty-migration topic on the agenda | 18:49 |
| rcarrillocruz | need help on that? | 18:49 |
| jeblair | dhellmann: yeah -- not actual suggestion yet, just trying to figure out how z-c doesn't fit. | 18:49 |
| fungi | anteaya: was just starting to go through it. been busy today, sorry | 18:49 |
| anteaya | no apologies | 18:50 |
| *** salv-orlando has joined #openstack-infra | 18:50 | |
| anteaya | just noticed the time myself | 18:50 |
| rcarrillocruz | i assume there's quite a bit of work for that | 18:50 |
| *** dmsimard has joined #openstack-infra | 18:50 | |
| dhellmann | jeblair : I'm happy to discuss this to see if we can make it simpler. I hesitate to change the contents of the current working dir, though, because sphinx is also likely to be in the middle of processing those files when it invokes reno. | 18:50 |
| *** sdake_ is now known as sdake | 18:50 | |
| *** mrmartin has joined #openstack-infra | 18:51 | |
| jeblair | dhellmann: it's starting to seem like if z-c were to put the proposed-future-branch-states into the local repo with their actual branch names, rather than simply checking something out as a detached head, this might "just work" | 18:51 |
| *** trown has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Convert gate-{name}-puppet-syntax to ubuntu-trusty https://review.openstack.org/289036 | 18:52 |
| dhellmann | jeblair : that would allow us to run the job from master all the time, yes | 18:52 |
| jeblair | dhellmann: maybe we should think about doing that in zuulv3 then... | 18:52 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: always build release notes from master https://review.openstack.org/290067 | 18:52 |
| dhellmann | jeblair : it's worth considering; I don't know if there are other things that would be confused by that change | 18:53 |
| fungi | i wonder if it should namespace the branches it updates so that it doesn't step on copies in use (in case of being called in a workdir on someone's workstation where they have changes staged on the actual branches) | 18:53 |
| dhellmann | TIL: virtualenv fails if your path to the env is too long | 18:53 |
| pleia2 | dhellmann: that's unpleasant | 18:53 |
| clarkb | dhellmann: it has to do with a change in the linux kernel almost as old as the linux kernel itself | 18:53 |
| fungi | dhellmann: 128 character path limit! | 18:53 |
| clarkb | bsd is not affected | 18:54 |
| dhellmann | "who would ever need more than 128 characters in a path?" | 18:54 |
| pabelanger | rcarrillocruz: Sure, plan to talk about it today, if time permits. Mostly need a infra-root to step up and help launch the new instance. I've tested locally with puppet and things work fine | 18:54 |
| clarkb | or rather bsd has a much longer limit to the path in the shebang | 18:54 |
| pabelanger | rcarrillocruz: fungi also suggested moving things to cinder to help the migration | 18:54 |
| clarkb | dhellmann: basically it has to do with how exec is processes the shebang in the kernel and it only allows a 127 + null terminal string | 18:55 |
| fungi | anteaya: yeah, so i did clean up the agenda immediately after last week's meeting. looks accurate now | 18:55 |
| jeblair | fungi: in zv3 checkouts will be pushed to the worker from the launcher, so we don't have to worry about local workstation git repos | 18:55 |
| anteaya | fungi: wonderful, thanks | 18:55 |
| *** sridhar_ram_afk is now known as sridhar_ram | 18:55 | |
| *** tphummel has joined #openstack-infra | 18:55 | |
| fungi | jeblair: ahh, yeah that case does get simpler | 18:55 |
| dhellmann | clarkb : thanks. I suppose I should rename all of my working directories with single character names. | 18:56 |
| anteaya | that sounds useful | 18:56 |
| fungi | were there any other pending patches that change a bunch of jobs? i keep meaning to ask if we should merge something else before i do the manual jjb updates for the mysql-prep macro change that merged, so i don't do more than one of these babysitting jobs a day :/ | 18:56 |
| jhesketh | Morning | 18:56 |
| *** jcoufal has quit IRC | 18:56 | |
| anteaya | morning jhesketh | 18:57 |
| *** jpr has quit IRC | 18:57 | |
| clarkb | I wonder how hard it would be to extend that limit in a modern kernel | 18:57 |
| dhellmann | fungi, AJaeger, jeblair : thanks for your help debugging and fixing this issue! | 18:57 |
| anteaya | AJaeger: did you have more patches to change jobs? | 18:57 |
| clarkb | if the code isn't terrible it is probably a one line update to a constnat | 18:57 |
| * clarkb clones the kernel | 18:57 | |
| AJaeger | fungi, https://review.openstack.org/289793 https://review.openstack.org/289824 https://review.openstack.org/289816 | 18:57 |
| *** IlyaG_ has joined #openstack-infra | 18:57 | |
| AJaeger | anteaya: those are the ones with many changes | 18:58 |
| AJaeger | full list: https://review.openstack.org/#/q/status:open topic:bindep | 18:58 |
| *** ihrachys has quit IRC | 18:58 | |
| fungi | AJaeger: just wondering if any of them modify hundreds of jobs that we should land them before the coming manual jjb update | 18:58 |
| jeblair | fungi: let's see if we can get clarkb's gearman-plugin change in... before we do another round of babysitting... | 18:58 |
| fungi | jeblair: oh, great call | 18:59 |
| AJaeger | fungi, those three change the most | 18:59 |
| *** akshai has quit IRC | 18:59 | |
| fungi | AJaeger: thanks | 18:59 |
| * dims braces for impact :) | 18:59 | |
| *** doug-fis_ has joined #openstack-infra | 18:59 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 18:59 |
| *** bdemers has quit IRC | 19:00 | |
| *** angdraug has joined #openstack-infra | 19:01 | |
| *** IlyaG has quit IRC | 19:01 | |
| *** flepied has joined #openstack-infra | 19:01 | |
| *** bapalm has quit IRC | 19:02 | |
| *** ihrachys has joined #openstack-infra | 19:02 | |
| *** tphummel has quit IRC | 19:02 | |
| *** doug-fish has quit IRC | 19:02 | |
| *** doug-fish has joined #openstack-infra | 19:03 | |
| *** ybathia has quit IRC | 19:03 | |
| *** doug-fish has quit IRC | 19:03 | |
| *** doug-fish has joined #openstack-infra | 19:03 | |
| *** doug-fis_ has quit IRC | 19:04 | |
| *** ashleighfarnham has joined #openstack-infra | 19:04 | |
| *** bdemers has joined #openstack-infra | 19:04 | |
| *** tphummel has joined #openstack-infra | 19:05 | |
| *** ldnunes has quit IRC | 19:05 | |
| *** dizquierdo_ has quit IRC | 19:06 | |
| *** _nadya_ has quit IRC | 19:06 | |
| *** akshai has joined #openstack-infra | 19:06 | |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 19:07 |
| *** annegentle has joined #openstack-infra | 19:08 | |
| anteaya | dhellmann: oh since I said I would do so, we have 15% of check jobs that zuul knows about as non-voting jobs, of all the jobs in check | 19:08 |
| anteaya | dhellmann: that is not to say all check jobs are being run, just the set of all check jobs zuul knows about | 19:08 |
| *** silos has joined #openstack-infra | 19:09 | |
| openstackgerrit | David Shrewsbury proposed openstack/gertty: Use diff long options and uncap GitPython https://review.openstack.org/290080 | 19:09 |
| *** tphummel has quit IRC | 19:09 | |
| *** ldnunes has joined #openstack-infra | 19:10 | |
| *** hashar has joined #openstack-infra | 19:10 | |
| *** abregman has quit IRC | 19:11 | |
| Shrews | jeblair: ^^^ just admitting defeat on the gitpython fight | 19:12 |
| *** esker has joined #openstack-infra | 19:12 | |
| *** esker has quit IRC | 19:12 | |
| *** esker has joined #openstack-infra | 19:12 | |
| *** arxcruz has joined #openstack-infra | 19:14 | |
| *** kzaitsev_mb has quit IRC | 19:14 | |
| *** amitgandhinz has quit IRC | 19:15 | |
| *** electrofelix has quit IRC | 19:15 | |
| *** infra-red has joined #openstack-infra | 19:15 | |
| *** amitgandhinz has joined #openstack-infra | 19:15 | |
| openstackgerrit | Merged openstack-infra/project-config: always build release notes from master https://review.openstack.org/290067 | 19:15 |
| *** kencjohnston has joined #openstack-infra | 19:17 | |
| *** infra-re_ has quit IRC | 19:18 | |
| *** bapalm has joined #openstack-infra | 19:19 | |
| *** rbrndt_ has quit IRC | 19:20 | |
| *** zul has quit IRC | 19:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for releasenotes jobs https://review.openstack.org/289824 | 19:20 |
| clarkb | dhellmann: the constant is BINPRM_BUF_SIZE | 19:20 |
| clarkb | dhellmann: in theory you can just s/128/2048/ and it will work? not sure how hard that would get flamed on lkml | 19:21 |
| hashar | o/ good morning | 19:21 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Use swap-partition.yaml environment https://review.openstack.org/289085 | 19:22 |
| *** dprince has quit IRC | 19:23 | |
| *** fawadkhaliq has quit IRC | 19:26 | |
| *** dprince has joined #openstack-infra | 19:27 | |
| *** Sukhdev has joined #openstack-infra | 19:27 | |
| *** infra-re_ has joined #openstack-infra | 19:30 | |
| *** tonytan4ever has quit IRC | 19:32 | |
| *** shardy has quit IRC | 19:33 | |
| *** exploreshaifali has quit IRC | 19:33 | |
| *** infra-red has quit IRC | 19:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: mistral, ironic-python-agent-jobs: Convert to ubuntu-trusty https://review.openstack.org/290092 | 19:35 |
| *** kzaitsev_mb has joined #openstack-infra | 19:36 | |
| *** zul has joined #openstack-infra | 19:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: astara-appliance-buildimage: Use ubuntu-trusty https://review.openstack.org/290093 | 19:40 |
| *** _nadya_ has joined #openstack-infra | 19:42 | |
| *** sarob has quit IRC | 19:42 | |
| *** rbrndt_ has joined #openstack-infra | 19:43 | |
| *** jpr has joined #openstack-infra | 19:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert bandit to ubuntu-trusty https://review.openstack.org/290094 | 19:44 |
| *** doug-fish has quit IRC | 19:45 | |
| *** doug-fish has joined #openstack-infra | 19:45 | |
| *** rguillebert has quit IRC | 19:47 | |
| *** kzaitsev_mb has quit IRC | 19:48 | |
| *** sridhar_ram has quit IRC | 19:49 | |
| *** sarob has joined #openstack-infra | 19:49 | |
| *** doug-fish has quit IRC | 19:50 | |
| *** claudiub has joined #openstack-infra | 19:50 | |
| *** daemontool has quit IRC | 19:51 | |
| *** arxcruz has quit IRC | 19:54 | |
| openstackgerrit | Merged openstack-infra/project-config: Add missing PROJECT definition https://review.openstack.org/289590 | 19:54 |
| *** rockyg has joined #openstack-infra | 19:54 | |
| *** sdake_ has joined #openstack-infra | 19:55 | |
| *** xarses_ has joined #openstack-infra | 19:55 | |
| *** xarses has quit IRC | 19:55 | |
| *** claudiub has quit IRC | 19:56 | |
| *** andreww has joined #openstack-infra | 19:56 | |
| *** claudiub has joined #openstack-infra | 19:56 | |
| *** sdake has quit IRC | 19:57 | |
| *** ihrachys has quit IRC | 19:58 | |
| *** zul has quit IRC | 19:58 | |
| *** xarses_ has quit IRC | 19:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add pabelanger to infra root https://review.openstack.org/290104 | 20:00 |
| *** amitgandhinz has quit IRC | 20:00 | |
| hashar | I really like lurking the infra meetings | 20:00 |
| pleia2 | ran out of time, but I'll be at FOSS Asia next week (chatting about how we use code review for our work!) | 20:00 |
| anteaya | wooooo | 20:00 |
| hashar | \O/ | 20:00 |
| anteaya | hashar: thanks for participating | 20:01 |
| *** kzaitsev_mb has joined #openstack-infra | 20:01 | |
| pleia2 | lunch now | 20:01 |
| anteaya | dhellmann: did you see my response earlier in channel? 15% of all check jobs that zuul knows about are non-voting | 20:01 |
| anteaya | pleia2: enjoy lunch | 20:01 |
| *** baoli_ has quit IRC | 20:02 | |
| clarkb | zaro if you are around want to review https://review.openstack.org/#/c/180371/3 ? jeblair updated it to address your comments | 20:02 |
| *** silos1 has joined #openstack-infra | 20:02 | |
| *** zul has joined #openstack-infra | 20:02 | |
| zaro | clarkb: yup, will take a look | 20:02 |
| clarkb | zaro and maybe we can get it tested quickly and deployed before more of our mass jenkins update | 20:02 |
| zaro | mass jenkins update? what others? | 20:03 |
| nibalizer | pleia2: awesome | 20:03 |
| *** Sukhdev has quit IRC | 20:03 | |
| nibalizer | pabelanger: congratz, welcome! | 20:03 |
| pabelanger | nibalizer: thanks | 20:03 |
| AJaeger | any project-config cores that want to review these so that they can go in for the jenkins update, please? https://review.openstack.org/289793 https://review.openstack.org/289824 https://review.openstack.org/289816 | 20:03 |
| clarkb | zaro: we have been transitioning from bare-trusty and devstack-trusty to ubuntu-trusty and that means updating the vast majority of our jenkins jobs | 20:03 |
| zaro | hashar: we have figured 2.11->2.8 rollback scenario so should be safe for you to upgrade :) | 20:03 |
| clarkb | zaro: which is very slow and innefficient which this change aims to address | 20:04 |
| hashar | zaro: gotta bump Wikimedia Zuul version first :-( | 20:04 |
| *** tphummel has joined #openstack-infra | 20:04 | |
| *** kencjohnston has quit IRC | 20:05 | |
| hashar | zaro: oh and Wikimedia runs Jenkins 1.642.2 if it is of any interest. Gearman plugin seems to work just fine | 20:05 |
| zaro | hashar: hmm, don't remember which version of zuul we were on when we did the upgrade, just remember that it was compatible. | 20:05 |
| *** silos has quit IRC | 20:06 | |
| clarkb | hashar: that is good to know, for us though the biggest jenkins issue is not addressed so not sure how much of a difference it makes | 20:06 |
| *** esikachev has joined #openstack-infra | 20:06 | |
| clarkb | hashar: there is a thread leak in the ssh-slvaes plugin which hasn't been updated since septemberish | 20:06 |
| zaro | hashar: ahh, that's good, i think timrc wanted that fix for 1.642 | 20:06 |
| hashar | zaro: we might jump straight to Gerrit 2.12. Will test out the interaction on some dev setup before, but I am not too worried | 20:06 |
| hashar | clarkb: restart Jenkins @daily ? :-D Wikimedia setup spawns less instances / has less slaves so I guess we have no impact there | 20:07 |
| zaro | hashar: want to test change clarkb referenced (above) on your jenkins? | 20:08 |
| hashar | oh the diff | 20:09 |
| nibalizer | jeblair: crinkle i have appeased the puppet-apply gods https://review.openstack.org/#/c/284466/ | 20:09 |
| zaro | clarkb: i'm slightly busy now, but will review that jenkins change by EOD. | 20:09 |
| clarkb | zaro: ok, thank you | 20:09 |
| nibalizer | i'm a bit leery of putting a for loop into a cron entry, maybe that should be its own shell script? | 20:09 |
| hashar | zaro: clarkb: funny thing when we had 3k jobs and refreshed them with JJB, that would eventually kill the gearman server and deadlock all pending changes in Zuul :D | 20:10 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 20:10 |
| clarkb | hashar: yes, the problem is without that change we update every job every time we update a job | 20:10 |
| clarkb | hashar: so it is N^2 job updates | 20:10 |
| clarkb | hashar: with this change it should be closer to N job updates and be much more reliable | 20:10 |
| *** gyee has quit IRC | 20:10 | |
| hashar | * number of jenkins master | 20:10 |
| *** matrohon has joined #openstack-infra | 20:11 | |
| clarkb | yes | 20:11 |
| zaro | hashar: might fix deadlock? | 20:11 |
| *** sdake_ is now known as sdake | 20:11 | |
| *** nwkarsten has quit IRC | 20:11 | |
| hashar | the deadlock I got it figured out in Zuul ( https://review.openstack.org/#/c/128921/ ) | 20:12 |
| *** nwkarsten has joined #openstack-infra | 20:12 | |
| fungi | hashar: as for having debian systems in our infrastructure, there is work underway within the community to have diskimage-builder generated debian images for our ci, which is the easiest (for some definition of the word) foothold for spreading it further into our toolchain | 20:12 |
| crinkle | nibalizer: did you mean to ping me? | 20:13 |
| crinkle | nibalizer: I have no strong opinions on that | 20:13 |
| zaro | hashar: i thought deadlock was in jenkins? | 20:13 |
| *** Sukhdev has joined #openstack-infra | 20:13 | |
| nibalizer | crinkle: is it valid code? | 20:14 |
| openstackgerrit | Ghe Rivero proposed openstack-infra/shade: Add quotas support https://review.openstack.org/285110 | 20:14 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron https://review.openstack.org/284466 | 20:15 |
| hashar | zaro: updating 3k jobs each having two labels caused 9k functions update for each of the 3k jobs. That some how caused the gearman server to timeout from time to time | 20:15 |
| crinkle | nibalizer: not having tested it i would be concerned how the cron type evaluates built-ins like 'for' but i'm not sure | 20:15 |
| nibalizer | i'm worried about a) that join being too clever, and b) the cron | 20:15 |
| nibalizer | crinkle: ok ill boot a vm and peer at it | 20:15 |
| hashar | zaro: in turn causing some merge:merge function to be marked as enqueued when really the gearman server timed out and the merge:merge would never occur. End result: changes sitting idle waiting for a merge:merge job to occur :D | 20:15 |
| *** nwkarste_ has joined #openstack-infra | 20:16 | |
| crinkle | nibalizer: i think the join is fine (cacti hosts don't have spaces in them right?) | 20:16 |
| hashar | fungi: yeah I did some dib patches for debian. We use Jessie for nodepool instances :) | 20:16 |
| hashar | fungi: though they are very minimal and most of the cruft is added via puppet | 20:16 |
| *** nwkarsten has quit IRC | 20:16 | |
| *** piet has joined #openstack-infra | 20:18 | |
| *** mrmartin has quit IRC | 20:18 | |
| *** silos1 has quit IRC | 20:19 | |
| AJaeger | pabelanger, fungi: Looking at bandit - see also https://review.openstack.org/#/c/290094/ , I wonder how we make these kind of integration jobs work if any of the involved projects ever uses a other-requirements.txt file | 20:20 |
| clarkb | AJaeger: integration jobs need to manage their own deps | 20:21 |
| clarkb | AJaeger: devstack for example | 20:21 |
| *** yamahata has quit IRC | 20:22 | |
| AJaeger | clarkb: So, does not work as is without extra changes if a project has extra needs, correct? | 20:22 |
| clarkb | AJaeger: ya, but that has typically been the case which is why devstack does what it does | 20:23 |
| pabelanger | AJaeger: Hmm, pretty sure the zuul-cloner code can be moved outside of shell. | 20:23 |
| *** jamesmcarthur has joined #openstack-infra | 20:23 | |
| *** mrmartin has joined #openstack-infra | 20:23 | |
| *** mrmartin has quit IRC | 20:23 | |
| fungi | right. devstack might grow a mechanism to iterate through the requirements lists of the projects it's asked to test and assemble a list of distro packages to install from that, but devstack's still in charge of that part | 20:23 |
| pabelanger | target-org is outdated | 20:23 |
| *** sridhar_ram has joined #openstack-infra | 20:23 | |
| AJaeger | thanks for confirming. | 20:24 |
| mtreinish | fungi: that probably won't ever happen automagically. It's a list of packages per service right now and I realistically don't see that changing | 20:24 |
| AJaeger | pabelanger: why? It could be openstack-dev or openstack-infra? | 20:24 |
| pabelanger | AJaeger: looking at https://github.com/openstack-infra/project-config/commit/5e01f811b08c7bca53a1f1db05da774004ccc912 that commit could be optimized a fair bit. | 20:24 |
| *** sdake has quit IRC | 20:25 | |
| fungi | mtreinish: it could become additive i suppose (a project could even add that on their own in a plugin) | 20:25 |
| AJaeger | pabelanger: feel free to send a patch | 20:25 |
| * AJaeger volunteers to review... | 20:25 | |
| pabelanger | AJaeger: I think that was the plan for the commit, but doing that is old school. It should be migrated to use the zuul-clone builder | 20:26 |
| pabelanger | AJaeger: sure, I can look at it in a bit | 20:26 |
| fungi | mtreinish: more just that we now have a general tool for parsing cross-platform lists of package dependencies, so there might be room to collaborate on what devstack's doing in that space | 20:26 |
| *** ddieterly has joined #openstack-infra | 20:27 | |
| AJaeger | pabelanger: we have a couple of these "old school" usages, happy to see them updated | 20:27 |
| *** |-paul-| has joined #openstack-infra | 20:27 | |
| *** _nadya_ has quit IRC | 20:27 | |
| AJaeger | pabelanger: I think I see what you mean... | 20:27 |
| *** salv-orlando has quit IRC | 20:28 | |
| *** angdraug has quit IRC | 20:29 | |
| *** rguillebert has joined #openstack-infra | 20:29 | |
| *** apoorvad has quit IRC | 20:30 | |
| *** apoorvad has joined #openstack-infra | 20:30 | |
| *** angdraug has joined #openstack-infra | 20:30 | |
| *** jamesmcarthur has quit IRC | 20:32 | |
| openstackgerrit | Peter Stachowski proposed openstack/requirements: Bump version of python-troveclient to 2.1.0 https://review.openstack.org/290115 | 20:33 |
| *** jamesmcarthur has joined #openstack-infra | 20:33 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Allow passing project_id to create_network https://review.openstack.org/288166 | 20:35 |
| *** ZZelle_ has joined #openstack-infra | 20:35 | |
| *** amitgandhinz has joined #openstack-infra | 20:37 | |
| *** austin81 has joined #openstack-infra | 20:37 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:39 |
| *** |-paul-| has quit IRC | 20:40 | |
| *** |-paul-| has joined #openstack-infra | 20:40 | |
| *** |-paul-| has joined #openstack-infra | 20:41 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 20:41 |
| *** zul has quit IRC | 20:42 | |
| *** oanson has joined #openstack-infra | 20:43 | |
| *** Daisy has joined #openstack-infra | 20:46 | |
| *** infra-re_ has quit IRC | 20:46 | |
| *** infra-red has joined #openstack-infra | 20:47 | |
| *** |-paul-| has quit IRC | 20:47 | |
| *** |-paul-| has joined #openstack-infra | 20:48 | |
| *** shardy has joined #openstack-infra | 20:48 | |
| *** Daisy has quit IRC | 20:51 | |
| *** ZZelle__ has joined #openstack-infra | 20:51 | |
| *** ZZelle__ has quit IRC | 20:51 | |
| *** apoorvad has quit IRC | 20:52 | |
| *** silos has joined #openstack-infra | 20:52 | |
| *** sdake has joined #openstack-infra | 20:52 | |
| *** apoorvad has joined #openstack-infra | 20:52 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 20:53 |
| *** zul has joined #openstack-infra | 20:55 | |
| *** abregman has joined #openstack-infra | 20:55 | |
| *** oanson has quit IRC | 20:56 | |
| AJaeger | pabelanger: no +2 power yet? Or just carefull? ;) | 20:56 |
| AJaeger | pabelanger: thanks for reviewing! | 20:56 |
| *** infra-red has quit IRC | 20:57 | |
| pabelanger | AJaeger: only +1s :) | 20:57 |
| *** ihrachys has joined #openstack-infra | 20:57 | |
| *** infra-red has joined #openstack-infra | 20:57 | |
| fungi | pabelanger: ooh, fixing now! | 20:57 |
| *** asettle has joined #openstack-infra | 20:58 | |
| fungi | pabelanger: you may need to refresh vote windows if you go to change your votes on changes | 20:58 |
| *** bapalm has quit IRC | 20:58 | |
| *** ddieterly is now known as ddieterly[away] | 20:58 | |
| *** keedya has quit IRC | 20:58 | |
| fungi | though also existing infra-core reviewers are free to treat your older +1 votes as effectively +2 | 20:59 |
| pabelanger | fungi: danke! | 21:00 |
| fungi | infra-root: show of support in the form of +2s on https://review.openstack.org/290104 would be cool when you get a moment | 21:00 |
| anteaya | are we doing alphabetical order now? | 21:02 |
| *** esikachev has quit IRC | 21:02 | |
| *** amitgandhinz has quit IRC | 21:02 | |
| anteaya | it just seems odd the new entry is above yolanda when nothing else is alphabetical order | 21:02 |
| clarkb | jeblair: have you already tested the update to the gearman-plugin? I have time now to put it in jenkins-dev and give it a quick spin if not | 21:03 |
| *** amitgandhinz has joined #openstack-infra | 21:03 | |
| AJaeger | pabelanger: do you want to change your +1 for those change where a +2 already exists and approve? | 21:03 |
| fungi | i guess the last 5 lines are in alpha order, though i don't think it matters | 21:03 |
| *** rfolco has quit IRC | 21:04 | |
| *** infra-re_ has joined #openstack-infra | 21:04 | |
| fungi | at least i never considered that file to be an order of precedence | 21:05 |
| anteaya | that's the order they were added in, just happens to be alphabetical | 21:05 |
| *** sdake_ has joined #openstack-infra | 21:05 | |
| fungi | i'll leave it to others to debate | 21:05 |
| anteaya | was just wondering | 21:06 |
| *** jpr has quit IRC | 21:06 | |
| *** infra-red has quit IRC | 21:07 | |
| clarkb | doesn't look like we copy the hpi off to tarballs.o.o on check | 21:07 |
| *** bdemers has quit IRC | 21:07 | |
| *** rockyg has quit IRC | 21:07 | |
| *** sdake has quit IRC | 21:07 | |
| jeblair | clarkb: i have not tested it, no | 21:07 |
| jeblair | i was hoping zaro could do so quickly; i'd have to bootstrap an env | 21:07 |
| clarkb | jeblair: last time I just used jenkins-dev with a foregrounded localhost geard | 21:07 |
| jeblair | clarkb: that sounds like an excellent plan | 21:08 |
| fungi | and i'm free to help deploy it since this is delaying me rerunning jjb everywhere | 21:08 |
| clarkb | was going to give that a go nowish | 21:08 |
| jeblair | clarkb: i'm standing by to help as needed | 21:08 |
| fungi | sounds good | 21:08 |
| pabelanger | AJaeger: in a meeting ATM. So, I can re-review in about 45mins | 21:08 |
| *** rockyg has joined #openstack-infra | 21:09 | |
| clarkb | wow installing maven installs a lot of stuff | 21:09 |
| *** rockyg has quit IRC | 21:09 | |
| *** sambetts is now known as sambetts|afk | 21:09 | |
| AJaeger | pabelanger: no worries... thanks | 21:10 |
| * AJaeger waves good-bye | 21:10 | |
| *** baoli has joined #openstack-infra | 21:11 | |
| *** esker has quit IRC | 21:11 | |
| nibalizer | fungi: i dont have lp on phone but +2 on the pabelanger tch | 21:12 |
| *** esker has joined #openstack-infra | 21:12 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for neutron bug 1553595 https://review.openstack.org/289712 | 21:12 |
| openstack | bug 1553595 in neutron "test_external_network_visibility intermittent failure" [Critical,Fix released] https://launchpad.net/bugs/1553595 - Assigned to Kevin Benton (kevinbenton) | 21:12 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add neutron lbaas dashboard to it's gerrit bot channel https://review.openstack.org/290137 | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| *** bapalm has joined #openstack-infra | 21:14 | |
| jeblair | clarkb: we could recheck and hold the node ;) | 21:15 |
| *** notnownikki has quit IRC | 21:15 | |
| clarkb | jeblair: I think my locak build is almost done :) | 21:15 |
| *** Sukhdev has quit IRC | 21:15 | |
| *** boris-42 has joined #openstack-infra | 21:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Use ubuntu-trusty for jobs using infra-publish-jobs https://review.openstack.org/289829 | 21:18 |
| clarkb | ok stopping jenkins on jenkins-dev now | 21:18 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: (WIP) Add functional tests https://review.openstack.org/288840 | 21:18 |
| *** amrith is now known as _amrith_ | 21:19 | |
| *** jtomasek has quit IRC | 21:20 | |
| *** kencjohnston has joined #openstack-infra | 21:21 | |
| *** ddieterly[away] is now known as ddieterly | 21:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Convert storyboard-docs, -announce-release to ubuntu-trusty https://review.openstack.org/289816 | 21:22 |
| *** kgiusti has left #openstack-infra | 21:23 | |
| sridhar_ram | infra-team: https://review.openstack.org/#/c/289013 | 21:23 |
| sridhar_ram | infra-team: ^^^ above patchset uses "Closes-Bug: #1553715" | 21:24 |
| openstack | bug 1553715 in python-tackerclient "Clean up hacking.rst and doc" [Medium,Confirmed] https://launchpad.net/bugs/1553715 - Assigned to zeih (zeih) | 21:24 |
| *** dprince has quit IRC | 21:24 | |
| sridhar_ram | infra-team: but the patchset is not linked to this launchpad bug | 21:24 |
| sridhar_ram | what could cause this ? | 21:24 |
| *** zul has quit IRC | 21:25 | |
| clarkb | jeblair: the initial REST_ABILITIES and CAN_DO is working | 21:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Remove pabelanger from asterisk and stackalytics https://review.openstack.org/290143 | 21:27 |
| clarkb | now I need to drive enough of the state machine to see it CANT_DO after deleting a project | 21:27 |
| anteaya | sridhar_ram: you used a # in front of the bug number | 21:28 |
| clarkb | updating another project's config was sufficient for that | 21:28 |
| mtreinish | pabelanger: ^^^ you can access to indiviual machines? I didn't know that | 21:28 |
| anteaya | sridhar_ram: I don't know as the system knows what to do wtih the # | 21:28 |
| *** _nadya_ has joined #openstack-infra | 21:28 | |
| sridhar_ram | anteaya: thanks.. will try removing the # | 21:28 |
| fungi | sridhar_ram: anteaya: the # is fine | 21:28 |
| pabelanger | mtreinish: Ya, depending on the system, non infra-root have access to help | 21:28 |
| pabelanger | mtreinish: pbx.o.o had me and russellb to help maintain it | 21:29 |
| clarkb | changing labels around seems to work | 21:29 |
| sridhar_ram | fungi: something else at work then ? | 21:29 |
| clarkb | now we need to kill the geard server and start it again and see that jobs are reregistered | 21:29 |
| mtreinish | pabelanger: man, I wish I knew that, it could have potentially made my life a lot easier for the stuff I setup | 21:29 |
| clarkb | jeblair: yup I get a RESET_ABILITIES and a bunch of CAN_DOs on restart | 21:29 |
| clarkb | er when I restart the geard | 21:30 |
| anteaya | sridhar_ram: sorry to steer you incorrectly | 21:30 |
| sridhar_ram | anteaya: no worries | 21:30 |
| fungi | sridhar_ram: yes, the bug supervisor needs to be a group including the openstack-infra account used by the bug updating hook | 21:30 |
| mtreinish | pabelanger: I guess I'll just have to bug you too now :) | 21:30 |
| *** amitgandhinz has quit IRC | 21:30 | |
| fungi | sridhar_ram: https://bugs.launchpad.net/python-tackerclient says the bug supervisor is a single person instead | 21:30 |
| clarkb | jeblair: fungi: any other situations you want to test? I tested updating labels (both add and remove), deleting a job, and restarting the gerad out from under jenkins | 21:30 |
| clarkb | jeblair: fungi: I suppose adding a new job would be good let me do that | 21:31 |
| * sridhar_ram is looking up | 21:31 | |
| *** cloudtrainme has quit IRC | 21:31 | |
| pabelanger | mtreinish: Ya, guess it depends on the system. I setup grafana.o.o but didn't have access, mostly cause everything could be done via puppet. pbx.o.o was a little different since we needed to some tuning on conferencing and debugging. | 21:31 |
| clarkb | and that looks good too | 21:31 |
| pabelanger | mtreinish: Sure, I can try and help | 21:31 |
| sridhar_ram | fungi: it is the same for https://bugs.launchpad.net/tacker as well and most patchset correctly tracks the Closes-Bug header | 21:32 |
| anteaya | fungi: I'm looking at that page and don't see the bug-supervisor, do I need to be signed in to launchpad? | 21:32 |
| *** _nadya_ has quit IRC | 21:32 | |
| *** cloudtrainme has joined #openstack-infra | 21:33 | |
| *** amitgandhinz has joined #openstack-infra | 21:33 | |
| fungi | sridhar_ram: behavior of the updating depends on whether it needs to reassign the bug. if the submitter of the change is not the asignee of the bug the script will try to reassign. if it's not a bug supervisor for the project that api call will fail and it will fail to perform other actions based on that trigger event | 21:33 |
| fungi | anteaya: no idea if you have to be logged into lp to see the bug supervisor setting | 21:34 |
| clarkb | zaro: jeblair fungi updated the gearman-plugin change with a summary of what I have tested, my +2 stands it lgtm | 21:35 |
| *** arxcruz has joined #openstack-infra | 21:35 | |
| *** jpr has joined #openstack-infra | 21:35 | |
| anteaya | fungi: now I see it, thanks | 21:35 |
| clarkb | granted I wrote much of that change so my +2 should be taken with a bit of judgement on that | 21:37 |
| clarkb | but I say lets merge the change and get this on at least one jenkins master | 21:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Make sure wheel-mirror works on centos7 https://review.openstack.org/290146 | 21:38 |
| *** aysyd has quit IRC | 21:38 | |
| ianw | pabelanger: ^ if you can take on getting a centos7 wheel-build online, i can make sure it's producing the right thing | 21:39 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Fix heat template gate service errors https://review.openstack.org/290147 | 21:39 |
| sridhar_ram | fungi: thanks for the demystifying the workflow. I'll plan to create a 'tacker bug team' group and set that as the supervisor... | 21:39 |
| *** zul has joined #openstack-infra | 21:39 | |
| pabelanger | ianw: Ya, I think it would be a worth while effort | 21:39 |
| sridhar_ram | fungi: again, I need to add "OpenStack Administrators" as the owner of this group ? | 21:39 |
| sridhar_ram | fungi: I'm looking at https://launchpad.net/~nova-bugs which is the bug supervisor for nova | 21:40 |
| clarkb | ianw: pabelanger is afs working on centos7 now? | 21:40 |
| pabelanger | clarkb: ianw: I haven't tested it yet. But I did see a patch to support it | 21:40 |
| *** nwkarste_ has quit IRC | 21:40 | |
| fungi | sridhar_ram: if tacker is an official openstack project, then yes i think so. i've honestly never been 100% clear on when we require projects set openstack administrators as the owner in lp | 21:40 |
| fungi | ttx: jeblair: ^ was tehre ever a policy around that? | 21:40 |
| sridhar_ram | fungi: tacker (as of last week) is an official openstack project | 21:41 |
| ianw | clarkb: probably not, what nodes would use it atm? | 21:41 |
| *** nwkarsten has joined #openstack-infra | 21:41 | |
| clarkb | ianw: the one you are proposing to add | 21:41 |
| clarkb | ianw: for the wheel mirror | 21:41 |
| jeblair | fungi: i don't think it's a written policy, but i certainly think it's required | 21:41 |
| clarkb | ianw: I suppose that will be self testing? | 21:42 |
| ttx | fungi: that would be an undocumented policy | 21:42 |
| ianw | clarkb: yeah :) so totally expecting to debug that as one of the first issues | 21:42 |
| ttx | we should add that in the project team guide if it's not | 21:42 |
| *** tqtran has joined #openstack-infra | 21:42 | |
| fungi | ttx: jeblair: sridhar_ram: thanks, so sounds like consensus is yes, openstack administrators as project owner in lp for official projects | 21:42 |
| jeblair | oh er, team owner | 21:43 |
| *** baoli_ has joined #openstack-infra | 21:43 | |
| jeblair | i don't think we need to necessarily be the project owner, yeah? just own the team that owns the project | 21:43 |
| ttx | yep, undocumented, unless it's somewhere in the infra guide | 21:43 |
| ttx | not in the project team guide fwiw | 21:44 |
| fungi | ttx: it may be in the project creators howto | 21:44 |
| ttx | looking | 21:44 |
| fungi | yeah, it's in there | 21:44 |
| fungi | sridhar_ram: http://docs.openstack.org/infra/manual/creators.html#set-up-launchpad | 21:44 |
| ttx | it is documented! We are saved. | 21:45 |
| ttx | ENOTOURFAULT | 21:45 |
| fungi | we've started to forget that we've documented things! we're doomed! | 21:45 |
| sridhar_ram | fungi: thanks.. will take a look. infra team is saved.. ENOBODYTOBLAME | 21:45 |
| *** nwkarsten has quit IRC | 21:45 | |
| ianw | pabelanger / abregman : let's use https://etherpad.openstack.org/p/centos7-wheel-mirror for sync | 21:46 |
| jeblair | clarkb: cool, i went ahead and approved it; do you want to install your build or wait for it to land? | 21:46 |
| clarkb | jeblair: lets use the one from tarballs.o.o | 21:46 |
| *** baoli has quit IRC | 21:46 | |
| *** |-paul-| has quit IRC | 21:47 | |
| *** ihrachys has quit IRC | 21:47 | |
| sridhar_ram | ttx: fungi: related question .. we have one lp project for python-tackerclient and another lp project for both tacker and tacker-horizon. does that sound reasonable ? | 21:47 |
| ttx | yes | 21:49 |
| fungi | sridhar_ram: yes, though you'll want to update gerrit/projects.yaml in openstack-infra/project-config to set "groups: tacker" on the openstack/tacker-horizon repo | 21:49 |
| openstackgerrit | Merged openstack-infra/gearman-plugin: Register the diff of functions https://review.openstack.org/180371 | 21:49 |
| sridhar_ram | fungi: noting it down. thanks | 21:49 |
| fungi | that way tacker-horizon bugs are searched for in the tacker project when our bug update hook fires on those changes | 21:49 |
| fungi | otherwise it assumes the short repo name matches the lp project name | 21:49 |
| sridhar_ram | fungi: ah, I see | 21:50 |
| *** regXboi has quit IRC | 21:50 | |
| fungi | AJaeger: lifeless: discussing an issue with pre-release tarballs for neutron-vpnaas in #openstack-release, but it looks constraints related http://logs.openstack.org/07/070a154420e557bd9b6bd407280b5db14fe2124c/pre-release/neutron-vpnaas-tarball/4a00518/console.html.gz#_2016-03-03_10_09_42_810 | 21:52 |
| fungi | AJaeger: i see your https://review.openstack.org/288930 change altered some of this in that project a few days ago, so looking for some insight | 21:52 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add neutron lbaas dashboard to it's gerrit bot channel https://review.openstack.org/290137 | 21:52 |
| fungi | AJaeger: the {name}-tarball jobs used in pre-release and release pipelines don't set up the constraints file, but the tox.ini seems to expect it to be there | 21:53 |
| clarkb | jeblair: fungi http://tarballs.openstack.org/ci/gearman-plugin/gearman-plugin-0.1.3.7.c5637ce.hpi | 21:53 |
| *** nwkarsten has joined #openstack-infra | 21:54 | |
| *** infra-re_ has quit IRC | 21:55 | |
| jeblair | clarkb: woot | 21:55 |
| clarkb | jeblair: fungi I think you can install it via the web ui by dropping that url into the get plugin from here location | 21:55 |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add a bandit non-voting for Kolla https://review.openstack.org/290070 | 21:55 |
| clarkb | maybe start with jenkins01 or 02 since they are slower and it iwll be apparent how much this helps | 21:55 |
| fungi | clarkb: giving it a try on 01 now | 21:56 |
| *** jpr has quit IRC | 21:56 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 21:57 |
| clarkb | fungi: keep in mind that restarting the master may be required | 21:57 |
| clarkb | I can never figure out when that is or isn't required | 21:57 |
| jeblair | probably best to put both of those in shutdown mode | 21:57 |
| fungi | yeah, i'll do that | 21:57 |
| fungi | i also don't see a correct place to feed it an hpi url, seems to only allow uploading | 21:58 |
| jeblair | fungi: at our current usage, you could probably put 2 more in shutdown mode too | 21:58 |
| clarkb | fungi: let me find it | 21:58 |
| fungi | i'll just do 01-04 in that case | 21:58 |
| fungi | especiallt since 05-07 perform a bit better anyway | 21:59 |
| jeblair | ++ | 21:59 |
| *** baoli_ has quit IRC | 21:59 | |
| clarkb | fungi: oh I think it is upload but it will accept an http uri? | 21:59 |
| *** baoli has joined #openstack-infra | 21:59 | |
| clarkb | or you can dl locally then reupload to jenkins0* | 21:59 |
| *** matrohon has quit IRC | 22:00 | |
| openstackgerrit | Steven Dake proposed openstack-infra/project-config: Add non-voting bandit gate for kolla-mesos https://review.openstack.org/290157 | 22:00 |
| fungi | i'll do the latter if the former ends up not being an option | 22:00 |
| pabelanger | ianw: WFM. I'll look at JJB tomorrow and see how the wheel stuff works | 22:01 |
| *** sdake_ is now known as sdake | 22:03 | |
| fungi | okay, 01-04 are in prepare for shutdown. trying to upgrade the plugin on 01 now | 22:03 |
| *** baoli has quit IRC | 22:04 | |
| fungi | and yeah, the upload plugin option only has a browse button which pulls up a local dialogue browsing your local filesystem | 22:04 |
| clarkb | bah tripleo jobs not expected to finish for almost 2 hours | 22:05 |
| fungi | some of those take upwards of 3 to complete | 22:06 |
| fungi | yeah, it says jenkins needs to be restarted for the upgrade to take effect | 22:07 |
| fungi | i won't have it automatically restart jenkins and will leave that to us to do manually | 22:07 |
| fungi | uh, wow. when i try to uncheck that checkbox, it refreshes a second later and comes back re-checked | 22:08 |
| zaro | clarkb: looks like you guys merged the gearman plugin change. do you need a release? | 22:08 |
| clarkb | zaro: we will start with the hpi built by that merging | 22:09 |
| fungi | hobson's choice meets hudson's choice | 22:09 |
| clarkb | zaro: we should probably release if this works well for us to make it easier for others to use | 22:09 |
| jeblair | so... i just wondered why i haven't seen any tc related emails for a while.... | 22:10 |
| jeblair | and i just figured it out | 22:10 |
| fungi | you were subscribed to the tc list from your openstack.org address too? | 22:10 |
| jeblair | yup. | 22:10 |
| *** sdake_ has joined #openstack-infra | 22:11 | |
| zaro | clarkb: actually, i was thinking of approving this https://review.openstack.org/#/c/271543/ then doing a release. most everybody else wants update to work with latest(ish) jenkins. | 22:11 |
| *** shardy has quit IRC | 22:11 | |
| jeblair | i do not | 22:12 |
| fungi | okay, 01-04 are upgraded now, just a matter of waiting for jobs on them to complete | 22:12 |
| zaro | it's been verified by hashar and zxiiro. | 22:12 |
| jeblair | i only want to upgrade jenkins if it's broken | 22:12 |
| fungi | i'll also delete any ready nodes they've accumulated | 22:12 |
| hashar | yup https://review.openstack.org/#/c/271543/ works for Wikimedia | 22:12 |
| *** thorst_ has quit IRC | 22:12 | |
| jeblair | zaro: can you do that without requiring an upgrade? | 22:12 |
| hashar | but we were not affected by the issue it is fixing | 22:12 |
| *** sdake has quit IRC | 22:13 | |
| ianw | pabelanger: i wonder if there's really enough breaking changes that we can't use the ubuntu wheel's on centos anyway. we should probably be building a LCD wheel a-la dstufft's proposals that works "everywhere" (for sane versions of "everywhere") | 22:14 |
| ianw | i could even see docker being involved in this, which probably makes mordred happy :) | 22:15 |
| *** thorst_ has joined #openstack-infra | 22:15 | |
| pabelanger | ianw: I don't understand wheels enough to see if that is possible. Need to spend some cycles reading up on them | 22:15 |
| pabelanger | was hoping bringing online the centos7 wheel would help with some of that :) | 22:15 |
| fungi | ianw: discussed earlier, but the wheels where this cache is really beneficial to us are unlikely to be trivial to lcd without help from their upstreams, so are better done by them and published to pypi directly | 22:15 |
| fungi | it involves a lot of vendoring in the c libraries which are not part of the (very limited) manylinux1 list | 22:16 |
| fungi | and also means us maintaining something like centos 5 to do it in (whether that's via container images or whatever) | 22:16 |
| zaro | jeblair: i guess so, but we don't need to upgrade to that even if it does merge. i guess might be best to release 2 times? release a build not then one after that merge? | 22:17 |
| ianw | fungi: yes, fair point that if upstream makes those centos5 based wheels we can just consume them | 22:17 |
| zaro | *not/now | 22:17 |
| jeblair | zaro: i thought the commit message said it required a later version than what we run, but maybe i misunderstood | 22:17 |
| fungi | ianw: and making them ourselves when we're not upstream and don't have their perspective on what needs to be done to have a sane cross-platform wheel for them seems like a lot of extra responsibility to take on | 22:18 |
| *** thorst__ has joined #openstack-infra | 22:18 | |
| fungi | ianw: the thing is, the lengthy compile time wheels where this cache is of a real benefit to us are also likely to be some of the last packages to get manylinux1 working consistently or safely | 22:19 |
| *** thorst_ has quit IRC | 22:19 | |
| fungi | the ones where it's trivial to do also take almost no time to build from sdist | 22:19 |
| *** pvaneck has joined #openstack-infra | 22:20 | |
| zaro | jeblair: ok, yeah. i'll just hold off until i have tested that change with our current version of jenkins to make sure it still works. i think it should. | 22:20 |
| fungi | gonna take a break and grab some dinner while i wait for these remaining jobs on jenkins01-04 to burn down | 22:21 |
| jeblair | why are we talking about building lcd wheels when we can build actual wheels? | 22:21 |
| fungi | jeblair: i think the reason was so that we didn't have to maintain more than one wheel build worker | 22:21 |
| *** bgmccollum has left #openstack-infra | 22:21 | |
| fungi | but as i said, it's not really realistic | 22:21 |
| *** weshay has quit IRC | 22:21 | |
| *** annegentle has quit IRC | 22:22 | |
| fungi | for our use case | 22:22 |
| jeblair | we were certainly expecting to maintain more than one | 22:22 |
| *** thorst__ has quit IRC | 22:23 | |
| mordred | ya | 22:24 |
| mordred | I mean | 22:24 |
| mordred | that's why we made them worker nodes | 22:24 |
| *** hashar has quit IRC | 22:25 | |
| fungi | and manylinux1 wheels are generally something for the upstream authors of non-pure python packages to build and upload to pypi, not as reasonable a solution for people who are not redistributing wheels and have a limited number of platforms they need to support | 22:25 |
| *** ajmiller has quit IRC | 22:25 | |
| mordred | yah. sorry, I was AFK for forever - what's the problem with a centos wheel worker? | 22:25 |
| ianw | mordred: nothing, just getting started on it. afs is untested, we think | 22:25 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 22:26 |
| mordred | ianw: ah. kk | 22:26 |
| mordred | ianw: if you havne't seen it: https://gist.github.com/danifr/773d7b049454390d1e51 | 22:26 |
| mordred | cern wrote up things on using afs on centos7 (and other things) | 22:26 |
| fungi | aha, i didn't think about the fact that this means we get to work out other distros as afs clients | 22:26 |
| ianw | mordred: yeah, i'm hoping puppet knows about this (?) haven't looked | 22:27 |
| *** nwkarsten has quit IRC | 22:28 | |
| fungi | debian has very engaged openafs maintainers so the support on debian derivative distros is generally pretty seamless, but i have no idea how well non-debian distros keep it working for them | 22:28 |
| ianw | who knows, i might get wheels working *and* find the higgs boson | 22:28 |
| *** austin81 has quit IRC | 22:28 | |
| ianw | actually, they already found that, didn't they | 22:28 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Fix heat template gate service errors https://review.openstack.org/290147 | 22:29 |
| *** annegentle has joined #openstack-infra | 22:29 | |
| fungi | ianw: they found it a couple times in fact, but doesn't mean you can't find another | 22:29 |
| jeblair | the openafs project used to have builds for centos/rhel | 22:29 |
| *** jamielennox|away is now known as jamielennox | 22:29 | |
| jeblair | but i guess they aren't keeping those up to date anymore (as much)? | 22:30 |
| clarkb | jeblair: according to the gist you grab a source package and build it from source | 22:30 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Move tripleo.sh into tripleo-ci repo https://review.openstack.org/272210 | 22:30 |
| jeblair | right, here's the approach we used to use: https://review.openstack.org/273155 | 22:31 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Only reduce the node vm's when using introspection https://review.openstack.org/289973 | 22:31 |
| clarkb | is 1.6.11 newer than 1.6.7? odd that newer centos would have to use older afs | 22:32 |
| jeblair | clarkb: it is; perhaps the stopped building rhel7 but keep on with 5 and 6. | 22:33 |
| clarkb | mordred: are you back for a little while? want to update shade on nodepool.o.o and restart services again? there are ~4 images still uploading as of a minute ago | 22:34 |
| *** MarkAtwood has quit IRC | 22:34 | |
| *** amitgandhinz has quit IRC | 22:35 | |
| jeblair | https://lists.openafs.org/pipermail/openafs-info/2014-October/041076.html is relevant | 22:35 |
| mordred | clarkb: I'm at bad-internet-place | 22:35 |
| *** thorst_ has joined #openstack-infra | 22:35 | |
| mordred | clarkb: however, I restarted nodepool-builder last night with the patch installed | 22:35 |
| jeblair | mordred: i love that place | 22:35 |
| mordred | clarkb: so upgrading/restarting should be a no-op | 22:36 |
| clarkb | mordred: ya I mostly want to get off local patched before we forget | 22:36 |
| clarkb | mordred: but its not super urgetn | 22:36 |
| clarkb | mordred: I can give it a go if I catch when the images are done uploading | 22:36 |
| *** tpsilva has quit IRC | 22:37 | |
| *** silos has quit IRC | 22:37 | |
| ianw | jeblair: yeah, i think build-it-yourself is the gist i'm getting... | 22:37 |
| clarkb | mordred: also in theory the next time we pip install nodepool we will get latest shade then | 22:38 |
| clarkb | so it should self heal eventually | 22:38 |
| *** _ari_ has quit IRC | 22:39 | |
| *** thorst_ has quit IRC | 22:39 | |
| mordred | clarkb: yah | 22:39 |
| openstackgerrit | Merged openstack-infra/shade: Catch failures with particular clouds https://review.openstack.org/285882 | 22:39 |
| *** piet has quit IRC | 22:39 | |
| *** piet has joined #openstack-infra | 22:40 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:40 | |
| clarkb | maybe this is something we can worth with CERN on fixing better upstream? | 22:40 |
| clarkb | I have never built rpm packages so don't know that I am personally a great resources | 22:40 |
| jeblair | ianw: also this thread https://lists.centos.org/pipermail/centos-devel/2015-August/013730.html | 22:40 |
| jeblair | clarkb: ^ | 22:41 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump python-troveclient to 2.1.0 https://review.openstack.org/290168 | 22:43 |
| mriedem | dhellmann: ^ | 22:43 |
| clarkb | jeblair: sounds like it mostly just procedural stuff distro side | 22:43 |
| clarkb | openafs wants new packages for every kernel, centos wants new packages per point release | 22:45 |
| nibalizer | why do we need afs on centos? | 22:45 |
| mordred | nibalizer: wheel mirror | 22:45 |
| mordred | scuse me | 22:45 |
| nibalizer | oh blah | 22:45 |
| mordred | nibalizer: wheel builder | 22:45 |
| nibalizer | ya | 22:45 |
| nibalizer | just nfs from the ubuntu/afs to the centos host | 22:45 |
| nibalizer | call it good | 22:45 |
| mordred | nibalizer: I will stab you in the eye | 22:46 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Archive Ironic VM nodes console logs for 'old' https://review.openstack.org/290171 | 22:46 |
| mordred | nibalizer: with an eye | 22:46 |
| greghaynes | mordred: The wheel builder and mirror need to be colocated? | 22:46 |
| nibalizer | mordred: right but does your stabbing device have rpm packages | 22:46 |
| ianw | nibalizer: needs more samba | 22:46 |
| clarkb | greghaynes: the wheel mirror needs to write to the afs volume | 22:46 |
| mordred | greghaynes: no. but that is how we've implemented it currently | 22:46 |
| clarkb | greghaynes: the ideal way to do that is mount the afs volume and write to it | 22:46 |
| mordred | ++ | 22:46 |
| greghaynes | ah, yea | 22:47 |
| clarkb | er wheel builder | 22:47 |
| mordred | also, before we rube-goldberg a nibalizer solution, we should at least fail at getting AFS on centos :) | 22:47 |
| greghaynes | so other option is to build elsewhere and rsync, but I get the desire for simplicity | 22:47 |
| ianw | however, an scp ... | 22:47 |
| mordred | nope | 22:47 |
| clarkb | https://lists.centos.org/pipermail/centos-devel/2015-August/013747.html is the last message | 22:47 |
| mordred | scp would mean we'd have to do weird things with scp and kerberos tickets | 22:47 |
| *** ddieterly has quit IRC | 22:47 | |
| mordred | and then the centos builder would be completely different than the ubuntu builder | 22:48 |
| *** thiagop has quit IRC | 22:48 | |
| *** mriedem has quit IRC | 22:49 | |
| ianw | mordred: yeah, but it's going to be different anyway because we're seemingly going to have to maintain afs-ish stuff | 22:49 |
| nibalizer | we could build all kinds of rube goldberg machines to get the data from a centos host to an ubuntu host | 22:50 |
| mordred | ianw: totally | 22:50 |
| nibalizer | but the correct solution is to get afs on centos | 22:50 |
| ianw | to-be-determined -- i think the big problem with rolling out build-your-own modules is if you're doing it thousands of times, a-la cern | 22:50 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Only unset proxy for deploy command https://review.openstack.org/289556 | 22:50 |
| mordred | but we could potentially have the "maintain AFS" stuff be a thing that doesn't affect the "build wheels" thing | 22:50 |
| mordred | ianw: ++ | 22:50 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on default values on ValidationException Constructor https://review.openstack.org/290174 | 22:51 |
| ianw | i feel like we can probably let dkms handle it, and forget about it. which is i bet what everyone else does, and why it has got to that state | 22:51 |
| jeblair | yeah, to be honest, dkms is great, and people who care about the inefficiency (ie, not most) have the resources to do their own mirror | 22:52 |
| nibalizer | we dkms for ubuntu iirc | 22:52 |
| jeblair | yep | 22:52 |
| *** ccrouch has quit IRC | 22:53 | |
| mordred | oh - centos has dkms too? | 22:53 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on default values on ValidationException Constructor https://review.openstack.org/290174 | 22:53 |
| clarkb | nibalizer: but doesn't debuntu provide dkms packaging out of the box? | 22:54 |
| clarkb | nibalizer: we use it because its the thing, centos has no "thing" | 22:54 |
| jeblair | clarkb: yes that's correct | 22:54 |
| jeblair | we just use what debuntu provides which is dkms-based packages | 22:54 |
| clarkb | if we are stuck making our own thing, then I agree dkms seems like a good option to avoid the troubles in that long thread jeblair linked | 22:55 |
| jeblair | (also, put this on the "bad ideas" pile: use this repo: http://cbs.centos.org/repos/storage7-testing/x86_64/os/Packages/ ; and what is that for? why is centos building packages that aren't in the distro? i don't understand) | 22:55 |
| *** arxcruz has quit IRC | 22:56 | |
| ianw | alright, enough talking :) i'm going to bring up some centos and build this stuff and see what i can make work -> i'll put notes in https://etherpad.openstack.org/p/centos7-wheel-mirror | 22:56 |
| *** sdake_ has quit IRC | 22:56 | |
| clarkb | jeblair: if I have read that thread correctly they are building it but need to figure out how to have a new package every kernel release while keeping old packages for some reason | 22:56 |
| clarkb | jeblair: and until that happens its not in the main distro? | 22:56 |
| mordred | jeblair: https://wiki.centos.org/HowTos/CommunityBuildSystem | 22:57 |
| Clint | mordred: have you demanded that ibm relicense afs yet | 22:57 |
| jeblair | clarkb: it seems to me that ignoring that "requirement" would make things a lot easier :) | 22:57 |
| clarkb | jeblair: I agree :) | 22:57 |
| mordred | clarkb, jeblair: maybe we should use our insane clout in the world to convince distros to stop deleting packages from their archives | 22:58 |
| mordred | because, I konw that 10 years ago hard-drive space was hard to find | 22:58 |
| mordred | but I'm pretty sure that it's not anymore | 22:58 |
| mordred | so maybe we should stop optimizing for an i386 with 500M of hard drive | 22:58 |
| mordred | Clint: I don't have THAT much clout | 22:59 |
| * Clint sniffs. | 22:59 | |
| jeblair | ianw: ++ | 23:00 |
| mordred | ianw: +100 | 23:00 |
| *** _sarob has joined #openstack-infra | 23:00 | |
| *** ccrouch has joined #openstack-infra | 23:01 | |
| clarkb | I almost want ot put tripleo jobs on their own jenkins master | 23:01 |
| clarkb | then we can restart everything else faster | 23:01 |
| clarkb | ETA 49 minutes on jenkins01 | 23:02 |
| *** _sarob has quit IRC | 23:02 | |
| *** _sarob has joined #openstack-infra | 23:02 | |
| *** sarob has quit IRC | 23:02 | |
| *** annegentle has quit IRC | 23:05 | |
| *** annegentle has joined #openstack-infra | 23:05 | |
| *** baoli has joined #openstack-infra | 23:07 | |
| *** baoli has quit IRC | 23:07 | |
| StevenK | mordred: i386 with 500M is not fair, given the archive is a few hundred gigabytes | 23:07 |
| jhesketh | jeblair: any chance you have some spare cycles today to pick up on the zuulv3 discussion from yesterday? | 23:08 |
| mordred | StevenK: oooh. so it'll still fit on my laptop then? | 23:08 |
| *** cloudtrainme has quit IRC | 23:09 | |
| StevenK | mordred: Depending on what you want to mirror, it can fit. Roughly one suite and arch is ~40G | 23:09 |
| *** _sarob has quit IRC | 23:09 | |
| *** sarob has joined #openstack-infra | 23:09 | |
| mordred | StevenK: yah. thus my suggestion that distros stop aggressively pruning their repos | 23:10 |
| clarkb | down to 2 image uploads | 23:10 |
| mordred | StevenK: it'sn action that solves a problem that no longer exists | 23:10 |
| mordred | StevenK: and makes amore things hurt than are necessary | 23:10 |
| jeblair | jhesketh: what's up? | 23:10 |
| StevenK | mordred: in Ubuntu and Debian's case, a significant number of mirror operators have said "Grow any larger, and we can't mirror you any more" | 23:11 |
| mordred | StevenK: k | 23:11 |
| *** sc68cal has quit IRC | 23:11 | |
| clarkb | and now 1 | 23:11 |
| *** sdake has joined #openstack-infra | 23:11 | |
| mordred | StevenK: remind me again why there are ubuntu mirrors? | 23:12 |
| *** salv-orlando has joined #openstack-infra | 23:12 | |
| jhesketh | jeblair: we were discussing the merger component... should we drop the separate merger daemon and have the server clone the repos it needs to pull config from itself | 23:12 |
| jhesketh | perhaps in threads | 23:12 |
| *** sdague has quit IRC | 23:12 | |
| mordred | StevenK: given that it's a product from a company and that company actively asserts trademark infringement on people redistributing? | 23:12 |
| jhesketh | granted ,that doesn't scale as well now we have the cat function | 23:12 |
| jeblair | jhesketh: oh, no, i think it's important that happen off-server | 23:12 |
| mordred | StevenK: even though that action is protected under the GPL? | 23:12 |
| mordred | StevenK: but with debian ... fair | 23:13 |
| mordred | StevenK: although - I betcha infra could run a debian mirror that doesn't delete :) | 23:13 |
| jeblair | jhesketh: so i think if we want to drop the mergers (which i think we do) we should probably have the merger-launchers implement cat | 23:13 |
| clarkb | mordred: aiui it isn't redistributing so much as rebranding that is the issue, so mirrors should be fine | 23:13 |
| StevenK | mordred: There are a large number of Canonical and ex-Canonical people watching this argument WRT Ubuntu redistributing and sighing a LOT | 23:13 |
| clarkb | mordred: our ubuntu mirrors are clearly ubuntu and we don't alter anything in them | 23:13 |
| *** markvoelker has quit IRC | 23:14 | |
| jeblair | watch what happens if someone asks us to stop mirroring ubuntu (hint: we will) | 23:14 |
| mordred | StevenK: good. I'm glad they agree with me that the stance is exasperating :) | 23:14 |
| fungi | done eating and jenkins01-04 are almost idle now | 23:14 |
| clarkb | fungi: ya its mostly tripleo jobs at this point | 23:14 |
| StevenK | mordred: In a word, "OMG Yes" | 23:15 |
| *** yamahata has joined #openstack-infra | 23:15 | |
| jhesketh | jeblair: so we don't really have a merger-launcher... am I mis-remembering or is this a planned thing? | 23:15 |
| mordred | StevenK: that said - snark aside - I'd personally love to figure out a better/different solution to the distro mirror problem | 23:15 |
| jeblair | jhesketh: (to elaborate, i'm imagining a *large* number of repos will need to be consulted for configs, so i think having that be horizontally scaling is important) | 23:15 |
| Clint | clarkb: it's more that the mirrors aren't a revenue stream | 23:15 |
| mordred | StevenK: given that there is only one major non-corporate-backed distro anymore | 23:15 |
| jhesketh | jeblair: agreed | 23:15 |
| jhesketh | jeblair: it's already a *large* number ;-) | 23:16 |
| mordred | Clint: yup. but cloud-images are | 23:16 |
| Clint | i've heard that | 23:16 |
| jeblair | jhesketh: heh | 23:16 |
| *** salv-orlando has quit IRC | 23:16 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| jeblair | jhesketh: the ansible launcher is a merger-launcher | 23:17 |
| jeblair | jhesketh: see http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#execution pgraphs 2 and 3 | 23:17 |
| jeblair | jhesketh: also, i sketched that out in the initial implementation | 23:17 |
| jeblair | jhesketh: i'm being loose with terminology here... i used merger-launcher to indicate that the work of the merger is generally being subsumed into the launcher | 23:18 |
| jhesketh | jeblair: yep, I'm aware of that, just didn't realise you were referring to the ansible launcher when you said "merger launcher" | 23:18 |
| jeblair | it's not really a standalone merger at all | 23:18 |
| jhesketh | (I knew it did the merge) | 23:18 |
| jhesketh | jeblair: right, I follow now | 23:18 |
| *** xyang1 has quit IRC | 23:18 | |
| clarkb | fungi: one job remaining on jenkins01 | 23:18 |
| jeblair | ok, yeah, my fault for making it muddier instead of clearer with that :) | 23:18 |
| jhesketh | jeblair: so if we have a merger-launcher to cat jobs it's going to end up looking a lot like the current mergers... | 23:19 |
| jhesketh | jeblair: maybe we should wait until the launches mature a little and see if it's worth dropping the current mergers then | 23:19 |
| jeblair | jhesketh: oh, well, i just meant that the ansible launchers could also implement cat | 23:19 |
| clarkb | fungi: and its done, that was fast | 23:19 |
| *** ddieterly has joined #openstack-infra | 23:19 | |
| jeblair | jhesketh: it's not really their main function, but they are well equipped to do it at minimal cost | 23:19 |
| jhesketh | hmm okay | 23:19 |
| jhesketh | jeblair: so the ansible launchers were the next thing I wanted to discuss | 23:20 |
| jeblair | jhesketh: (and during initial config load, they will all be idle so there's no waiting; during a reload they may take a bit to respond, but i think that's okay) | 23:20 |
| jhesketh | jeblair: firstly, they are designed to be long lived right? | 23:20 |
| jeblair | jhesketh: yes | 23:20 |
| jhesketh | jeblair: that's true (re config loading) | 23:20 |
| clarkb | fungi: 03 is done as well | 23:21 |
| jhesketh | jeblair: so it might be nitty, but I think it's helpful to have the terminology correct... as far as the zuul scheduler is concerned, it asks for a set of work (aka job) to be done and the ansible launcher goes away and does it | 23:21 |
| clarkb | fungi: is the plugin updated on all of 01-04? | 23:21 |
| jeblair | jhesketh: right | 23:21 |
| jhesketh | jeblair: from the point of view that the ansible launcher is launching ansible I guess you'd call it a launcher, but as far as zuul is concerned it's just doing work | 23:21 |
| EmilienM | hello infra, if someone can approve my patch on login.defs https://review.openstack.org/#/c/288244/ - thanks a lot | 23:21 |
| jhesketh | jeblair: and for all zuul cares it could be jenkins or turbo hipster too (so long as they understand the new job format) | 23:22 |
| jhesketh | jeblair: so the nit is, should we call it a worker or something else (ie, not a launcher) | 23:22 |
| *** kzaitsev_mb has quit IRC | 23:22 | |
| fungi | StevenK: mordred: probably a more appropriate datapoint is debian's struggle to maintain sufficient disk space for snapshot.debian.org over the years | 23:22 |
| fungi | though i don't know whether it's a current challenge, as i'm not closely in touch with any of the dsa | 23:22 |
| StevenK | Oh yes | 23:22 |
| mordred | fungi: so, you're saying that we should just donate some cloud servers with some cinder to debian from Infra? | 23:23 |
| StevenK | Currently, snapshot consists of 24 terabytes of data in about 15 million files, and it appears to be growing at a rate of approximately 5 terabytes a year (or about 10 megabytes per minute). | 23:23 |
| StevenK | That's as of 2014 | 23:23 |
| *** annegentle has quit IRC | 23:23 | |
| mordred | sounds like a good job for AFS | 23:23 |
| StevenK | So I daresay the numbers have increased | 23:23 |
| david-lyle | quick ATC question. Does someone who is listed solely as a co-author on merged patches attain ATC status? | 23:23 |
| mordred | david-lyle: not automatically, but ATC status can be requested | 23:24 |
| david-lyle | I've heard no, but wanted to check | 23:24 |
| clarkb | jeblair: speaking of large data and AFS, do you have a plan for how we would shard the logs across afs volumes? | 23:24 |
| mordred | david-lyle: by the extra-atc process | 23:24 |
| mordred | clarkb: carefully | 23:24 |
| clarkb | jhesketh: I am not sure if you hvae dug into the distribution of logs | 23:24 |
| david-lyle | mordred: ok, I've done that before will go that route, thanks | 23:24 |
| clarkb | er jeblair ^ | 23:24 |
| mordred | david-lyle: woot | 23:24 |
| fungi | clarkb: cool, it looks like the checkbox didn't actually make it restart them anyway? | 23:24 |
| clarkb | fungi: jenkins special | 23:24 |
| anteaya | david-lyle: unfortunately for this round of elections, the eligibility window is closed | 23:25 |
| fungi | clarkb: and yes, upgraded on all 4 of them pending restart | 23:25 |
| anteaya | david-lyle: but put them in anyway and they will be eligible for next round | 23:25 |
| clarkb | fungi: cool I will delete the two slaves on 01 that are offline | 23:25 |
| david-lyle | anteaya: I'm not sure I care about elections as much as credit where it's due | 23:25 |
| jeblair | jhesketh: i think the directly-connected worker approach is more problematic in this situation, since we won't be registering individual jobs anymore | 23:25 |
| anteaya | david-lyle: okay great, there is no deadline for credit where it is due | 23:26 |
| clarkb | fungi: 01 should be reday for a restart | 23:26 |
| jeblair | jhesketh: so i thought the name change appropriate to reflect that is the intended design | 23:26 |
| david-lyle | anteaya: but good to know | 23:26 |
| david-lyle | anteaya: I'll pass that on too | 23:26 |
| anteaya | yup | 23:26 |
| clarkb | fungi: 03 as well | 23:26 |
| jhesketh | clarkb: in terms of how AFS works? No, I haven't really poked at that (probably hence the questions in the review) | 23:26 |
| fungi | clarkb: okay, restarting them now | 23:26 |
| jeblair | jhesketh: (and yes, theoretically we could make a jenkins launcher, i just don't want to design for it) | 23:26 |
| *** angdraug has quit IRC | 23:26 | |
| clarkb | jhesketh: I tab completed 'j' wrong I have the same question you did | 23:26 |
| jhesketh | clarkb: :-) | 23:26 |
| anteaya | tristanC tonyb would you like to spruce up the governance wikipage perhaps? https://wiki.openstack.org/wiki/Governance with links to current election wikipages? | 23:27 |
| clarkb | we have 100 "shards" already in the filesystem structure | 23:27 |
| jeblair | clarkb: traditionally, you'd make a bunch of volumes, and mount them.... | 23:27 |
| clarkb | but 100 afs volumes seems like overkill | 23:27 |
| jeblair | clarkb: yeah, so theoretically, if we wanted 100 volumes, we could do that. if we wanted fewer, we could, say, have 10 volumes and use symlinks. | 23:28 |
| jeblair | clarkb: however, it's worth noting universities, etc, have 2 volumes for every user, with 50,000 or more users | 23:28 |
| tonyb | anteaya: sure | 23:29 |
| anteaya | tonyb: thanks | 23:29 |
| jeblair | so 100 probably won't kill our install | 23:29 |
| fungi | clarkb: okay, 01 and 03 are on their way back up now | 23:29 |
| jhesketh | jeblair: fair enough, it's a minor thing (and I think launcher does fit well in some regards).. hypothetically though (and what I was leading to) if it were called or thought of as a worker, then we could think of the merge and cat commands as jobs. It's not a perfect fit but it's perhaps worth thinking of if we want to drop the merger daemons. | 23:29 |
| *** _nadya_ has joined #openstack-infra | 23:29 | |
| jeblair | jhesketh: that's an interesting idea :) | 23:29 |
| clarkb | fungi: I guess pick one and run jjb against it? | 23:29 |
| tristanC | anteaya: good idea, i'm on it | 23:30 |
| clarkb | fungi: in theory this should work fine with the server not in shutdown mode with the new plugin in place | 23:30 |
| jhesketh | jeblair: maybe the cat becomes a more generic way for the scheduler to get results or data back from a job | 23:30 |
| fungi | clarkb: yeah, since 01 is usually consistently terrible, i'll try it (once i see it running a job or two) | 23:30 |
| *** kevinsho_ has quit IRC | 23:30 | |
| anteaya | tristanC: great, well so is tonyb | 23:30 |
| jeblair | jhesketh: let's say the plan for the moment is to just have the launchers implement cat; however, i like the idea of revisiting that when things are fleshed out a bit more. it's entirely internal so would be easy for us to do. | 23:30 |
| anteaya | perhaps you should co-ordinate so as not to unedit each other | 23:30 |
| clarkb | fungi: it just started at least one job so that part of gearman is working | 23:30 |
| tonyb | if it's a race tristanC will win ;P | 23:30 |
| anteaya | tonyb: :) | 23:31 |
| fungi | clarkb: yeah, 01 has half a dozen underway now. i'll go ahead and jjb that sucker | 23:31 |
| jhesketh | jeblair: I was just typing the same thing :-). I'll roll onwards with the ansible stuff and ignore the mergers in the scheduler until we have a more mature job worker | 23:31 |
| jhesketh | eh, launcher | 23:31 |
| jeblair | heh, nicely done ;) | 23:32 |
| * jhesketh isn't sure the naming will click with his interpretation of what it does :P | 23:32 | |
| *** andreykurilin__ has quit IRC | 23:33 | |
| tristanC | anteaya: tonyb: it should be good now | 23:33 |
| anteaya | tristanC: thank you | 23:33 |
| * anteaya refreshes | 23:33 | |
| openstackgerrit | Max V.K. proposed openstack-infra/jenkins-job-builder: upd google-cloud-storage plugin support https://review.openstack.org/287497 | 23:33 |
| *** gordc has quit IRC | 23:34 | |
| *** _nadya_ has quit IRC | 23:34 | |
| fungi | clarkb: as usual i have it going under time, so i'll have some duration details once it (hopefully!) completes | 23:34 |
| *** sarob has quit IRC | 23:34 | |
| *** Qiming has joined #openstack-infra | 23:34 | |
| clarkb | fungi: awesome, I expect that the initial compile and figure out whic hthings need updating step will be the longest portion of the updates now | 23:35 |
| clarkb | fungi: but we shall see | 23:35 |
| anteaya | tristanC: it looks lovely, thank you | 23:35 |
| fungi | clarkb: if that ends up being the longest part, we're in great shape ;_ | 23:35 |
| *** markvoelker has joined #openstack-infra | 23:35 | |
| *** sarob has joined #openstack-infra | 23:35 | |
| *** sarob has quit IRC | 23:35 | |
| anteaya | tristanC tonyb since that is the page that links to our wiki main page: https://wiki.openstack.org/wiki/Main_Page that is my goto page for governance info | 23:35 |
| clarkb | 02 is down to one job and 04 is also down to one job | 23:36 |
| anteaya | not sure if other people use it much or not | 23:36 |
| *** sarob has joined #openstack-infra | 23:36 | |
| *** kencjohnston has quit IRC | 23:37 | |
| fungi | clarkb: yeah, once i get 02 and 04 restarted, i'll put 05-07 in graceful shutdown | 23:37 |
| clarkb | 04 is done, just waiting for nodepool to cleanup the last slave on it | 23:37 |
| fungi | clarkb: wow, that remaining job on 02 is projected to wrap up at ~2.5 hours (some 45 minutes from now) | 23:38 |
| fungi | cool, 04 looks clean now. restarting | 23:38 |
| *** markvoelker_ has joined #openstack-infra | 23:38 | |
| *** markvoelker has quit IRC | 23:40 | |
| *** amotoki has quit IRC | 23:40 | |
| fungi | jenkins01 jjb update is on to reconfiguring jobs now. it's in the c's | 23:41 |
| pleia2 | btw, noticed this in an women of openstack email that went out about the summit so I let them know I'd be happy to help out (seemed like something that could use an infra person hanging about) https://www.eventbrite.com/e/git-and-gerrit-lunch-and-learn-tickets-22552560315 | 23:41 |
| openstackgerrit | Sridhar Ramaswamy proposed openstack-infra/project-config: Remove groups for python-tackerclient https://review.openstack.org/290185 | 23:42 |
| anteaya | pleia2: thanks for being willing to attend | 23:43 |
| anteaya | hopefully you can help them avoid some of the trip hazards :) | 23:43 |
| pleia2 | anteaya: indeed :) or get out of them | 23:44 |
| clarkb | sandbox should be working again too | 23:44 |
| anteaya | clarkb: yay sandbox | 23:45 |
| clarkb | fungi: how far along is the jjb update? | 23:45 |
| anteaya | pleia2: :) you are so good at helping folks climb out of holes | 23:45 |
| ddieterly | when do post_test_hook and pre_test_hook run in relation to the tempest plugin tests in a gate job? | 23:45 |
| anteaya | ddieterly: depends on how the job is set up | 23:45 |
| ddieterly | is there some docs on this? | 23:46 |
| anteaya | usually people configure the job so that the pre hook runs before the tests and the post hook runs after | 23:46 |
| clarkb | typical order is devstack gate sets up repos, pre test hook runs, gate hook runs (which by default runs devstack and tempest), and then the post test hook runs | 23:46 |
| ddieterly | ok, that makes sense, thanks | 23:46 |
| anteaya | ddieterly: have you read this? http://docs.openstack.org/developer/tempest/plugin.html | 23:47 |
| fungi | clarkb: it's on the gate-c's now | 23:47 |
| *** dimtruck is now known as zz_dimtruck | 23:47 | |
| ddieterly | many times | 23:47 |
| anteaya | ddieterly: what kind of docs were you looking for? | 23:47 |
| ianw | clarkb / mordred / fungi / jeblair : so i see two options -> 1) build an AFS repo in a periodic job, publish it to static.openstack.org, consume that to install afs. 2) modify puppet-afs to build packages from source. preferences? | 23:47 |
| *** armax has quit IRC | 23:48 | |
| *** markvoelker_ has quit IRC | 23:48 | |
| *** dalgaaf has quit IRC | 23:48 | |
| clarkb | ianw: and when you say modify puppet to intsall from source you would use the source rpms that openafs publishes? | 23:48 |
| jeblair | clarkb, jhesketh: responded on 269928; let me know if that adds clarity or if more is needed | 23:48 |
| ianw | clarkb: yes. install source rpm + deps, rpmbuild it, have packages | 23:49 |
| clarkb | ianw: I think that would be my preference, but I don't know how big of a compile it is | 23:49 |
| jeblair | it was probably huge 30 years ago ;) | 23:49 |
| ianw | clarkb: about 10 minutes of build time | 23:49 |
| ianw | that was with no -j | 23:49 |
| clarkb | for a once in a while task that seems fine | 23:50 |
| fungi | ianw: while we do use puppet to build and install a few things from source, it's not well suited to that. then again, it's something we mostly already do while autobuilding and publishing our own distro packages currently is not | 23:50 |
| clarkb | just make sure puppet doesn't time it out | 23:50 |
| clarkb | fungi: well in this case it should be better because the build is represented in an rpm? | 23:50 |
| clarkb | maybe? | 23:50 |
| ianw | fungi: yes, i don't actually know how i would make puppet do that. my preference would be to have a fairly simple script run on a centos7 node do the build and copy into a repo somewhere static | 23:50 |
| fungi | clarkb: right, this is fairly similar to "build and install python package from source" | 23:51 |
| ianw | that way, the packages could be consumed globally if a centos7 needs afs access, even if it isn't puppeted | 23:51 |
| fungi | ianw: is there a one-liner command that will render binary packages from an rpm source package tree and install them? | 23:51 |
| openstackgerrit | James Polley proposed openstack-dev/pbr: Clarifications around tags and version numbers https://review.openstack.org/289630 | 23:52 |
| *** Jeffrey4l has quit IRC | 23:52 | |
| openstackgerrit | James Polley proposed openstack-dev/pbr: Correct typo - s/enabeld/enabled/ https://review.openstack.org/289623 | 23:52 |
| ianw | fungi: no, it's a few steps. you got to install the srpm then rpmbuild it | 23:52 |
| fungi | though you do make a compelling argument for downstream reusability of the packages (as long as that's something we want to get ourselves into) | 23:52 |
| *** nikhil_k has quit IRC | 23:52 | |
| *** ddieterly is now known as ddieterly[away] | 23:54 | |
| ianw | fungi: i don't see downstream being the whole world -- obviously we reserve the right to break it as it's really for our infra | 23:54 |
| jeblair | how does that compare to the (bad) idea of just using http://cbs.centos.org/repos/storage7-testing/x86_64/os/Packages/ ? | 23:54 |
| ianw | jeblair: it's going to be a pretty trivial script to build it and create a repo, and we know it's pinned then | 23:55 |
| lifeless | fungi: related, but not very? the file doesn't exist... | 23:55 |
| anteaya | my dryer which is 9 months old may have given up the ghost | 23:56 |
| anteaya | my soggy whites might just be rolling around to no benefit | 23:56 |
| fungi | lifeless: related insofar as the recent change that went into neutron-vpnaas tox.ini to support using constraints for everything forgot there are still some pipelines where we run tox but can't support constraints yet | 23:56 |
| anteaya | this makes me sad | 23:56 |
| fungi | lifeless: so probably it needs some sort of fallback/safechecking still | 23:56 |
| jeblair | fungi: the zuul gear server is not overwhelmed | 23:57 |
| ianw | fung / jeblair : i'll write up the script, do a bit of testing, we can then discuss on something concrete | 23:57 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Updating the meeting chair to Ales Komarek for the 3/8/16 meeting. https://review.openstack.org/289964 | 23:57 |
| fungi | jeblair: that's an excellent sign, though also i'm only running jjb on one jenkins master so far, and it's teh slowest of them | 23:58 |
| fungi | not to say puppet isn't also trying to fire up jjb on other jenkins masters | 23:58 |
| craige | o/ | 23:59 |
| *** Swami has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!