| *** dims has joined #openstack-infra | 00:00 | |
| *** davidhadas has joined #openstack-infra | 00:00 | |
| jeblair | mordred, clarkb: about 1 month ago, our rate of discarding devstack vms because it took too long for them to boot was 0.17. it is now 0.93. | 00:00 |
|---|---|---|
| jeblair | mordred, clarkb: that's a big change in hpcloud behavior | 00:00 |
| mordred | jeblair: oy | 00:01 |
| mordred | jeblair: does 0.93 mean we discard 93% of them? | 00:01 |
| jeblair | mordred: ye | 00:01 |
| jeblair | s | 00:01 |
| *** Ryan_Lane has quit IRC | 00:01 | |
| jeblair | mordred: yesterday we discarded 22884 and kept 1605 | 00:01 |
| *** Ryan_Lane has joined #openstack-infra | 00:02 | |
| *** Ryan_Lane has quit IRC | 00:02 | |
| mordred | wow | 00:02 |
| fungi | holy crap that's broken | 00:02 |
| *** Ryan_Lane has joined #openstack-infra | 00:02 | |
| mordred | WOW | 00:02 |
| jeblair | mordred: a month ago we discarded 1117 and kept 5366 (we kept more because we needed more because we were very busy) | 00:02 |
| jeblair | so it's actually increased as our load has decreased | 00:02 |
| mordred | what's our timeout? | 00:02 |
| jeblair | 60 seconds | 00:02 |
| jeblair | i don't have actual values, so i can't like do a box plot of what we actually see.... sorry :( | 00:03 |
| clarkb | jeblair: that is why I wrote that patch | 00:03 |
| clarkb | jeblair: I think you merged it, it allows us to configure a timeout in nodepool | 00:03 |
| jeblair | clarkb: yeah, and we may need to use it. i'm trying to be a good hpcloud customer though | 00:03 |
| jeblair | and also, i'm slightly worried about how long a box that takes > 60 seconds to start sshd will take to run tempest | 00:04 |
| fungi | i guess knowing a rough sla for vm provisioning in hpcloud might allow us to set more reasonable expectations in our tooling | 00:04 |
| clarkb | jeblair: ya, its good to communicate with hpcloud, I just realized that hardcoding 60 seconds probably won't work in all cases | 00:04 |
| clarkb | jeblair: and it was a fairly straightforward patch | 00:04 |
| fungi | oh, wait, this is 60 seconds to boot *after* hpcloud says the vm is ready? | 00:05 |
| jeblair | fungi: yes | 00:05 |
| fungi | holy crap that's broken | 00:05 |
| jeblair | fungi: it may involve network weirdness too (perhaps it take 30 seconds to boot and another 30 seconds for whatever network madness they have going on to converge) | 00:05 |
| jeblair | fungi: we get 2 different network errors while trying to connect, which makes me suspect that | 00:06 |
| fungi | 5 seconds to boot and 55 seconds for the rube goldberg machine to update their cgnat | 00:06 |
| jeblair | ya | 00:06 |
| fungi | so openstack(nova) thinks everything's all set and then... clunky network happenings | 00:07 |
| jeblair | mordred, clarkb: can one of you attempt to convey that message? we can patch nodepool to keep more detailed timings, but i probably won't get to that till next week | 00:08 |
| mordred | yes. conveying now | 00:08 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Visualize Zuul's NNFI scheduler https://review.openstack.org/47569 | 00:11 |
| jeblair | that may not be ready for merging ^ but it's probably ready for review | 00:11 |
| mordred | jeblair: whe does the 60 second timeout count begin? | 00:12 |
| jeblair | I want to see more complicated scenarios with that before we merge it, just to make sure it doesn't splat nonsense on the screen. | 00:12 |
| mordred | jeblair: after the node returns OK? | 00:12 |
| jeblair | mordred: when the vm becomes ACTIVE | 00:12 |
| *** ryanpetrello has quit IRC | 00:13 | |
| mordred | jeblair: thank you | 00:14 |
| mordred | jeblair: you don't happen to have a graph showing this do you? | 00:14 |
| *** Ryan_Lane has quit IRC | 00:18 | |
| *** Ryan_Lane has joined #openstack-infra | 00:18 | |
| jeblair | mordred: no, i just did some greps; i can make a chart and graph real quick if it would help | 00:21 |
| mordred | jeblair: not if it's going to be effort - if you have one already I thought that some folks like pictures and stuff | 00:22 |
| *** sarob has quit IRC | 00:22 | |
| *** sarob has joined #openstack-infra | 00:23 | |
| *** gyee has quit IRC | 00:24 | |
| *** bodepd_ has quit IRC | 00:27 | |
| *** bodepd has quit IRC | 00:27 | |
| *** sarob has quit IRC | 00:27 | |
| *** bodepd has joined #openstack-infra | 00:27 | |
| *** nicedice_ has joined #openstack-infra | 00:30 | |
| *** michchap has quit IRC | 00:34 | |
| *** michchap has joined #openstack-infra | 00:34 | |
| *** sarob has joined #openstack-infra | 00:41 | |
| *** sarob_ has joined #openstack-infra | 00:44 | |
| *** sarob has quit IRC | 00:44 | |
| *** ericw has quit IRC | 00:45 | |
| *** ericw has joined #openstack-infra | 00:49 | |
| *** sandywalsh_ has quit IRC | 00:53 | |
| *** wchrisj has quit IRC | 00:54 | |
| *** zehicle has quit IRC | 00:56 | |
| *** davidhadas has quit IRC | 00:59 | |
| *** sarob_ has quit IRC | 01:00 | |
| *** davidhadas has joined #openstack-infra | 01:00 | |
| lifeless | how does one make http://activity.openstack.org/data/display/OPNSTK2/Technical+Contributors update company affiliation? | 01:04 |
| *** zehicle_at_dell has quit IRC | 01:08 | |
| *** zehicle has joined #openstack-infra | 01:09 | |
| *** ericw has quit IRC | 01:10 | |
| *** ericw has joined #openstack-infra | 01:15 | |
| clarkb | lifeless: reed is the person to ask, he doesn't seem to be on irc currently | 01:15 |
| *** ericw has quit IRC | 01:16 | |
| lifeless | clarkb: I need gearman-job-server + the python gearman package, right? | 01:18 |
| lifeless | clarkb: anything else? | 01:18 |
| *** Ryan_Lane has quit IRC | 01:18 | |
| *** Ryan_Lane has joined #openstack-infra | 01:18 | |
| clarkb | lifeless: you can use just the python "gear" package, it comes with client, worker, and server code | 01:19 |
| clarkb | server code was initially for testing, but it works pretty well and handilly does the queueing method we prefer without extra config (C server needs some fiddling with to do the right queueing) | 01:20 |
| lifeless | clarkb: whats a good python source tree to read to crib from ? | 01:20 |
| clarkb | hmm, zuul is probably too complicated and the logstash workers are a bit simple... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/logstash/log-gearman-client.py | 01:21 |
| clarkb | lifeless: ^ does it the simple way, which is submit job and never wait for a result | 01:21 |
| clarkb | lifeless: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/gearman.py does it the more object asynchronousy way | 01:22 |
| lifeless | clarkb: are all things peers? | 01:22 |
| clarkb | lifeless: no, clients submit work and workers perform work and the all talk to the gearman server instead of directly talking to each other | 01:23 |
| lifeless | righto | 01:23 |
| lifeless | so I need a client example and a worker example | 01:23 |
| clarkb | lifeless: you can however make workers clients and so on | 01:23 |
| lifeless | clarkb: if the broker is to detect dropped connections from $-gate nodes | 01:23 |
| lifeless | clarkb: should it be a worker or client? | 01:24 |
| clarkb | lifeless: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/logstash/log-gearman-worker.py is my simple worker example. not sure if there is anything better | 01:24 |
| clarkb | lifeless: I think the client is notified if the worker performing its work goes away so client | 01:24 |
| clarkb | jeblair: ^ is that correct? | 01:24 |
| clarkb | lifeless: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/launcher/gearman.py#n93 | 01:25 |
| lifeless | righto I've pulled zuul and nodepool down | 01:25 |
| lifeless | I will study and fiddle | 01:26 |
| lifeless | geard -d is enough to get going right ? | 01:26 |
| *** ericw has joined #openstack-infra | 01:26 | |
| clarkb | lifeless: yes should be | 01:26 |
| clarkb | as -d prevents all of the daemon reqs from being necessary | 01:26 |
| lifeless | cool, thanks | 01:27 |
| *** zehicle has quit IRC | 01:27 | |
| lifeless | -> gate etc etc | 01:27 |
| lifeless | clarkb: daemon reqs? | 01:27 |
| clarkb | lifeless: filesystem things for the pid file and so on | 01:27 |
| lifeless | ah | 01:27 |
| *** sandywalsh has joined #openstack-infra | 01:28 | |
| *** zehicle has joined #openstack-infra | 01:40 | |
| *** CaptTofu has joined #openstack-infra | 01:43 | |
| *** davidhadas has quit IRC | 01:59 | |
| *** davidhadas has joined #openstack-infra | 02:00 | |
| *** hashar has joined #openstack-infra | 02:22 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Let devstack to decide about swift storage https://review.openstack.org/46663 | 02:25 |
| *** dcramer_ has joined #openstack-infra | 02:27 | |
| *** jerryz has quit IRC | 02:40 | |
| *** jergerber has quit IRC | 02:45 | |
| *** jergerber has joined #openstack-infra | 02:50 | |
| *** davidhadas has quit IRC | 02:59 | |
| *** davidhadas has joined #openstack-infra | 03:00 | |
| *** dkliban has joined #openstack-infra | 03:03 | |
| *** nati_ueno has joined #openstack-infra | 03:05 | |
| *** dkliban has quit IRC | 03:07 | |
| *** ericw has quit IRC | 03:11 | |
| *** melwitt has quit IRC | 03:14 | |
| *** Ryan_Lane has quit IRC | 03:15 | |
| *** nati_ueno has quit IRC | 03:16 | |
| *** CaptTofu has quit IRC | 03:21 | |
| *** CaptTofu has joined #openstack-infra | 03:22 | |
| *** annegentle has quit IRC | 03:33 | |
| *** annegentle has joined #openstack-infra | 03:35 | |
| *** vipul is now known as vipul-away | 03:38 | |
| *** nati_ueno has joined #openstack-infra | 03:39 | |
| *** vipul-away is now known as vipul | 03:41 | |
| *** nicedice_ has quit IRC | 03:41 | |
| *** nicedice has joined #openstack-infra | 03:42 | |
| *** mrodden1 has joined #openstack-infra | 03:43 | |
| *** mrodden has quit IRC | 03:44 | |
| *** davidhadas has quit IRC | 03:59 | |
| *** davidhadas has joined #openstack-infra | 04:00 | |
| *** nati_ueno has quit IRC | 04:03 | |
| *** nati_ueno has joined #openstack-infra | 04:04 | |
| *** jergerber has quit IRC | 04:07 | |
| *** nati_ueno has quit IRC | 04:08 | |
| *** vipul is now known as vipul-away | 04:13 | |
| *** vipul-away is now known as vipul | 04:15 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack-dev/cookiecutter: Made the link to testrepository actually work https://review.openstack.org/47669 | 04:22 |
| *** adalbas has quit IRC | 04:27 | |
| *** vipul is now known as vipul-away | 04:35 | |
| *** vipul-away is now known as vipul | 04:35 | |
| *** HenryG has quit IRC | 04:42 | |
| *** Ryan_Lane has joined #openstack-infra | 04:50 | |
| *** davidhadas has quit IRC | 05:00 | |
| *** davidhadas has joined #openstack-infra | 05:01 | |
| *** blamar has quit IRC | 05:36 | |
| *** carlp-away has quit IRC | 05:39 | |
| *** hashar has quit IRC | 05:40 | |
| *** Ryan_Lane has quit IRC | 06:00 | |
| *** davidhadas has quit IRC | 06:00 | |
| *** davidhadas has joined #openstack-infra | 06:01 | |
| *** Ryan_Lane has joined #openstack-infra | 06:02 | |
| *** thomasbiege has joined #openstack-infra | 06:05 | |
| *** michchap_ has joined #openstack-infra | 06:08 | |
| *** michchap has quit IRC | 06:11 | |
| *** michchap_ has quit IRC | 06:13 | |
| *** boris-42 has quit IRC | 06:16 | |
| *** michchap has joined #openstack-infra | 06:18 | |
| *** plomakin_ has joined #openstack-infra | 06:20 | |
| *** plomakin has quit IRC | 06:20 | |
| *** samalba has quit IRC | 06:20 | |
| *** samalba has joined #openstack-infra | 06:21 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:26 | |
| *** SlickNik has quit IRC | 06:35 | |
| *** SlickNik has joined #openstack-infra | 06:35 | |
| *** thomasbiege has quit IRC | 06:40 | |
| *** cthulhup has joined #openstack-infra | 06:54 | |
| *** cthulhup has quit IRC | 06:55 | |
| *** davidhadas_ has joined #openstack-infra | 07:01 | |
| *** davidhadas has quit IRC | 07:01 | |
| *** vipul has quit IRC | 07:07 | |
| *** vipul has joined #openstack-infra | 07:07 | |
| *** yolanda has joined #openstack-infra | 07:07 | |
| *** vipul is now known as vipul-away | 07:25 | |
| *** boris-42 has joined #openstack-infra | 07:26 | |
| *** vipul-away is now known as vipul | 07:56 | |
| *** davidhadas_ has quit IRC | 08:00 | |
| *** davidhadas has joined #openstack-infra | 08:01 | |
| *** vipul is now known as vipul-away | 08:11 | |
| *** vipul-away is now known as vipul | 08:21 | |
| *** SergeyLu_ has joined #openstack-infra | 08:39 | |
| *** SergeyLukjanov has quit IRC | 08:40 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:41 | |
| *** SergeyLu_ has quit IRC | 08:44 | |
| *** boris-42 has quit IRC | 08:48 | |
| *** boris-42 has joined #openstack-infra | 08:50 | |
| *** cody-somerville has joined #openstack-infra | 08:52 | |
| *** davidhadas has quit IRC | 09:01 | |
| *** davidhadas has joined #openstack-infra | 09:01 | |
| *** boris-42 has quit IRC | 09:09 | |
| *** ryanpetrello has joined #openstack-infra | 09:17 | |
| *** alexpilotti has joined #openstack-infra | 09:37 | |
| *** davidhadas has quit IRC | 10:01 | |
| *** davidhadas has joined #openstack-infra | 10:01 | |
| *** thomasbiege has joined #openstack-infra | 10:08 | |
| *** alexpilotti has quit IRC | 10:14 | |
| *** michchap has quit IRC | 10:18 | |
| *** alexpilotti has joined #openstack-infra | 10:18 | |
| *** thomasbiege has quit IRC | 10:19 | |
| *** markmcclain has quit IRC | 10:34 | |
| *** alexpilotti has quit IRC | 10:44 | |
| *** fifieldt has joined #openstack-infra | 10:47 | |
| *** SergeyLukjanov has quit IRC | 10:48 | |
| *** davidhadas has quit IRC | 11:00 | |
| *** davidhadas has joined #openstack-infra | 11:01 | |
| *** michchap has joined #openstack-infra | 11:13 | |
| *** michchap has quit IRC | 11:33 | |
| *** mdenny has joined #openstack-infra | 11:45 | |
| *** MIDENN_ has quit IRC | 11:45 | |
| *** fifieldt has quit IRC | 11:48 | |
| *** thomasbiege has joined #openstack-infra | 11:56 | |
| *** davidhadas has quit IRC | 12:00 | |
| *** davidhadas has joined #openstack-infra | 12:01 | |
| *** thomasbiege has quit IRC | 12:04 | |
| *** yolanda has quit IRC | 12:04 | |
| *** rcleere has quit IRC | 12:05 | |
| *** rcleere has joined #openstack-infra | 12:10 | |
| *** rcleere has joined #openstack-infra | 12:12 | |
| *** xchu has joined #openstack-infra | 12:14 | |
| *** dkliban has joined #openstack-infra | 12:45 | |
| *** zul has quit IRC | 12:54 | |
| *** zul has joined #openstack-infra | 12:55 | |
| *** davidhadas_ has joined #openstack-infra | 13:02 | |
| *** davidhadas has quit IRC | 13:02 | |
| *** yamahata has quit IRC | 13:06 | |
| *** yamahata has joined #openstack-infra | 13:07 | |
| *** xchu has quit IRC | 13:22 | |
| *** jergerber has joined #openstack-infra | 13:22 | |
| *** yamahata has quit IRC | 13:31 | |
| *** fbo_away is now known as fbo | 13:34 | |
| *** CaptTofu has quit IRC | 13:38 | |
| *** CaptTofu has joined #openstack-infra | 13:38 | |
| *** alexpilotti has joined #openstack-infra | 13:40 | |
| *** dkliban has quit IRC | 13:44 | |
| *** adalbas has joined #openstack-infra | 13:49 | |
| *** dizquierdo has joined #openstack-infra | 13:49 | |
| *** adalbas has quit IRC | 13:51 | |
| *** davidhadas_ has quit IRC | 14:01 | |
| *** davidhadas has joined #openstack-infra | 14:02 | |
| *** nicedice has quit IRC | 14:02 | |
| *** michchap has joined #openstack-infra | 14:19 | |
| *** michchap has quit IRC | 14:24 | |
| *** blamar has joined #openstack-infra | 14:36 | |
| jeblair | lifeless: also, simple versions of each kind of gear client are in the docs: http://gear.readthedocs.org/en/latest/ | 14:40 |
| jeblair | lifeless: what are you using gear for? | 14:40 |
| *** dizquierdo has quit IRC | 14:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: Clean up flake8 issues before switch https://review.openstack.org/47640 | 14:57 |
| *** davidhadas has quit IRC | 15:01 | |
| *** CaptTofu has quit IRC | 15:01 | |
| *** davidhadas has joined #openstack-infra | 15:02 | |
| *** CaptTofu has joined #openstack-infra | 15:02 | |
| clarkb | jeblair: lifeless needs a multi node test env broker for tripleo | 15:03 |
| clarkb | jeblair: something that when run by a jenkins slave returns an unused "cluster" of baremetal devices free for testing. I believe the intention is to use gearman for that | 15:04 |
| jeblair | clarkb: is this going to run in openstack's ci system? | 15:07 |
| clarkb | jeblair: yes, that is the goal. I suppose I need to describe the plan that was hatched on a whiteboard over the last week | 15:09 |
| clarkb | jeblair: it turns out lxc containers are not able to be stand ins for proper VMs and do tripleo testing for reasons such as iscsid cannot run in a container | 15:09 |
| jeblair | clarkb, lifeless: yes, it would be great if you could write that up and start an email thread about it. | 15:10 |
| clarkb | jeblair: so we are looking at some sort of actual baremetal testing with hardware provided by lifeless. There is an etherpad /me looks for it | 15:12 |
| clarkb | jeblair: tl;dr is we would initially run tests in experimental to prevent blocking anyone else, and nodepool would provision slaves in a cloud that can talk to the baremetal, this will be our entry point to the system | 15:12 |
| jeblair | clarkb: what's the plan to get that running on something that isn't just run by robert? | 15:14 |
| jeblair | clarkb: because we can't have an open infrastructure team responsible for a system critical for the operation of the openstack project that's run on private hardware... | 15:15 |
| clarkb | jeblair: https://etherpad.openstack.org/tripleo-test-cluster is the etherpad. | 15:15 |
| clarkb | jeblair: right, so a few things there. The tripleo team would be responsible for stuff below the nodepool provisioned node (eg tests break help us to debug and so on). | 15:17 |
| jeblair | "On site hands available if things go sideways - place to file tickets, what is SLA?" from https://etherpad.openstack.org/real-baremetal-testing-requirements is what i'm particularly concerned about since you said "hardware provided by lifeless" | 15:18 |
| jeblair | i don't want to be in a position where if lifeless gets hit by the lottery, we can't test tripleo anymore. | 15:20 |
| clarkb | jeblair: the reason for starting with the experimental queue is in part to learn where friction will happen so that we can sort those things out without impacting others. | 15:20 |
| clarkb | jeblair: right, so I am not sure all of those details have been sorted. I think mordred has some schemes in the works. There was talk of a quick kill switch for the jobs should something go really sideways | 15:21 |
| jeblair | i'm okay with working incrementally and experimentally, i'd just like to know if we're starting on a path that actually has an end. | 15:21 |
| *** senk has joined #openstack-infra | 15:30 | |
| *** CaptTofu has quit IRC | 15:44 | |
| *** ArxCruz has joined #openstack-infra | 15:45 | |
| *** CaptTofu has joined #openstack-infra | 15:45 | |
| mordred | morning all | 15:54 |
| clarkb | mordred: morning | 15:54 |
| mordred | jeblair: the two sides of the coin from my perspective are a) the experimental queue / kill switch, so we can develop the things needed to acomplish this without potentially actually getting the project stuck | 15:56 |
| mordred | and b) growing the resources to a place where robert's cloud is not the only cloud doing this (such as a rack from blue box) | 15:57 |
| mordred | or osuosl | 15:57 |
| mordred | or both | 15:57 |
| *** EmilienM has quit IRC | 15:57 | |
| jeblair | mordred: cool, that sounds like a plan. | 15:57 |
| mordred | (long term, I think it would be great if we had something at both osuosl and blue box, because that would be double great) | 15:58 |
| mordred | jeblair: I'm in my own apartment. I'm very confused | 15:58 |
| *** EmilienM has joined #openstack-infra | 15:58 | |
| jeblair | ++ double great | 15:58 |
| jeblair | mordred: you have an apartment? | 15:58 |
| jeblair | mordred: does it come with 500 starwood points? | 15:59 |
| mordred | jeblair: yes! | 15:59 |
| clarkb | ha | 15:59 |
| mordred | jeblair: btw - I have officially hit the 125k/diamond now. sad part is - now there are no further status levels to attempt to achieve | 15:59 |
| mordred | I might just stay home for the rest of the year | 15:59 |
| clarkb | the osl folks hang out in #osuosl perhaps we should hop over there at some point when people are actually working and not doing other things | 16:00 |
| jeblair | mordred: and i thought the point was to _use_ the benefits? | 16:00 |
| mordred | jeblair: thank you. that reminds me why I fly | 16:00 |
| *** davidhadas has quit IRC | 16:01 | |
| mordred | lifeless: when we were talking about base nodes for your tripleo testing - I _completely_ forgot that the script that nodepool runs to prepare the nodes is completely configurable/overridable | 16:01 |
| jeblair | clarkb: yes | 16:01 |
| mordred | lifeless: so you can write your own to prep the hosts in whatever way is appropriate (such as pre-caching ubuntu images in the way you want) without jumping through tons of hoops | 16:01 |
| mordred | lifeless: it turns out jeblair designs things to be, well, used | 16:02 |
| mordred | :) | 16:02 |
| *** davidhadas has joined #openstack-infra | 16:02 | |
| * mordred goes to add note to etherpad | 16:02 | |
| *** senk has quit IRC | 16:05 | |
| mordred | ok. updated etherpad | 16:08 |
| jeblair | clarkb, mordred: can you decode "T" and "C" and "CT" on that list for me? | 16:08 |
| mordred | jeblair: TripleO, Infra, Combo of TripleO+Infra | 16:08 |
| mordred | (C might have been "clark") | 16:08 |
| clarkb | no it is CI | 16:09 |
| clarkb | pleia2 can help as well we went over that stuff yesterday with zaro and anteaya too | 16:10 |
| * mordred just expanded names | 16:11 | |
| mordred | clarkb: awesome | 16:11 |
| *** ArxCruz has quit IRC | 16:11 | |
| *** dcramer_ has quit IRC | 16:16 | |
| *** cody-somerville has quit IRC | 16:19 | |
| *** cody-somerville has joined #openstack-infra | 16:20 | |
| *** cody-somerville has quit IRC | 16:25 | |
| *** cody-somerville has joined #openstack-infra | 16:38 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set nodepool boot-timeout to 120 seconds https://review.openstack.org/47688 | 16:40 |
| *** davidhadas has quit IRC | 16:40 | |
| *** seagulls is now known as beagles | 16:40 | |
| *** davidhadas has joined #openstack-infra | 16:41 | |
| openstackgerrit | A change was merged to openstack-infra/config: Set nodepool boot-timeout to 120 seconds https://review.openstack.org/47688 | 16:45 |
| *** SergeyLukjanov has joined #openstack-infra | 16:45 | |
| *** annegentle has quit IRC | 16:50 | |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Made the link to testrepository actually work https://review.openstack.org/47669 | 16:50 |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Add pypy to the list of default envs https://review.openstack.org/46416 | 16:50 |
| openstackgerrit | A change was merged to openstack-dev/cookiecutter: Add upcall on setUp method https://review.openstack.org/47657 | 16:50 |
| *** annegentle has joined #openstack-infra | 16:52 | |
| *** SergeyLukjanov has quit IRC | 16:55 | |
| *** davidhadas has quit IRC | 17:01 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:01 | |
| *** davidhadas has joined #openstack-infra | 17:02 | |
| *** SergeyLukjanov has quit IRC | 17:09 | |
| jeblair | summer in berkeley just ended -- it's raining | 17:10 |
| pleia2 | yeah, here too | 17:11 |
| pleia2 | guess I brought it home from Seattle, you're welcome | 17:11 |
| *** SergeyLukjanov has joined #openstack-infra | 17:13 | |
| *** dims has quit IRC | 17:14 | |
| mordred | oh. wow. there's a dvd drive in the thinkpad docking station | 17:15 |
| pleia2 | neat | 17:15 |
| *** dims has joined #openstack-infra | 17:16 | |
| *** davidhadas has quit IRC | 17:17 | |
| mordred | pleia2: so I take it you and anteaya eventually made it | 17:26 |
| *** dims has quit IRC | 17:26 | |
| pleia2 | mordred: heh, yeah, got in around 10:30 | 17:26 |
| pleia2 | meeting up around noonish today to hash out details for the codechix workshop tomorrow \o/ | 17:26 |
| pleia2 | good activity for a rainy sf day | 17:26 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Fix logic in doc impact https://review.openstack.org/46721 | 17:28 |
| *** dims has joined #openstack-infra | 17:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move large-ops test to check pipeline https://review.openstack.org/46840 | 17:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Add nodepool to cacti https://review.openstack.org/47614 | 17:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Use templates for Install Guides, Training Guide https://review.openstack.org/47613 | 17:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Add Python 3.3 as Experimental for Oslo-Incubator https://review.openstack.org/47170 | 17:34 |
| openstackgerrit | A change was merged to openstack-infra/config: Add an openstack-sos mailing list https://review.openstack.org/47454 | 17:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Adds support to specify a different jenkis_jobs_builder username https://review.openstack.org/47156 | 17:37 |
| openstackgerrit | A change was merged to openstack-infra/gerritlib: Add comments option to query https://review.openstack.org/47396 | 17:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Trigger post-mirror for diskimage-builder, tripleo-image-elements https://review.openstack.org/47461 | 17:39 |
| *** CaptTofu has quit IRC | 17:45 | |
| *** CaptTofu has joined #openstack-infra | 17:57 | |
| *** Steely_Dan is now known as dansmith | 18:08 | |
| *** zul has quit IRC | 18:09 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use templates for old manual jobs https://review.openstack.org/47691 | 18:19 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use Jenkins templates for old manual jobs https://review.openstack.org/47691 | 18:24 |
| *** zul has joined #openstack-infra | 18:25 | |
| *** SergeyLukjanov has quit IRC | 18:29 | |
| *** senk has joined #openstack-infra | 18:34 | |
| *** anteaya has joined #openstack-infra | 18:38 | |
| anteaya | just had breakfast at the grove | 18:38 |
| *** hashar has joined #openstack-infra | 18:39 | |
| *** yportnova has quit IRC | 18:40 | |
| *** CaptTofu has quit IRC | 18:42 | |
| *** GheRivero is now known as Ghe_Holiday | 18:46 | |
| *** Ghe_Holiday is now known as Ghe_Holidays | 18:51 | |
| anteaya | pleia2, shall I mosey over? | 18:55 |
| pleia2 | anteaya: yeah sure :) | 18:55 |
| pleia2 | and the grove is wonderful! | 18:55 |
| * anteaya packs her laptop and kleenex | 18:56 | |
| anteaya | it is! | 18:56 |
| anteaya | doing well on concierge suggestions this trip | 18:56 |
| *** anteaya has quit IRC | 18:56 | |
| openstackgerrit | Yulia Portnova proposed a change to openstack-infra/config: New project: manila, python-manilaclient https://review.openstack.org/46919 | 19:09 |
| openstackgerrit | A change was merged to openstack-infra/config: New project: manila, python-manilaclient https://review.openstack.org/46919 | 19:19 |
| *** annegentle has quit IRC | 19:21 | |
| *** annegentle has joined #openstack-infra | 19:23 | |
| *** cody-somerville has quit IRC | 19:24 | |
| *** yolanda has joined #openstack-infra | 19:29 | |
| *** dkehn has quit IRC | 19:38 | |
| *** dkehn has joined #openstack-infra | 19:39 | |
| *** anteaya has joined #openstack-infra | 19:42 | |
| anteaya | yay cats | 19:43 |
| anteaya | pleia2 has cats | 19:43 |
| mordred | anteaya likes cats | 19:43 |
| anteaya | I do | 19:43 |
| anteaya | especially when sick | 19:43 |
| anteaya | cats are great when I'm sick | 19:43 |
| *** fbo is now known as fbo_away | 19:51 | |
| *** boris-42 has joined #openstack-infra | 19:56 | |
| *** thomasm has joined #openstack-infra | 20:05 | |
| *** sarob has joined #openstack-infra | 20:06 | |
| *** davidhadas has joined #openstack-infra | 20:08 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use Jenkins templates for old manual jobs https://review.openstack.org/47691 | 20:13 |
| *** fallenpegasus has joined #openstack-infra | 20:18 | |
| *** sarob has quit IRC | 20:20 | |
| *** sarob has joined #openstack-infra | 20:21 | |
| *** senk has quit IRC | 20:25 | |
| *** sarob has quit IRC | 20:25 | |
| *** annegentle has quit IRC | 20:35 | |
| *** annegentle has joined #openstack-infra | 20:36 | |
| * hashar eats cats | 20:38 | |
| openstackgerrit | Alex Gaynor proposed a change to openstack/requirements: Added pypy env to tox https://review.openstack.org/47697 | 20:41 |
| Alex_Gaynor | dstufft: timeouts to pypi observed: https://jenkins01.openstack.org/job/gate-requirements-install/249/console :/ | 20:42 |
| Alex_Gaynor | dstufft: nothing obvious in the timeline on s.p.o | 20:42 |
| Alex_Gaynor | dstufft: seeing someting weird, that file downloads fine in my browser, but curl/wget are hanging | 20:43 |
| Alex_Gaynor | err, they stopped hanging when i removed the #hashtag from the URL | 20:43 |
| dstufft | interesting | 20:44 |
| dstufft | thrift again | 20:44 |
| *** senk has joined #openstack-infra | 20:45 | |
| Alex_Gaynor | dstufft: FWIW 3 of them all just failed on thrift | 20:47 |
| dstufft | yea something strange about thrift | 20:47 |
| Alex_Gaynor | Computers :/ | 20:47 |
| *** yolanda has quit IRC | 20:47 | |
| dstufft | https://gist.github.com/dstufft/6654034 | 20:48 |
| Alex_Gaynor | dstufft: what does that even mean? | 20:50 |
| dstufft | 503 service unavailable is strange | 20:51 |
| Alex_Gaynor | dstufft: I'm getting 200s, but sitll the "* <url> malformed" thing | 20:52 |
| dstufft | Alex_Gaynor: immediately you're getting 200's or after a minute or so? | 20:52 |
| Alex_Gaynor | dstufft: just issued another request, this one is taking a while (20+ seconds) | 20:53 |
| dstufft | it'll probably give you a service unavaialble at around th e1 minute mark | 20:53 |
| Alex_Gaynor | ok, 503 there | 20:54 |
| Alex_Gaynor | ok, wtf is going on? | 20:54 |
| dstufft | Alex_Gaynor: can you reproduce using http://pypi.int.python.org instead of https://pypi.python.org | 20:54 |
| Alex_Gaynor | dstufft: instant 200s with "* <url> malformed" | 20:55 |
| dstufft | Alex_Gaynor: sent a request to Fastly | 20:56 |
| dstufft | we'll see what they say | 20:56 |
| Alex_Gaynor | dstufft: what's with the URL malformed issue on our end? | 20:56 |
| dstufft | not a clue what that means | 20:57 |
| Alex_Gaynor | dstufft: cool, is there somewhere I can follow the request? | 20:57 |
| dstufft | Alex_Gaynor: do you have a PSF fastly account? | 20:57 |
| Alex_Gaynor | dstufft: yeah I do | 20:57 |
| *** sarob has joined #openstack-infra | 20:57 | |
| Alex_Gaynor | I'm watching the thing now (why is our hit rate so low?) | 20:58 |
| dstufft | Alex_Gaynor: dunno if it'll work but you can try logging into https://fastly.zendesk.com and seeing if that gives you anything | 20:58 |
| *** senk has quit IRC | 20:58 | |
| dstufft | it shows me and noah there but I don't know if they maintain seperate DB's for that or not | 20:58 |
| *** thomasm has quit IRC | 21:05 | |
| *** davidhadas has quit IRC | 21:07 | |
| *** hashar has quit IRC | 21:08 | |
| *** davidhadas has joined #openstack-infra | 21:08 | |
| *** ryanpetrello has quit IRC | 21:10 | |
| *** ryanpetrello has joined #openstack-infra | 21:20 | |
| *** senk has joined #openstack-infra | 21:28 | |
| *** blamar has quit IRC | 21:30 | |
| *** anteaya has quit IRC | 21:32 | |
| *** fallenpegasus has quit IRC | 21:32 | |
| *** ryanpetrello has quit IRC | 21:35 | |
| *** senk has quit IRC | 21:36 | |
| *** fallenpegasus has joined #openstack-infra | 21:38 | |
| *** ryanpetrello has joined #openstack-infra | 21:42 | |
| * mordred needs to learn how elastic-recheck works | 21:44 | |
| *** ryanpetrello has quit IRC | 21:45 | |
| Alex_Gaynor | mordred: I think right now it's just a static list of patterns to check a log against whenever a failure occurs | 21:45 |
| * mordred wonderis if it could be used to recheck things that are associated with pypi network issues | 21:46 | |
| mordred | Alex_Gaynor: (btw, I just rechecked your patch from above) | 21:46 |
| Alex_Gaynor | thanks | 21:46 |
| *** alexpilotti_ has joined #openstack-infra | 21:47 | |
| *** changbl has quit IRC | 21:48 | |
| *** alexpilotti has quit IRC | 21:48 | |
| *** alexpilotti_ is now known as alexpilotti | 21:48 | |
| *** davidhadas has quit IRC | 21:54 | |
| *** changbl has joined #openstack-infra | 21:59 | |
| *** fallenpegasus has quit IRC | 22:13 | |
| *** dcramer_ has joined #openstack-infra | 22:21 | |
| clarkb | mordred it could be used for that | 22:23 |
| *** DennyZhang has joined #openstack-infra | 22:27 | |
| *** DennyZhang has quit IRC | 22:34 | |
| *** DennyZhang has joined #openstack-infra | 22:34 | |
| *** sarob has quit IRC | 22:38 | |
| *** sarob has joined #openstack-infra | 22:39 | |
| *** sarob has quit IRC | 22:43 | |
| *** blamar has joined #openstack-infra | 22:45 | |
| *** hashar has joined #openstack-infra | 23:25 | |
| *** CaptTofu has joined #openstack-infra | 23:25 | |
| *** boris-42 has quit IRC | 23:27 | |
| *** DennyZhang has quit IRC | 23:30 | |
| openstackgerrit | A change was merged to openstack/requirements: Require python-novaclient>=2.15.0 https://review.openstack.org/47185 | 23:33 |
| *** senk has joined #openstack-infra | 23:34 | |
| *** senk has quit IRC | 23:38 | |
| *** SlickNik_ has joined #openstack-infra | 23:39 | |
| *** sarob has joined #openstack-infra | 23:39 | |
| *** sarob has quit IRC | 23:44 | |
| Alex_Gaynor | What exactly do the post-mirror jobs do? | 23:47 |
| mordred | Alex_Gaynor: they trigger runs of our mirror script to rebuild our mirror | 23:49 |
| Alex_Gaynor | mordred: hmm, I see there's one per version, should I have added apypy one at some point? | 23:49 |
| mordred | hrm. interesting question. we have one per version because the requirements file does transitive depends different based on version (some people say things like "if pyversion < 2.7: install_requires = blah") | 23:51 |
| mordred | which, I believe, btw, is a terrible way to do things, but whatev | 23:51 |
| Alex_Gaynor | I guess if they're all version based, then no, I don't | 23:51 |
| mordred | so, if you think that pypy might affect the set of things that get downloaded | 23:51 |
| Alex_Gaynor | Don't think so | 23:51 |
| *** dims has quit IRC | 23:52 | |
| mordred | good. you represent sanity then | 23:53 |
| *** dims has joined #openstack-infra | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!