| *** thomasm has joined #openstack-infra | 00:04 | |
| clarkb | jog0: logs coming up | 00:06 |
|---|---|---|
| jog0 | clarkb: so the bug isn't in classify | 00:09 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Increase log worker retry timeout length. https://review.openstack.org/49289 | 00:09 |
| *** hogepodge has quit IRC | 00:10 | |
| clarkb | jog0: do you understand what is going on? | 00:10 |
| jog0 | clarkb: nope | 00:11 |
| *** w_ has joined #openstack-infra | 00:12 | |
| *** olaph has quit IRC | 00:15 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 00:15 |
| jog0 | this ^ may help shine some light? | 00:16 |
| jog0 | gerritlib is logging it received the right information but Stream isn't working or something | 00:16 |
| clarkb | jog0: I will take a look once I have convinced myself the new backoff inthe gearman works is not broken | 00:18 |
| jog0 | clarkb: cool | 00:19 |
| jog0 | so the strange thing about e-r failing is when I run locally in non-bot mode it all works | 00:19 |
| *** nati_ueno has quit IRC | 00:21 | |
| *** nati_ueno has joined #openstack-infra | 00:21 | |
| *** matsuhashi has joined #openstack-infra | 00:32 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Increase log worker retry timeout length. https://review.openstack.org/49289 | 00:34 |
| clarkb | I did break it >_> latest patchset should be good. I am applying it | 00:34 |
| *** dkliban has joined #openstack-infra | 00:44 | |
| *** thomasm has quit IRC | 00:49 | |
| clarkb | ok logstash workers all kicked. Going to review the bot change and merge then kick the bot | 00:49 |
| clarkb | jog0: still around? you added a log.setlevel but aren't we configuring that in a file? | 00:50 |
| openstackgerrit | A change was merged to openstack-infra/config: Make gate-hacking-pypy non-gating https://review.openstack.org/48352 | 00:55 |
| openstackgerrit | A change was merged to openstack-infra/zuul: doc: point directly to zuul.conf section https://review.openstack.org/48416 | 00:59 |
| openstackgerrit | A change was merged to openstack-infra/config: Retrieve logs > DEBUG level. https://review.openstack.org/48508 | 01:00 |
| clarkb | jog0: I -1'd with a comment. Let me know if I am crazy | 01:00 |
| clarkb | going to restart elastic-recheck now to see if it is any better with the new log workers | 01:01 |
| *** nosnos has joined #openstack-infra | 01:04 | |
| *** oubiwann has joined #openstack-infra | 01:09 | |
| *** jaypipes has quit IRC | 01:11 | |
| *** dcramer_ has quit IRC | 01:24 | |
| *** melwitt has quit IRC | 01:25 | |
| *** ryanpetrello has quit IRC | 01:29 | |
| *** schwicht has quit IRC | 01:36 | |
| *** dhellman_ has joined #openstack-infra | 01:37 | |
| *** dhellman_ has quit IRC | 01:40 | |
| *** ryanpetrello has joined #openstack-infra | 01:41 | |
| *** esker has joined #openstack-infra | 01:42 | |
| *** ryanpetrello has quit IRC | 01:48 | |
| *** nati_ueno has quit IRC | 01:51 | |
| *** oubiwann has quit IRC | 01:56 | |
| clarkb | jog0: I wonder if it is related to the elastic-recheck html/js page | 01:57 |
| clarkb | jog0: because that page appears to be updating maybe | 01:57 |
| *** ryanpetrello has joined #openstack-infra | 01:57 | |
| dstufft | mordred: ping | 02:12 |
| *** dcramer_ has joined #openstack-infra | 02:18 | |
| clarkb | jog0: https://review.openstack.org/#/c/49466/ doubling the test processes for tempest seems to help | 02:20 |
| clarkb | jog0: that change takes the tests to just under 30 minutes on hpcloud VMs | 02:20 |
| clarkb | jog0: and tempest didn't fail in new and exciting ways | 02:20 |
| jeblair | clarkb: i don't think the /srv/static/status/elastic-recheck/graph.json file is being updated, despite the cron job being run. i'm not sure what the problem is. | 02:23 |
| *** oubiwann has joined #openstack-infra | 02:24 | |
| *** oubiwann has quit IRC | 02:24 | |
| clarkb | jeblair: jog0 has a change up to add more debug output, that should help us sort this, but it needs a small change | 02:24 |
| *** oubiwann has joined #openstack-infra | 02:25 | |
| *** oubiwann has quit IRC | 02:25 | |
| jeblair | clarkb: different thing i think. | 02:25 |
| clarkb | I see you ahve reviewed it | 02:25 |
| clarkb | jeblair: ah | 02:25 |
| jeblair | clarkb: https://review.openstack.org/#/c/49417/ | 02:25 |
| jeblair | clarkb: which uses https://review.openstack.org/#/c/49410/ | 02:25 |
| clarkb | I see. I will have to look at all of this stuff much more closely tomorrow. Very excited about the elastic-recheck status | 02:27 |
| clarkb | jeblair: would it maybe be more beneficial to have elastic-recheck output to graphite? | 02:27 |
| clarkb | rather than using a cron? | 02:27 |
| jeblair | clarkb: since we're interested in data about bugs, which are transient, i don't think it's a great match (we'll end up with lots of whisper files for bugs we don't care about) | 02:28 |
| jeblair | i mean, ideally they are transient. :) | 02:28 |
| clarkb | jeblair: gotcha | 02:28 |
| jeblair | eg, if we _do_ care about how much we've seen a bug over the past 3 years, we're doomed. doomed. | 02:29 |
| *** erfanian has joined #openstack-infra | 02:30 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add /usr/local/bin to recheck graph cron path https://review.openstack.org/49468 | 02:34 |
| jeblair | clarkb: that's it ^ | 02:34 |
| jeblair | grumble puppet cron path grumble | 02:35 |
| clarkb | one thing about that query timeframe is we will need to be careful about not adding really expensive queries to the list (so far so good though) | 02:38 |
| clarkb | it really helps that we restrict queries to specific files | 02:38 |
| clarkb | also the gate tests seem to be a lot better today | 02:38 |
| jeblair | clarkb: yeah i just timed it at 6 seconds for all queries (i've seen it take 20 or 30 seconds though) | 02:40 |
| jeblair | clarkb: but that's also part of the reason it's on a 15 min cron (i mean, we _could_ actually do it all in realtime, but that would be silly) | 02:41 |
| clarkb | ++ | 02:41 |
| clarkb | I am super excited about all of the stuff happening around logstash/elasticsearch. I was secretly hoping that it would pick up traction with other people reducing the amount of work I need to do :) I do need to upgrade the whole system sometime soon though. that will be fun | 02:43 |
| jesusaurus | clarkb: yeah, you need to get es up to a 0.90.x version | 02:44 |
| clarkb | jesusaurus: yup, it is on the list right after spinning up a new etherpad | 02:44 |
| jesusaurus | clarkb: are you going to worry about reindexing old data into the new system? if so i think i have a python wrapper around the reindex plugin somewhere | 02:46 |
| clarkb | need to test a new etherpad with cloud db before the summit | 02:46 |
| clarkb | jesusaurus: I am going to do an in place upgrade which converts the indexes I think and if that fails I will discard the old stuff | 02:46 |
| clarkb | jesusaurus: I can go in a forward direction with the data but not backward | 02:47 |
| clarkb | jesusaurus: logstash schema changes though, so I may not worry about it | 02:47 |
| clarkb | they have flattened fields into the global keyspace | 02:48 |
| jesusaurus | oh, right | 02:48 |
| *** oubiwann has joined #openstack-infra | 02:48 | |
| *** oubiwann has quit IRC | 02:48 | |
| clarkb | (which I think is a good thing otherwise I would try to keep the old schema) | 02:48 |
| jesusaurus | i forgot about the logstash upgrade part | 02:48 |
| clarkb | jesusaurus: though now that we are dealing with much less volume (thank you sdague) I may switch to the http output rather than the normal output for es | 02:49 |
| clarkb | not sure yet | 02:49 |
| jesusaurus | the http output has been working well for me | 02:50 |
| *** ArxCruz has quit IRC | 02:52 | |
| *** oubiwann has joined #openstack-infra | 02:54 | |
| *** jerryz has quit IRC | 03:04 | |
| *** dguitarbite has joined #openstack-infra | 03:08 | |
| *** markmcclain has quit IRC | 03:16 | |
| *** ryanpetrello has quit IRC | 03:20 | |
| *** ryanpetrello has joined #openstack-infra | 03:32 | |
| *** UtahDave has joined #openstack-infra | 03:36 | |
| *** vipul has quit IRC | 03:37 | |
| *** vipul has joined #openstack-infra | 03:37 | |
| *** CaptTofu has quit IRC | 03:51 | |
| *** CaptTofu has joined #openstack-infra | 03:51 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 03:53 |
| *** UtahDave has quit IRC | 03:55 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 03:59 |
| *** dguitarbite has quit IRC | 04:04 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 04:05 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 04:17 |
| *** sarob has joined #openstack-infra | 04:17 | |
| *** mtreinish has quit IRC | 04:28 | |
| *** anteaya has quit IRC | 04:30 | |
| zaro | anybody know why pip install git-review doesn't work? http://paste.openstack.org/show/47890/ | 04:34 |
| zaro | i get error: invalid command 'egg_info' | 04:36 |
| *** AlexF has joined #openstack-infra | 04:40 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Create new gates for openstack-manuals https://review.openstack.org/48128 | 04:40 |
| *** mtreinish has joined #openstack-infra | 04:42 | |
| *** asalkeld has joined #openstack-infra | 04:54 | |
| asalkeld | hi. Heat has users that are keen to vote/ask for features. besides writing up blueprints what can they do? | 04:55 |
| *** SergeyLukjanov has joined #openstack-infra | 04:56 | |
| asalkeld | actually a vote on blueprints would work well | 04:56 |
| clarkb | asalkeld: two possible methods come to mind. users can vote on changes in gerrit (maybe not code implementing features but those that propose them) or you can use launchpad team polls | 05:03 |
| asalkeld | ok | 05:04 |
| asalkeld | I'll look at team polls | 05:04 |
| *** sarob has quit IRC | 05:12 | |
| *** sarob has joined #openstack-infra | 05:12 | |
| clarkb | survey monkey or similar may be useful too. I didn't take the openstack user survey but maybe what they used? | 05:14 |
| asalkeld | yeah, that's right | 05:15 |
| *** sarob has quit IRC | 05:16 | |
| *** AlexF has quit IRC | 05:22 | |
| *** AlexF has joined #openstack-infra | 05:34 | |
| *** davidhadas has quit IRC | 05:35 | |
| *** shardy is now known as shardy_afk | 05:37 | |
| *** AlexF has quit IRC | 05:38 | |
| *** jaypipes has joined #openstack-infra | 05:39 | |
| *** pblaho has joined #openstack-infra | 06:00 | |
| *** sergmelikyan has quit IRC | 06:02 | |
| *** yolanda has joined #openstack-infra | 06:08 | |
| *** tvb has joined #openstack-infra | 06:14 | |
| *** tvb has quit IRC | 06:14 | |
| *** tvb has joined #openstack-infra | 06:14 | |
| *** SergeyLukjanov has quit IRC | 06:17 | |
| *** rwsu is now known as rwsu-away | 06:21 | |
| *** davidhadas has joined #openstack-infra | 06:21 | |
| *** sarob has joined #openstack-infra | 06:23 | |
| ttx | jeblair: nice (elastic-recheck) | 06:26 |
| ttx | jeblair: I would still leave the bug title visible though | 06:26 |
| *** mkoderer_ is now known as mkoderer | 06:26 | |
| *** sarob has quit IRC | 06:28 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:41 | |
| *** SergeyLukjanov has quit IRC | 06:42 | |
| *** odyssey4me has joined #openstack-infra | 06:48 | |
| ttx | jeblair: (if it's meant as a replacement for the original page) | 06:50 |
| *** ryanpetrello has quit IRC | 06:54 | |
| *** osanchez has joined #openstack-infra | 07:01 | |
| *** jcoufal has joined #openstack-infra | 07:08 | |
| *** afazekas has joined #openstack-infra | 07:11 | |
| *** Ryan_Lane has joined #openstack-infra | 07:14 | |
| *** shardy_afk has quit IRC | 07:16 | |
| *** flaper87|afk has quit IRC | 07:17 | |
| *** flaper87|afk has joined #openstack-infra | 07:18 | |
| *** AlexF has joined #openstack-infra | 07:18 | |
| *** flaper87|afk is now known as flaper87 | 07:18 | |
| *** shardy_afk has joined #openstack-infra | 07:22 | |
| *** Ryan_Lane has quit IRC | 07:26 | |
| *** flaper87 has quit IRC | 07:30 | |
| *** flaper87 has joined #openstack-infra | 07:30 | |
| *** che-arne has quit IRC | 07:34 | |
| *** boris-42 has joined #openstack-infra | 07:38 | |
| *** dkliban has quit IRC | 07:41 | |
| *** osanchez has quit IRC | 07:43 | |
| *** mkerrin has quit IRC | 07:48 | |
| *** DinaBelova has joined #openstack-infra | 07:54 | |
| *** fbo_away is now known as fbo | 07:55 | |
| *** dizquierdo has joined #openstack-infra | 07:55 | |
| *** AlexF has quit IRC | 07:59 | |
| *** johnthetubaguy has joined #openstack-infra | 08:00 | |
| *** mrmartin has joined #openstack-infra | 08:01 | |
| *** davidhadas_ has joined #openstack-infra | 08:02 | |
| *** odyssey4me has quit IRC | 08:03 | |
| *** davidhadas has quit IRC | 08:04 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:04 | |
| *** tkammer has joined #openstack-infra | 08:05 | |
| *** davidhadas_ has quit IRC | 08:05 | |
| *** davidhadas has joined #openstack-infra | 08:07 | |
| *** DinaBelova has quit IRC | 08:09 | |
| *** locke105 has joined #openstack-infra | 08:10 | |
| *** locke1051 has quit IRC | 08:10 | |
| *** odyssey4me has joined #openstack-infra | 08:11 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Bump hacking version in requirements https://review.openstack.org/49486 | 08:13 |
| *** mkerrin has joined #openstack-infra | 08:15 | |
| *** yassine has joined #openstack-infra | 08:23 | |
| *** thomasbiege1 has joined #openstack-infra | 08:27 | |
| *** thomasbiege1 has quit IRC | 08:36 | |
| *** jpich has joined #openstack-infra | 08:38 | |
| *** shardy_afk is now known as shardy | 08:41 | |
| *** che-arne has joined #openstack-infra | 08:43 | |
| *** DinaBelova has joined #openstack-infra | 08:54 | |
| *** AlexF has joined #openstack-infra | 08:54 | |
| *** tkammer has quit IRC | 08:56 | |
| *** saper_ is now known as saper | 08:59 | |
| *** matsuhashi has quit IRC | 09:01 | |
| *** matsuhashi has joined #openstack-infra | 09:09 | |
| *** thomasbiege1 has joined #openstack-infra | 09:16 | |
| *** thomasbiege2 has joined #openstack-infra | 09:23 | |
| *** johnthetubaguy1 has joined #openstack-infra | 09:24 | |
| *** johnthetubaguy has quit IRC | 09:24 | |
| *** thomasbiege3 has joined #openstack-infra | 09:25 | |
| *** johnthetubaguy has joined #openstack-infra | 09:26 | |
| *** thomasbiege1 has quit IRC | 09:26 | |
| *** thomasbiege2 has quit IRC | 09:27 | |
| *** johnthetubaguy1 has quit IRC | 09:28 | |
| *** DinaBelova has quit IRC | 09:28 | |
| *** thomasbiege3 has quit IRC | 09:35 | |
| *** DinaBelova has joined #openstack-infra | 09:36 | |
| *** AlexF has quit IRC | 09:36 | |
| *** tvb has quit IRC | 09:36 | |
| *** davidhadas has quit IRC | 09:38 | |
| *** davidhadas_ has joined #openstack-infra | 09:38 | |
| *** tvb has joined #openstack-infra | 09:40 | |
| *** derekh has joined #openstack-infra | 09:44 | |
| *** DinaBelova has quit IRC | 09:45 | |
| *** AlexF has joined #openstack-infra | 10:00 | |
| *** tvb has quit IRC | 10:02 | |
| openstackgerrit | Julien Danjou proposed a change to openstack/requirements: Update sphinxcontrib-pecanwsme https://review.openstack.org/49502 | 10:05 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack is launched with proper tenant isolation setting https://review.openstack.org/49504 | 10:05 |
| *** DinaBelova has joined #openstack-infra | 10:07 | |
| *** AlexF has quit IRC | 10:12 | |
| *** boris-42 has quit IRC | 10:18 | |
| *** AlexF has joined #openstack-infra | 10:21 | |
| *** schwicht has joined #openstack-infra | 10:29 | |
| *** tvb has joined #openstack-infra | 10:33 | |
| *** tvb has joined #openstack-infra | 10:33 | |
| *** AlexF has quit IRC | 10:39 | |
| *** AlexF has joined #openstack-infra | 10:40 | |
| *** flaper87 is now known as flaper87|afk | 10:40 | |
| *** tvb has quit IRC | 10:41 | |
| *** tvb has joined #openstack-infra | 10:45 | |
| *** mrmartin has quit IRC | 10:55 | |
| *** nosnos has quit IRC | 10:56 | |
| *** matsuhashi has quit IRC | 10:57 | |
| *** weshay has joined #openstack-infra | 10:59 | |
| *** esker has quit IRC | 11:00 | |
| *** dprince has joined #openstack-infra | 11:00 | |
| *** esker has joined #openstack-infra | 11:00 | |
| *** esker has quit IRC | 11:05 | |
| *** DinaBelova has quit IRC | 11:06 | |
| *** AlexF has quit IRC | 11:07 | |
| *** markmc has joined #openstack-infra | 11:09 | |
| *** esker has joined #openstack-infra | 11:10 | |
| *** DinaBelova has joined #openstack-infra | 11:13 | |
| *** esker has quit IRC | 11:15 | |
| *** AlexF has joined #openstack-infra | 11:15 | |
| *** boris-42 has joined #openstack-infra | 11:15 | |
| *** tvb has quit IRC | 11:19 | |
| *** pblaho has quit IRC | 11:22 | |
| *** dprince has quit IRC | 11:24 | |
| *** adalbas has joined #openstack-infra | 11:25 | |
| *** CaptTofu has quit IRC | 11:27 | |
| *** CaptTofu has joined #openstack-infra | 11:27 | |
| *** che-arne has quit IRC | 11:30 | |
| *** esker has joined #openstack-infra | 11:31 | |
| *** giulivo has joined #openstack-infra | 11:32 | |
| giulivo | guys, again I'm sorry but I'm not sure why this stable/grizzly is failing on the neutron vm... apparently even before getting to tempes | 11:32 |
| giulivo | https://review.openstack.org/#/c/49491/ | 11:33 |
| giulivo | any idea what is going wrong there? | 11:33 |
| *** alexpilotti has joined #openstack-infra | 11:33 | |
| giulivo | I see it "leaked file descriptor" again but seems related to jenkins not the actual job | 11:34 |
| *** esker has quit IRC | 11:35 | |
| *** AlexF has quit IRC | 11:37 | |
| *** pblaho has joined #openstack-infra | 11:40 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added base_dir parameter to get_projects_info https://review.openstack.org/49518 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added the server parameter to get_changes https://review.openstack.org/49519 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Pep8 compliant https://review.openstack.org/49520 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some optinos to openreviews https://review.openstack.org/49521 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options to openapproved https://review.openstack.org/49522 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options to reviewers https://review.openstack.org/49523 | 11:40 |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options to reviews_for_bugs https://review.openstack.org/49524 | 11:40 |
| *** pcm_ has joined #openstack-infra | 11:41 | |
| *** jpich has quit IRC | 11:44 | |
| *** jpich has joined #openstack-infra | 11:44 | |
| *** esker has joined #openstack-infra | 11:49 | |
| *** esker has quit IRC | 11:54 | |
| *** adalbas has quit IRC | 11:58 | |
| *** thomasm has joined #openstack-infra | 11:59 | |
| *** CaptTofu has quit IRC | 12:03 | |
| *** CaptTofu has joined #openstack-infra | 12:03 | |
| *** flaper87|afk is now known as flaper87 | 12:04 | |
| *** CaptTofu has quit IRC | 12:08 | |
| *** dcramer_ has quit IRC | 12:08 | |
| *** asalkeld has left #openstack-infra | 12:09 | |
| *** hashar has joined #openstack-infra | 12:10 | |
| *** jcoufal_ has joined #openstack-infra | 12:13 | |
| *** jcoufal has quit IRC | 12:16 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/requirements: Added upper version boundary for netaddr https://review.openstack.org/49530 | 12:21 |
| *** hashar has quit IRC | 12:22 | |
| *** che-arne has joined #openstack-infra | 12:23 | |
| *** pblaho has quit IRC | 12:24 | |
| *** ArxCruz has joined #openstack-infra | 12:25 | |
| *** hashar has joined #openstack-infra | 12:26 | |
| openstackgerrit | Lucas Alvares Gomes proposed a change to openstack/requirements: Added lower version boundary for netaddr https://review.openstack.org/49530 | 12:26 |
| *** pblaho has joined #openstack-infra | 12:33 | |
| *** hashar has quit IRC | 12:35 | |
| EmilienM | Hi here, i'm currently investigating how could i implement https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin in JJB. The relevant fact here is that add a top level XML value. | 12:36 |
| *** tvb has joined #openstack-infra | 12:39 | |
| *** tvb has joined #openstack-infra | 12:39 | |
| fungi | EmilienM: by "top level xml value" do you mean in the global config.xml but outside the plugins container? | 12:39 |
| EmilienM | fungi: hey, yeah. Actually <com.cloudbees.plugins.flow.BuildFlow plugin="build-flow-plugin@0.10"> replaces <poject> | 12:40 |
| *** hashar has joined #openstack-infra | 12:41 | |
| EmilienM | project* | 12:41 |
| hashar | JJB is very active | 12:41 |
| hashar | I am wondering how it suddenly got attention | 12:41 |
| EmilienM | hashar: i'm wondering how could i bring buildflow support in jjb | 12:42 |
| hashar | never used that plugin | 12:43 |
| EmilienM | XML regarding, it replaces <project> by <com.cloudbees.plugins.flow.BuildFlow plugin="build-flow-plugin@0.10"> | 12:43 |
| hashar | I am using Zuul definition for the workflow :/ | 12:43 |
| *** CaptTofu has joined #openstack-infra | 12:44 | |
| EmilienM | and "dsl" in "properties" | 12:44 |
| EmilienM | hashar: interesting. I guess this is already supported by JJB, right ? | 12:44 |
| hashar | not really | 12:45 |
| hashar | I got a flow example with a job fetching a debian package sources to build a tarball | 12:45 |
| hashar | the tar ball is then saved as an artifact | 12:45 |
| hashar | when complete, Zuul would trigger another job that fetch the artifact from the source job and build the package | 12:45 |
| hashar | but you never know whether you are fetching the proper sources since another job can have generated another tar ball :/ | 12:46 |
| hashar | anyway, my use case was merely to fail the change as soon as possible, for example when a PHP file is invalid | 12:46 |
| hashar | or skip the integration tests when the fast tests are failling | 12:46 |
| hashar | can show you a rough demo tonight if you want | 12:47 |
| EmilienM | hashar: sounds like a plan we have :) | 12:47 |
| EmilienM | hashar: while this, i would like to start the work. Do you know if JJB is able now to detect if a plugin is installed or not ? | 12:48 |
| hashar | I know the XML configuration generated by Jenkins list the plugin version | 12:49 |
| hashar | but haven't found an API entry point to list the plugin versions | 12:49 |
| hashar | might need something upstream | 12:49 |
| hashar | would be very nice to have JJB fail the job generation when a plugin is missing | 12:49 |
| EmilienM | hashar: ok, i'm working on that now. Let's talk irl about that later :) | 12:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Add /usr/local/bin to recheck graph cron path https://review.openstack.org/49468 | 12:52 |
| *** flaper87 is now known as flaper87|afk | 12:52 | |
| *** dkranz has quit IRC | 12:54 | |
| *** rfolco has joined #openstack-infra | 12:54 | |
| *** psedlak|afk is now known as psedlak | 12:56 | |
| *** pblaho has quit IRC | 12:57 | |
| hashar | EmilienM: /ci/pluginManager/api/json?pretty=true&depth=1 | 12:57 |
| hashar | EmilienM: list all the plugins with short / long names, their version , whether there is an update available and dependencies | 12:58 |
| hashar | I guess that should be added to python-jenkins bzr+ssh://bazaar.launchpad.net/+branch/python-jenkins/ | 12:58 |
| fungi | giulivo: the "process leaked file descriptors" message is benign. seems like i see that at the end of every devstack test even when successful. looks like devstack setup died somewhere in the middle of executing quantum commands, so possibly something is still not 100% supported in the quantum->neutron compat wrapper? looking at some of the more detailed logs now | 12:59 |
| EmilienM | hashar: nice shot, thanks ! | 13:00 |
| giulivo | fungi, thanks for that, apparently there is a bug about https://bugs.launchpad.net/openstack-ci/+bug/1234181 | 13:00 |
| uvirtbot | Launchpad bug 1234181 in openstack-ci "stable/grizzly patches are failing jenkins in check-tempest-devstack-vm-neutron" [Undecided,New] | 13:00 |
| hashar | EmilienM: JJB already extend the Jenkins object, so you can probably hack something inside JJB | 13:00 |
| *** dprince has joined #openstack-infra | 13:00 | |
| fungi | giulivo: ahh, yes i thought maybe you were working on that bug. i should have pointed you to it | 13:00 |
| hashar | EmilienM: and we can submit the code back to upstream. James / Monty used a similar approach to manages slaves in Jenkins. They have their own myjenkins class in node pool. | 13:00 |
| fungi | still not sure why the bug is in openstack-ci | 13:01 |
| EmilienM | hashar: everything should be upstream :) | 13:01 |
| hashar | ideally | 13:01 |
| *** matty_dubs|gone is now known as matty_dubs | 13:01 | |
| hashar | argg | 13:02 |
| hashar | poor Old Gerrit | 13:02 |
| fungi | hashar: did something break? | 13:02 |
| *** oubiwann_ has joined #openstack-infra | 13:02 | |
| *** osanchez has joined #openstack-infra | 13:03 | |
| hashar | fungi: na it works. Just missing a few features :-] | 13:03 |
| fungi | okay, good | 13:04 |
| * fungi gets back to getting things done for a change | 13:04 | |
| hashar | sorry | 13:04 |
| * hashar heads to some conf room | 13:04 | |
| *** hashar has quit IRC | 13:04 | |
| *** pblaho has joined #openstack-infra | 13:05 | |
| *** flaper87|afk is now known as flaper87 | 13:06 | |
| *** davidhadas_ has quit IRC | 13:06 | |
| *** weshay has quit IRC | 13:08 | |
| *** davidhadas has joined #openstack-infra | 13:08 | |
| *** mriedem has joined #openstack-infra | 13:08 | |
| *** pblaho has quit IRC | 13:09 | |
| *** pblaho has joined #openstack-infra | 13:09 | |
| openstackgerrit | A change was merged to openstack-infra/config: Use twine to upload python package to pypi. https://review.openstack.org/48507 | 13:12 |
| *** hashar has joined #openstack-infra | 13:14 | |
| *** michchap has quit IRC | 13:16 | |
| *** michchap_ has joined #openstack-infra | 13:16 | |
| *** fbo is now known as fbo_away | 13:16 | |
| *** ryanpetrello has joined #openstack-infra | 13:17 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove gate-django_openstack_auth-noop job. https://review.openstack.org/49274 | 13:18 |
| *** esker has joined #openstack-infra | 13:22 | |
| *** julim has joined #openstack-infra | 13:23 | |
| *** julim has quit IRC | 13:23 | |
| *** julim has joined #openstack-infra | 13:25 | |
| *** fbo_away is now known as fbo | 13:25 | |
| *** davidhadas_ has joined #openstack-infra | 13:29 | |
| *** davidhadas has quit IRC | 13:31 | |
| *** pblaho has quit IRC | 13:31 | |
| *** oubiwann_ has quit IRC | 13:35 | |
| *** davidhadas has joined #openstack-infra | 13:40 | |
| *** changbl has quit IRC | 13:40 | |
| *** adalbas has joined #openstack-infra | 13:40 | |
| *** markmcclain has joined #openstack-infra | 13:40 | |
| *** davidhadas_ has quit IRC | 13:42 | |
| *** oubiwann has quit IRC | 13:49 | |
| *** oubiwann has joined #openstack-infra | 13:49 | |
| *** dkliban has joined #openstack-infra | 13:49 | |
| *** amotoki_ has joined #openstack-infra | 13:52 | |
| *** garyk has joined #openstack-infra | 13:54 | |
| garyk | mordred: ping | 13:54 |
| garyk | sdague: ping | 13:55 |
| mordred | garyk: sup? | 13:55 |
| garyk | mordred: hi, ups and downs. you? | 13:55 |
| garyk | mordred: i am trying to look into the stable/grizzly failures | 13:55 |
| *** weshay has joined #openstack-infra | 13:56 | |
| garyk | seems like the problem occurs when the ovs external bridge is set to up | 13:56 |
| garyk | any idea why the VM's now running on gating have 2 interfaces. previously they had one | 13:56 |
| garyk | not sure that this is related | 13:56 |
| mordred | garyk: same. possibly occasionally too much wine | 13:57 |
| mordred | hrm | 13:57 |
| *** prad has joined #openstack-infra | 13:57 | |
| garyk | :) | 13:57 |
| garyk | i think the problem happens on https://github.com/openstack-dev/devstack/blob/stable/grizzly/lib/quantum#L328 (not 100% though). | 13:57 |
| mordred | I'm not sure - we've been running them on hpcloud the whole time - but perhaps hpcloud has grown a second interface | 13:57 |
| sdague | garyk: can you point to a fail? | 13:58 |
| mriedem | garyk: ah, so thinking that causes the network to drop and that's why the devstack neutron gate job silently fails | 13:58 |
| mriedem | sdague: https://bugs.launchpad.net/openstack-ci/+bug/1234181 | 13:59 |
| uvirtbot | Launchpad bug 1234181 in openstack-ci "stable/grizzly patches are failing jenkins in check-tempest-devstack-vm-neutron" [Undecided,Invalid] | 13:59 |
| *** alcabrera has joined #openstack-infra | 13:59 | |
| mriedem | there are 4 stable/grizzly patches in there, they all fail on the same job | 13:59 |
| mriedem | but from what i see in the logs, they fail in different parts of the quantum network setup in devstack | 13:59 |
| mriedem | sometimes setting up a router, sometimes setting up a subnet, etc | 13:59 |
| *** davidhadas_ has joined #openstack-infra | 14:00 | |
| garyk | mriedem: they are all around the same set of code - the l3 agent runs every interval so it is just updating the routing table etc following the config chnage | 14:00 |
| sdague | garyk: can you set -o xtrace in lib/neutron, should at least let the fail get explained better | 14:01 |
| garyk | sdague: sure. i am also adding in debug info | 14:01 |
| garyk | will push a patch in 5 | 14:01 |
| *** dkranz has joined #openstack-infra | 14:02 | |
| *** davidhadas has quit IRC | 14:03 | |
| *** amotoki has quit IRC | 14:03 | |
| *** amotoki_ is now known as amotoki | 14:04 | |
| *** hashar has quit IRC | 14:05 | |
| *** DennyZhang has joined #openstack-infra | 14:07 | |
| *** rwsu-away is now known as rwsu | 14:07 | |
| garyk | mriedem: sdague: i have pushed a patch with debug info. hopefully it will shed some light | 14:09 |
| garyk | is there anyway we can get access to a terminal that is running with gating? that will help us solve the issue in less than no time? | 14:09 |
| *** hashar has joined #openstack-infra | 14:10 | |
| sdague | garyk: you need fungi to hold the guest | 14:11 |
| sdague | that being said, these programs should really be written for first failure discovery, and if neutron can't handle machines with 2 interfaces... that seems weird | 14:12 |
| fungi | i need to see if the nodepool hold option is working as intended these days. i think it's functional now | 14:12 |
| fungi | might be easier if i just spun up a vm and ran it myself though, so i don't have to worry about it getting inadvertently scrapped | 14:13 |
| *** sileht has quit IRC | 14:14 | |
| *** thedodd has joined #openstack-infra | 14:14 | |
| fungi | garyk: worth noting, it's not hard at all to fire up a vm and run tests in exactly the same way. i went to extreme lengths to document the process in detail in the devstack-gate readme, because we do that regularly to vet embargoed security fixes | 14:14 |
| *** hashar has quit IRC | 14:15 | |
| *** sileht has joined #openstack-infra | 14:16 | |
| garyk | sdague: i think that neutron does handles this. problem is the l3 agent starts to mess with the routing table | 14:16 |
| garyk | it should be in a network namespace. i was unable to reproduce on a new and clean instance that i ran with the stable/grizzly code | 14:17 |
| garyk | fungi: first and foremost thanks for the documentaion. it is brilliant | 14:17 |
| amotoki | garyk: ping | 14:17 |
| garyk | amotoki: hi, whats up? | 14:18 |
| amotoki | am investigating stable grizzly issue. | 14:18 |
| amotoki | do we need to use quantumclient branch for python-neutronclient? | 14:18 |
| garyk | cool. have you been able to see anything | 14:18 |
| *** DinaBelova has quit IRC | 14:18 | |
| fungi | garyk: i'll give the tempest-neutron job a shot on grizzly branches and see what happens | 14:18 |
| garyk | fungi: thanks | 14:18 |
| amotoki | exercise script does not work in stable/grizzly branch of devstack | 14:19 |
| fungi | garyk: and once it (inevitably) fails, i'll get you ssh access into it | 14:19 |
| *** thedodd has quit IRC | 14:19 | |
| garyk | amotoki: sadly we have not even got there with the devstack run. the instance hangs after the subnet is create or a router is created | 14:19 |
| garyk | i am able to use quantum <X> commands | 14:19 |
| amotoki | garyk: yes. i see that. it is very strange. i haven't identify it yet. | 14:21 |
| *** thedodd has joined #openstack-infra | 14:21 | |
| sdague | garyk: the instance hangs, or you just cut off all the network connectivity to it? | 14:21 |
| garyk | amotoki: i think that when mark did the transition he added support for quantum/neutron | 14:22 |
| *** tvb has quit IRC | 14:22 | |
| fungi | sdague: network access seems not to be cut off, because it still transfers logs to jenkins afterward | 14:23 |
| amotoki | garyk: thank. i need more investigation. let you know if i have a progress. | 14:25 |
| *** oubiwann_ has joined #openstack-infra | 14:26 | |
| garyk | amotoki: thanks | 14:26 |
| *** anteaya has joined #openstack-infra | 14:28 | |
| *** mrodden has joined #openstack-infra | 14:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: Inspect the Gearman queue for immediate demand https://review.openstack.org/48517 | 14:38 |
| mordred | anteaya: morning | 14:38 |
| anteaya | morning | 14:38 |
| anteaya | I think my cold is finally gone, yay | 14:38 |
| mordred | anteaya: could you give me the owncloud url again? | 14:39 |
| anteaya | sure | 14:39 |
| anteaya | http://15.185.188.187/owncloud | 14:39 |
| mordred | anteaya: and would it possible for me to get admin access so that I could add a couple of accounts? I want to show work-in-progress at the board meeting | 14:39 |
| * fungi will hopefully remember to dial into it for a change | 14:40 | |
| *** datsun180b has joined #openstack-infra | 14:40 | |
| anteaya | mordred: you are admin | 14:40 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 14:41 |
| anteaya | exercise your powers, so that I know the setting is correct | 14:41 |
| anteaya | fungi: in the list of priorities it is kinda low right now, but your feedback is always welcome | 14:42 |
| fungi | anteaya: yeah, i'm hoping to dig myself free enough to poke at it, but may wait until after i hear mordred talk about it with the board | 14:42 |
| anteaya | makes sense | 14:43 |
| anteaya | let's hear what the board are wanting from it | 14:43 |
| *** changbl has joined #openstack-infra | 14:43 | |
| dkranz | mordred: Did your patch to run stable jobs against master client libs ever get merged? | 14:46 |
| dkranz | mordred: I seem to have lost the patch url for that. | 14:46 |
| *** DinaBelova has joined #openstack-infra | 14:46 | |
| sdague | dkranz: yes, that's our default | 14:46 |
| sdague | I'm pretty sure | 14:46 |
| jeblair | dkranz: https://review.openstack.org/#/c/41931/ | 14:47 |
| dkranz | sdague: If so we should be able to close the blueprint about that | 14:47 |
| dkranz | jeblair: Thanks. So it has not merged. | 14:47 |
| dkranz | jeblair: Is there an issue with it, or just got lost in reviews? | 14:48 |
| jeblair | dkranz: see the comment; it requires this change https://review.openstack.org/#/c/41945/ | 14:48 |
| sdague | oh, this is the other piece of that, sorry got confused | 14:49 |
| dkranz | jeblair: That has multiple +2 but no +A | 14:49 |
| jeblair | dkranz: and actually, i think the first patch needs to be updated because of changes to the second. | 14:49 |
| * fungi wanted to let mordred decide when/how it merged because of the interdependency between those changes | 14:50 | |
| mordred | what did I do? | 14:51 |
| mordred | His question was about standards and he was interested in how Swift and Cinder were compatible with ISO standards around storage? He also wanted to know if there was a CDMI conflict. | 14:51 |
| mordred | 14:51 | |
| mordred | GAH | 14:51 |
| mordred | two times in two days | 14:51 |
| mordred | speaking of - does anyone here know the answer to that? I was about to ask people. :) | 14:51 |
| fungi | mordred: time for a better mouse | 14:51 |
| mordred | fungi: it's my fault - I've started taking advantage of middle-click on url opens the url | 14:51 |
| mordred | it's a bad choice | 14:52 |
| fungi | ahh, yup | 14:52 |
| jeblair | haha | 14:52 |
| fungi | middle-click in your xterm pastes your secret conversations | 14:52 |
| mordred | yah | 14:52 |
| jeblair | mordred: more like: middle-click performs some action; indeterminate as to which. | 14:52 |
| mordred | jeblair: and yes, the first patch needs to be updated to match the second | 14:52 |
| mordred | jeblair: it's almost like sabdfl designed middle-mouse-click | 14:53 |
| jeblair | mordred, fungi, dkranz: but i think the second patch is safe to merge now, without the first being updated yet. | 14:53 |
| mordred | jeblair, fungi: I belive the d-g patch is safe to land at any time - it should be a meaningless patch | 14:53 |
| mordred | jeblair: yes | 14:53 |
| jeblair | but i'll still let mordred press the button on that. :) | 14:53 |
| mordred | (by meaningless, I mean it should be a no-op until we land it | 14:53 |
| dkranz | mordred: Sure. I was just kicking this because it is a valuable test that is already implemented but not yet running. | 14:55 |
| fungi | okay, agreed | 14:55 |
| dkranz | fungi: And there is a blueprint about it assigned to me :) | 14:55 |
| mordred | dstufft: agree | 14:56 |
| mordred | gah | 14:56 |
| mordred | dkranz: agree | 14:56 |
| fungi | dkranz: clearly your fault when it fails then ;) | 14:56 |
| mordred | dkranz: and thank you | 14:56 |
| dstufft | mordred: lol I was like "what am I being agreed with?" | 14:56 |
| mordred | dkranz: we can't make it go live until stable/grizzly works again though :) | 14:56 |
| dkranz | mordred: Thank you for doing it! I'm sure it would have taken me longer. | 14:56 |
| mordred | dstufft: everything | 14:56 |
| dkranz | mordred: Ugh | 14:56 |
| *** rcleere has joined #openstack-infra | 14:56 | |
| jeblair | mordred: man, you're really mordreding your irc conversations this morning. | 14:56 |
| dstufft | mordred: btw can you do a mirror run against https://preview-pypi.python.org/simple/ | 14:56 |
| *** mrodden has quit IRC | 14:57 | |
| dstufft | to see if it works for you | 14:57 |
| *** mrodden has joined #openstack-infra | 14:58 | |
| mordred | dstufft: yes. yes I can. | 14:58 |
| dstufft | mordred: awesome thanks | 15:00 |
| dstufft | mordred: it's not behind a CDN or anything so it may be slower fwiw | 15:00 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION https://review.openstack.org/49560 | 15:02 |
| *** salv-orlando has joined #openstack-infra | 15:02 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 15:04 |
| sdague | fungi / mordred / jeblair: https://review.openstack.org/#/c/49560/ and https://review.openstack.org/4950 | 15:06 |
| sdague | I miffed actually getting tenant isolation turned on correctly for those neutron jobs | 15:06 |
| sdague | salv-orlando has the right fixes there if people could take a look | 15:06 |
| jeblair | sdague: second url isn't correct | 15:07 |
| sdague | https://review.openstack.org/49504 | 15:07 |
| jeblair | that looks better | 15:07 |
| sdague | sorry, it's the last 2 revs in channel above my comment as well :) | 15:07 |
| sdague | I apparently am not as good with middle click, though not pasting foundation email in the process | 15:07 |
| fungi | sdague: any particular order you want those merged in? | 15:08 |
| fungi | also mtreinish beat me to the typo in that first one | 15:08 |
| *** boris-42 has quit IRC | 15:09 | |
| *** dstufft has quit IRC | 15:10 | |
| sdague | fungi: I think any order is fine | 15:10 |
| mtreinish | sdague: I think the 2nd one should also be reworked to handled like the other variables | 15:10 |
| sdague | nothing happens until both are in | 15:10 |
| jeblair | salv-orlando, sdague: there's a fix needed for each of those; but once they're updated, we should be able to merge config change first, and then when it lands, merge the d-g change | 15:10 |
| jeblair | salv-orlando, sdague: that order makes it self-gating. | 15:11 |
| jeblair | fungi: ^ | 15:11 |
| *** flaper87 is now known as flaper87|afk | 15:11 | |
| sdague | jeblair: right | 15:11 |
| *** rnirmal has joined #openstack-infra | 15:11 | |
| *** flaper87|afk is now known as flaper87 | 15:11 | |
| * fungi nods | 15:12 | |
| *** dstufft has joined #openstack-infra | 15:12 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION https://review.openstack.org/49560 | 15:15 |
| dstufft | mordred: oh yea, https://gist.github.com/dstufft/6811616 waiting for some feedback before making them "official" | 15:21 |
| mordred | dstufft: nice! | 15:22 |
| *** johnthetubaguy has quit IRC | 15:22 | |
| dstufft | they include setuptools in them | 15:22 |
| dstufft | should make things a bit nicer for windows users at least | 15:22 |
| *** davidhadas_ has quit IRC | 15:24 | |
| openstackgerrit | A change was merged to openstack-infra/config: Rename to DEVSTACK_GATE_TEMPEST_ALLOW_TENANT_ISOLATION https://review.openstack.org/49560 | 15:25 |
| *** jcoufal_ has quit IRC | 15:25 | |
| amotoki | garyk: still around? | 15:26 |
| openstackgerrit | Alex Gaynor proposed a change to openstack-infra/config: Gate and check oslo.version against PyPy https://review.openstack.org/49563 | 15:26 |
| *** johnthetubaguy has joined #openstack-infra | 15:26 | |
| fungi | garyk: it definitely killed my inbound ssh when setting up neutron... http://paste.openstack.org/show/47913 | 15:27 |
| garyk | amotoki: yes | 15:28 |
| amotoki | garyk: it seems that quantum_create_initial_network seems succeed. q-svc log says router-gateway-set was called. | 15:29 |
| garyk | fungi: yes, that is exactly the spot | 15:29 |
| garyk | amotoki: please look at http://paste.openstack.org/show/47913/ | 15:29 |
| amotoki | garyk: it seems that quantum_setup_debug is called. | 15:29 |
| garyk | last command was quantum router-gateway-set | 15:30 |
| fungi | garyk: i can still ssh back into it. seems to have two ethX interfaces as previously asserted, though no idea if that has anything to do with what's going on... http://paste.openstack.org/show/47914 | 15:31 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs https://review.openstack.org/49566 | 15:31 |
| fungi | garyk: this particular vm is an 8gb flavor on rackspace nova | 15:31 |
| amotoki | garyk: according to http://logs.openstack.org/39/48539/5/check/check-tempest-devstack-vm-neutron/b9e6559/logs/screen-q-svc.txt.gz, after router-gateway-set, GET /networks and GET /subnets are called twice for each. | 15:32 |
| garyk | fungi: thanks | 15:32 |
| *** dizquierdo has left #openstack-infra | 15:32 | |
| garyk | amotoki: correct. | 15:32 |
| *** tvb has joined #openstack-infra | 15:32 | |
| fungi | garyk: if you want ssh access into it in the current (post-failure) state, i'm happy to oblige | 15:33 |
| fungi | i'm eager to get things flowing in the grizzly gate again, since this puts us in a bad way for security backports until it's fixed | 15:34 |
| garyk | in my local setup i did not set Q_USE_DEBUG_COMMAND as True. I'll try that now | 15:34 |
| garyk | fungi: amotoki has pointed out something that i missed on my local setup. | 15:34 |
| fungi | ooh, excellent | 15:34 |
| garyk | hopefully i'll be able to reproduce. restacking now | 15:34 |
| amotoki | garyk: we need to investigate quantum_setup_debug. the next of it is start_quantum_agent and there is no log of q-agt. | 15:35 |
| *** tvb has quit IRC | 15:35 | |
| *** tvb has joined #openstack-infra | 15:35 | |
| *** tvb has quit IRC | 15:35 | |
| *** tvb has joined #openstack-infra | 15:35 | |
| amotoki | devstack log is from screen log, so there is some time lag and somthing may not be logged. it is very trick.... | 15:36 |
| fungi | garyk: for the record, i followed the "simulating devstack gate tests" section of http://git.openstack.org/cgit/openstack-infra/devstack-gate/plain/README.rst and then substituted these shell commands at the end http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml#n201 | 15:36 |
| garyk | amotoki: https://github.com/openstack-dev/devstack/blob/stable/grizzly/stack.sh#L996 (i had print outs before this and I did not see them) | 15:36 |
| fungi | oh, and used ZUUL_BRANCH=stable/grizzly of course | 15:37 |
| garyk | amotoki: i owe you a beer in hong kong. when debug is true then it terminates | 15:37 |
| fungi | oh, yikes | 15:37 |
| amotoki | garyk: :-) | 15:38 |
| *** DinaBelova has quit IRC | 15:40 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 15:42 |
| garyk | amotoki: i added in debug info and it passed. hmmm. | 15:43 |
| garyk | amotoki: please see https://review.openstack.org/#/c/49547/ (setup_debug_quantum) | 15:44 |
| garyk | when I add the echo "…" locally this passes (it failed without it). | 15:45 |
| *** johnthetubaguy has quit IRC | 15:45 | |
| *** johnthetubaguy1 has joined #openstack-infra | 15:45 | |
| sdague | jeblair: so when trying to carve tempest off into another user, we get issues like this | 15:45 |
| sdague | http://paste.openstack.org/show/47916/ | 15:46 |
| amotoki | garyk: hmm.. am runnign too. but it is a great hint. | 15:46 |
| sdague | you cool with chowning the whole tempest dir? | 15:46 |
| *** david-lyle has quit IRC | 15:46 | |
| sdague | or should we issolate further | 15:46 |
| jeblair | sdague: yes; does devstack need to write to it? we may either need to give it permissions, or chown it after devstack runs and before tempest. | 15:47 |
| sdague | yeh, I was going to chown just before we run tempest | 15:47 |
| jeblair | sdague: ++ | 15:47 |
| zaro | anybody know why pip install git-review doesn't work? http://paste.openstack.org/show/47890/ | 15:49 |
| zaro | i get error: invalid command 'egg_info' | 15:49 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 15:49 |
| *** dkehn_ has joined #openstack-infra | 15:50 | |
| garyk | amotoki: + quantum-debug --os-tenant-name admin --os-username admin --os-password openstack probe-create --device-owner compute 72011347-4a72-4598-b3a2-c702b440226f | 15:50 |
| garyk | 'ClientManager' object has no attribute 'quantum' | 15:50 |
| *** dkehn has quit IRC | 15:51 | |
| amotoki | garyk: i got the same. it is exactlly same when i ran exercise scripts after stack.sh succeeds. | 15:51 |
| *** yassine has quit IRC | 15:53 | |
| *** dkehn has joined #openstack-infra | 15:54 | |
| fungi | zaro: i can't reproduce that issue. what operating system/distribution are you trying it on? | 15:55 |
| garyk | amotoki: looks like the quantum-debug command is redirected to the client instead of calling the executable | 15:55 |
| garyk | maybe we can just use the full path - but that is fugly | 15:55 |
| *** dkehn_ has quit IRC | 15:56 | |
| fungi | zaro: ahh, nevermind. just noticed your paste url | 15:58 |
| *** davidhadas has joined #openstack-infra | 15:59 | |
| zaro | fungi: i'm on ubuntu hp vm | 15:59 |
| *** matty_dubs is now known as matty_dubs|lunch | 15:59 | |
| jeblair | fungi: did you look into https://review.openstack.org/#/c/49407/1 ? | 16:00 |
| jeblair | fungi: are there changes in gerrit for those repos? | 16:00 |
| fungi | jeblair: yes, renaming fuel-ostf-tests to fuel-ostf and deleting fuel-ostf-plugin | 16:01 |
| jeblair | fungi: does fuel-ostf-plugin have changes? | 16:01 |
| *** marun has quit IRC | 16:01 | |
| fungi | oh, i thought you meant did we need to make changes to their layout in gerrit | 16:01 |
| *** marun has joined #openstack-infra | 16:02 | |
| fungi | i have not looked at the current state of those repositories, but they've separately asked me to blow away and reimport their contents | 16:02 |
| fungi | because they asked to be added to stackforge but then kept developing on their old github repos instead | 16:02 |
| jeblair | there is a change in the fuel-ostf-plugin repo | 16:03 |
| jeblair | so i don't think we should delete it | 16:03 |
| *** Ryan_Lane has joined #openstack-infra | 16:04 | |
| *** DinaBelova has joined #openstack-infra | 16:05 | |
| jeblair | the only time we've deleted a project, to my knowledge, is due to a typo, and before it actually got used | 16:05 |
| jeblair | we don't have a process for deleting a real, used project from gerrit | 16:05 |
| fungi | ahh, okay. what's the best way to deprecate a project in that case? | 16:06 |
| jeblair | and we've never even decided whether that would even be desirable | 16:06 |
| jeblair | further, stackforge projects are intended to be self-service and not take away from time we spend working on openstack | 16:06 |
| jeblair | so i don't believe we should spend the time needed to come up with a policy and process for this case | 16:07 |
| *** markmcclain has quit IRC | 16:07 | |
| fungi | granted. i can tell them they'll have to live with a dead project there | 16:07 |
| amotoki | all, i have a question about stable/grizzly gate. which quantumclient/neutornclient should we use? quantumclient branch or master branch? | 16:07 |
| garyk | amotoki: i spoke with mark and he has added - https://review.openstack.org/#/c/49584/ | 16:08 |
| amotoki | when investigating stable/grizzly gate failure, quantum-debug command calls quantumclient. We need to fix either quantumclient or quantum-debug. | 16:08 |
| fungi | jeblair: we probably need *much* stronger wording about projects needing to stabilize their organization and pause development to migrate to stackforge. we've had far too many jump the gun and ask to be added before they knew how they wanted to work, and then keep coming back to us asking to fix things which aren't/can't be made self-serve | 16:09 |
| amotoki | thanks garyk. | 16:09 |
| garyk | i am testing it now. i need to head on home i'll continue a bit later | 16:10 |
| jeblair | fungi: i have no problem with that. | 16:10 |
| jeblair | fungi: http://paste.openstack.org/show/47917/ look good? | 16:11 |
| jeblair | gah timezone url is wrong | 16:12 |
| *** afazekas has quit IRC | 16:12 | |
| amotoki | garyk: one more question. stable/grizzly gate uses master branch of neutronclient. Should we use quatnumclient branch? | 16:12 |
| jeblair | fungi correct url is http://www.timeanddate.com/worldclock/fixedtime.html?iso=20131005T16 | 16:12 |
| fungi | ahh, yeah, date was wrong | 16:12 |
| *** AlexF has joined #openstack-infra | 16:13 | |
| fungi | yup, lgtm other than that url | 16:13 |
| *** garyk has quit IRC | 16:13 | |
| *** odyssey4me has quit IRC | 16:15 | |
| *** flaper87 is now known as flaper87|afk | 16:16 | |
| clarkb | fungi ++++++++ | 16:16 |
| *** boris-42 has joined #openstack-infra | 16:16 | |
| *** flaper87|afk is now known as flaper87 | 16:17 | |
| fungi | top issues are, 1. imported cruft branches/tags which they want deleted, 2. kept merging changes outside gerrit and want to re-level the repositories because submitting changes for X00 commits would be too time consuming, 3. want to split/combine/delete projects | 16:18 |
| *** AlexF has quit IRC | 16:20 | |
| fungi | any others spring to mind? i'll work up a paragraph or two for the stackforge howto intro with strong cautionary wording | 16:20 |
| jeblair | fungi: i have found that you can join the board conf call without registering | 16:20 |
| fungi | jeblair: i pressed # | 16:20 |
| jeblair | fungi: yep | 16:20 |
| *** osanchez has quit IRC | 16:21 | |
| clarkb | jeblair: fungi: does it still require webex? | 16:22 |
| *** DinaBelova has quit IRC | 16:22 | |
| *** sarob has joined #openstack-infra | 16:22 | |
| fungi | clarkb: depends on whether you want to see the death by powerpoint | 16:22 |
| fungi | i used a telephone | 16:22 |
| *** SergeyLukjanov has quit IRC | 16:22 | |
| jeblair | me too | 16:22 |
| fungi | archaic i know | 16:22 |
| *** sarob has quit IRC | 16:23 | |
| *** sarob has joined #openstack-infra | 16:24 | |
| clarkb | https://wiki.openstack.org/wiki/Governance/Foundation/3Oct2013BoardMeeting doesn't document a phone number? does the webex site provide that? in any case I didn't bother dialing in because the wiki said webex | 16:24 |
| *** shardy is now known as shardy_afk | 16:24 | |
| jeblair | clarkb: webex site has the # | 16:24 |
| * clarkb notes this for next time though that will be in hong kong, maybe we can all listen in together | 16:25 | |
| jeblair | i guess the budget document jbryce referred to was sent to the sekrit board list :( | 16:25 |
| *** thingee_zzz is now known as thingee | 16:26 | |
| *** afazekas has joined #openstack-infra | 16:27 | |
| openstackgerrit | A change was merged to openstack-infra/config: Increase log worker retry timeout length. https://review.openstack.org/49289 | 16:29 |
| jeblair | there's an #openstack-foundation channel if anyone feels like chatting about the board meeting | 16:30 |
| fungi | clarkb: if you get the invitation e-mails (i forget which ml they go to, maybe the public foundation one?) it says phone numbers and meeting code | 16:33 |
| *** AlexF has joined #openstack-infra | 16:34 | |
| fungi | clarkb: http://lists.openstack.org/pipermail/foundation/2013-October/001480.html | 16:34 |
| *** afazekas_ has joined #openstack-infra | 16:36 | |
| *** DennyZhang has quit IRC | 16:38 | |
| jog0 | jeblair: http://status.openstack.org/elastic-recheck/ | 16:38 |
| *** DennyZhang has joined #openstack-infra | 16:38 | |
| jog0 | the dates aren't the same for each graph | 16:38 |
| jeblair | jog0: https://review.openstack.org/#/c/49566/ | 16:39 |
| jog0 | awesome | 16:39 |
| jeblair | jog0: (that will at least align the end-times to "now", though the start times will still be "first-hit". | 16:40 |
| jog0 | jeblair: thats not ideal IMHO but much better | 16:40 |
| clarkb | fungi: thanks | 16:40 |
| jeblair | jog0: yeah, shouldn't be hard to align the start times too in a later change | 16:41 |
| jog0 | agreed | 16:41 |
| fungi | clarkb: it's a fairly low-volume list and a good project-governance-oriented one to subscribe to if you don't already | 16:42 |
| clarkb | fungi: I don't think I am on it. The problem with all of these lists is between -dev, general, and -infra ETOOMUCHMAIL | 16:42 |
| *** AlexF has quit IRC | 16:42 | |
| *** weshay has quit IRC | 16:43 | |
| fungi | i have actually started filtering -infra into a separate maildir so i can give it more priority, but i just dump all the rest of the openstack lists together into one and sift through them as time permits | 16:43 |
| pleia2 | clarkb: if we could carve out some time today or tomorrow to chat about https://review.openstack.org/#/c/49454/ and confirm it's what we discussed in seattle I'd appreciate it (also if there are any glaring problems I need to fix) | 16:43 |
| *** jpich has quit IRC | 16:44 | |
| anteaya | time isn't permitting much these days | 16:45 |
| clarkb | pleia2: sure, catching up on mail now should have time shortly | 16:45 |
| *** weshay has joined #openstack-infra | 16:48 | |
| openstackgerrit | Anita Kuno proposed a change to openstack-infra/puppet-dashboard: Changing the module to use the sodabrew fork of puppet-dashboard https://review.openstack.org/49392 | 16:49 |
| *** Ryan_Lane has quit IRC | 16:50 | |
| *** dizquierdo has joined #openstack-infra | 16:51 | |
| *** alcabrera is now known as alcabrera|afk | 16:51 | |
| *** gyee has joined #openstack-infra | 16:53 | |
| *** hashar has joined #openstack-infra | 16:54 | |
| *** jerryz has joined #openstack-infra | 16:55 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Fix zuul status graph with failed change at head https://review.openstack.org/49590 | 16:56 |
| clarkb | what ever happened to the jenkins slowness a couple days ago? | 16:56 |
| *** alcabrera|afk is now known as alcabrera | 16:57 | |
| anteaya | clarkb: the periodic jobs running over night North American time seems to have played a role | 16:58 |
| anteaya | not sure if that has been addressed | 16:58 |
| fungi | clarkb: it magically cleared up with no indication | 16:58 |
| *** mrodden1 has joined #openstack-infra | 16:58 | |
| fungi | anteaya: the time did not actually correlate to our periodic jobs | 16:58 |
| fungi | current baseless guess is that it's periodic jvm gc | 16:59 |
| anteaya | ah | 16:59 |
| anteaya | yes you had mentioned that as a possibility | 17:00 |
| anteaya | a monthly run perhaps | 17:00 |
| fungi | user cpu shot up to 80% solid and stayed there for about 14 hours straight, then quietly returned to normal | 17:00 |
| *** odyssey4me has joined #openstack-infra | 17:00 | |
| * anteaya looks at cacti | 17:00 | |
| fungi | i only speculated maybe something monthly because it was the 1st of the month | 17:00 |
| *** mrodden has quit IRC | 17:00 | |
| *** matty_dubs|lunch is now known as matty_dubs | 17:01 | |
| anteaya | it is a worthy speculation | 17:01 |
| anteaya | and it didn't occur in the last 24 hours | 17:01 |
| anteaya | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=816&rra_id=all | 17:01 |
| *** jerryz has quit IRC | 17:02 | |
| clarkb | jeblair: I have approved https://review.openstack.org/#/c/49566/ I am assuming that it will update properly on its own | 17:02 |
| *** moted has quit IRC | 17:03 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Fix zero-fill logic in graphs https://review.openstack.org/49566 | 17:04 |
| *** moted has joined #openstack-infra | 17:04 | |
| *** hashar has quit IRC | 17:04 | |
| *** odyssey4me2 has joined #openstack-infra | 17:04 | |
| *** odyssey4me has quit IRC | 17:05 | |
| *** oubiwann has quit IRC | 17:05 | |
| *** nati_ueno has joined #openstack-infra | 17:05 | |
| *** oubiwann has joined #openstack-infra | 17:06 | |
| *** garyk has joined #openstack-infra | 17:08 | |
| *** derekh has quit IRC | 17:11 | |
| jeblair | clarkb: yeah, it shoud (vcsrepo) | 17:13 |
| *** garyk has quit IRC | 17:13 | |
| mtreinish | jeblair: https://review.openstack.org/#/c/49504/ I think it's better now | 17:15 |
| *** markmcclain has joined #openstack-infra | 17:15 | |
| *** amotoki has quit IRC | 17:17 | |
| *** hashar has joined #openstack-infra | 17:18 | |
| *** garyk has joined #openstack-infra | 17:19 | |
| *** jerryz has joined #openstack-infra | 17:20 | |
| *** jerryz has quit IRC | 17:20 | |
| *** jerryz has joined #openstack-infra | 17:20 | |
| *** flaper87 is now known as flaper87|afk | 17:21 | |
| *** SergeyLukjanov has joined #openstack-infra | 17:23 | |
| *** hashar has quit IRC | 17:23 | |
| jog0 | clarkb: ping | 17:26 |
| jog0 | how hard would it be to inject extra jobs into zuul now that dev is quite | 17:27 |
| jog0 | and thus elasticsearch doesn't have a big dataset | 17:27 |
| clarkb | jog0: not terribly difficult depending on how you want to do it. We could just set up hourly periodic jobs initially | 17:30 |
| clarkb | jog0: its more complicated if you want to do something that uses available resources | 17:30 |
| *** nati_ueno has quit IRC | 17:31 | |
| *** hogepodge has joined #openstack-infra | 17:31 | |
| *** markmcclain1 has joined #openstack-infra | 17:31 | |
| *** nati_ueno has joined #openstack-infra | 17:31 | |
| jog0 | clarkb: available resources would be preferred. as don't need this if zuul is busy | 17:31 |
| *** nati_ueno has quit IRC | 17:31 | |
| jog0 | clarkb: can you periodically check zuuls queue and base it on that | 17:32 |
| *** markmcclain has quit IRC | 17:32 | |
| *** nati_ueno has joined #openstack-infra | 17:32 | |
| clarkb | jog0: you can. jeblair is working on nodepool stuff that looks at demand to determine how to schedule pool builds. We should probably try to make it similar to that, but I haven't had a chance to look at it yet | 17:34 |
| clarkb | jeblair: ^ any thoughts on using available d-g slave resources to do opportunistic testing to run more tests during quiet times? | 17:34 |
| jog0 | cool thanks | 17:35 |
| *** ArxCruz has quit IRC | 17:35 | |
| clarkb | pleia2: reviewed | 17:37 |
| jeblair | clarkb, jog0: i think there's a bug about it, but anyway, yeah, i think it should be a new trigger type in zuul to do that. | 17:39 |
| pleia2 | clarkb: thanks! the current plan is to not run toci yet, we're just getting the framework up to run this script: https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test.sh | 17:39 |
| *** ArxCruz has joined #openstack-infra | 17:39 | |
| pleia2 | small steps :) | 17:39 |
| pleia2 | once we have POC we'll go from there | 17:39 |
| clarkb | pleia2: yup | 17:41 |
| jomara | jeblair: hey, just a summary from yesterday | 17:42 |
| *** johnthetubaguy1 has quit IRC | 17:42 | |
| *** AlexF has joined #openstack-infra | 17:42 | |
| jog0 | jeblair: awesome | 17:42 |
| jog0 | with elastic-recheck etc we should get some great data from doing this | 17:42 |
| jomara | jeblair: i'm setting up openshift ask bot really quick as a stop gap, in the background we will work on puppet for askbot + postgres for openstack-infra - that's work that I'm supposed to do, right? and then we hand that off to you guys, probably described in a doc i havent read yet? | 17:43 |
| *** melwitt has joined #openstack-infra | 17:43 | |
| pabelanger | clarkb, fungi: For your puppetmaster, how to do manage client (node) certs? Do you just back them up or actually store them in a repo? | 17:45 |
| pabelanger | looking at launch-node, it looks like they are just accessed from the local puppetmaster | 17:45 |
| fungi | pabelanger: right now we just live with the need to regenerate and distribute them in a disaster | 17:46 |
| pabelanger | fungi, okay, that's what I thought | 17:46 |
| fungi | but that said, we are looking at secure ways to back them up | 17:46 |
| pabelanger | Trying to come up with a plan for easy disaster recovery when the puppetmaster is lost | 17:47 |
| pabelanger | figured I see what you guys were doing | 17:47 |
| fungi | i don't recall whether we settled on a specific solution yet. my favorite was to encrypt the sensitive data and back that up with bup while excluding the plaintext locations for that data | 17:48 |
| *** jergerber has joined #openstack-infra | 17:49 | |
| *** afazekas_ has quit IRC | 17:49 | |
| *** afazekas has quit IRC | 17:49 | |
| clarkb | hub_cap: jeblair: do cloud dbs provide SSL connectivity to clients? I can't find anything indicating a cert is generated or how to provide a pregenerated cert | 17:49 |
| hub_cap | nawr not yet | 17:50 |
| hub_cap | clarkb: ^ ^ its somewhere on the list | 17:50 |
| hub_cap | but low i think | 17:50 |
| hub_cap | dealbreaker? | 17:50 |
| clarkb | hub_cap: not sure yet. But rackspace's network security certainly isn't trusted. I am not sure we want to rely on the internal network for data and credential security | 17:51 |
| hub_cap | clarkb: rot13 dude | 17:51 |
| clarkb | mordred: fungi jeblair ^ thoughts on using cloud dbs in the clear via rackspace's internal network? | 17:52 |
| pabelanger | fungi, okay, will keep an ear out when you guys work it up again. | 17:52 |
| fungi | it does worry me, though for the moment that's only traversing rackspace's network | 17:53 |
| *** che-arne has quit IRC | 17:53 | |
| clarkb | fungi: re puppetmaster backups the idea to encrypt stuff we care about then bup that sounded good to me | 17:53 |
| fungi | but certainly dns hijacking would be a quick way to mitm it outside rackspace | 17:53 |
| *** Ajaeger has joined #openstack-infra | 17:53 | |
| fungi | keeping the db hostname hidden in hiera helps there | 17:54 |
| mordred | clarkb: reading | 17:54 |
| *** qba73_ has joined #openstack-infra | 17:54 | |
| Ajaeger | infra team, could you review and approve https://review.openstack.org/#/c/48674/, please? | 17:54 |
| Ajaeger | the build will fail now since somebody was a bit to eager to remove the documentation instead of waiting for the patch above to go in. | 17:55 |
| mordred | hub_cap: are we saying that other customers at rackspace can snoop a network between our compute and db nodes and potentially steal traffic? | 17:56 |
| fungi | Ajaeger: lgtm. approved | 17:56 |
| clarkb | mordred: I think it depends on how the internal network is implemented. Without direct control we can't assume it will always do what we expect (iirc this is why we run local firewalls on all machines) | 17:56 |
| Ajaeger | fungi and clarkb : you're ultraquick! Thanks a lot! | 17:56 |
| fungi | mordred: in theory our db traffic should not be reflected to another vm | 17:57 |
| clarkb | but what happens when someone cohabitant with us floods arp tables? | 17:57 |
| clarkb | and so on | 17:57 |
| fungi | i hope tricks like arp cache poisoning wouldn't work through the rackspace cloud at any rate | 17:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Removes openstack-api-programming doc build https://review.openstack.org/48674 | 17:57 |
| fungi | but yes | 17:57 |
| fungi | arp flooding too | 17:58 |
| *** qba73_ has quit IRC | 17:58 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: decouple mysql setup from gerrit module https://review.openstack.org/49448 | 17:58 |
| hub_cap | mordred: i can find out, gimme ~10 | 17:58 |
| *** thomasbiege1 has joined #openstack-infra | 17:59 | |
| fungi | in theory the vswitch should filter arp replies for ip addresses not assigned to the vm? not familiar enough with nova network or whatever's in use there | 17:59 |
| clarkb | (I am trying to be more paranoid about things :) ) | 18:00 |
| *** nati_ueno has quit IRC | 18:01 | |
| *** nati_ueno has joined #openstack-infra | 18:02 | |
| fungi | yes, you make very good points | 18:02 |
| fungi | as i said, it does worry me | 18:03 |
| annegentle_ | pleia2: around? | 18:03 |
| pleia2 | annegentle_: yep | 18:04 |
| *** AlexF has quit IRC | 18:05 | |
| annegentle_ | pleia2: hey I see your codechix workshop handout is cc-by-sa, I'm copying some of the links, do you know if I have to use cc-by-sa for my copy or can I use cc-by? | 18:05 |
| *** qba73_ has joined #openstack-infra | 18:05 | |
| *** AlexF has joined #openstack-infra | 18:05 | |
| annegentle_ | pleia2: I am not a lawyer but I do want to ask permission! | 18:06 |
| pleia2 | annegentle_: I hereby grant you the permission to relicense :) | 18:06 |
| annegentle_ | pleia2: I can show you what I have and you can see if you want attribution (and where to put it)? | 18:06 |
| * jeblair goes afk for a few hours | 18:06 | |
| annegentle_ | pleia2: or I could just use cc-by-sa just like yours, but all the OpenStack docs are cc-by 3.0 | 18:06 |
| annegentle_ | pleia2: woo thanks! | 18:07 |
| pleia2 | annegentle_: I think it's all generic enough that it's not really neccesary to attribute | 18:07 |
| annegentle_ | pleia2: yeah I'm with ya | 18:07 |
| annegentle_ | pleia2: would you mind giving mine a once-over if you have a minute? | 18:07 |
| pleia2 | annegentle_: sure thing | 18:07 |
| *** AlexF has quit IRC | 18:07 | |
| zaro | fungi: 'related-bug' & 'partial-bug' git commit comments are not reported back to LP ? | 18:08 |
| clarkb | zaro: they should be | 18:08 |
| clarkb | zaro: do you have an example? | 18:08 |
| zaro | clarkb: https://review.openstack.org/#/c/49448/ | 18:09 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 18:09 |
| fungi | zaro: your first patchset lacked the header. comments are only added for the initial patchset so as to avoid spamming the bug subscribers | 18:10 |
| fungi | since the update_bug script does not scrape the bug comments to see if it already added a comment, nor does it keep a database to which it can refer | 18:11 |
| * fungi needs to go grab an extremely late lunch now. bbiab | 18:11 | |
| zaro | fungi: ahh, but i always forget to add the bug info on my first patch. | 18:11 |
| *** thomasbiege1 has quit IRC | 18:12 | |
| clarkb | we could probably have it query the bug to see if the existing change has been linked already (use lp as the state DB) | 18:12 |
| zaro | ++ | 18:12 |
| *** AlexF has joined #openstack-infra | 18:13 | |
| * zaro enters bug for this matter | 18:13 | |
| *** w_ is now known as olaph | 18:15 | |
| clarkb | zaro: thanks | 18:15 |
| *** mrodden1 has quit IRC | 18:16 | |
| *** mrodden has joined #openstack-infra | 18:17 | |
| *** Ajaeger has quit IRC | 18:17 | |
| *** changbl has quit IRC | 18:18 | |
| mordred | clarkb: http://www.rackspace.com/blog/network-security-in-the-cloud/ | 18:20 |
| *** markmcclain1 has quit IRC | 18:26 | |
| clarkb | mordred: I figured protections are in place. But since we don't have visibility into them we tend to be extra cautious. This is why we run local firewalls for example. In this case it may be an acceptable risk. Just pointing it out there as I was looking for SSL connectivity as part of the etherpad move | 18:27 |
| mordred | gotcha | 18:27 |
| mordred | I believe SSL connectivity with mysql is broken anyway, btw | 18:28 |
| *** CaptTofu has quit IRC | 18:28 | |
| clarkb | mordred: :( | 18:28 |
| *** CaptTofu has joined #openstack-infra | 18:28 | |
| pleia2 | clarkb: re: first comment on https://review.openstack.org/#/c/49454/1/modules/openstack_project/files/jenkins_job_builder/config/tripleo.yaml so does the job template have to do with the target (ie rackspace vs hpcloud)? or something else? | 18:29 |
| *** markmc has quit IRC | 18:29 | |
| pleia2 | pipelines, nodes, what :) | 18:29 |
| clarkb | pleia2: the templates that your template is based on exist so that different pipelines (eg gate, check) can run on different nodes | 18:30 |
| clarkb | pleia2: right now we want all gate tests to run on hpcloud for throughput reasons, but have started running check tests on both hpcloud and rackspace | 18:30 |
| clarkb | pleia2: in the case of tripleo there is only one provider so probably don't need distinctions like we have with the other tempest jobs | 18:31 |
| pleia2 | ah, I see | 18:31 |
| clarkb | (all pipelines will run this test on the same cloud and node) | 18:31 |
| pleia2 | so we do hope to have a 2nd provider at some point, but I don't think it's close | 18:32 |
| clarkb | pleia2: yeah, I think we can worry about that when it happens as we may want to distribute load differently than we do with hpcloud and rackspace for the other tests | 18:32 |
| pleia2 | so how do I fix this? | 18:32 |
| *** CaptTofu has quit IRC | 18:33 | |
| clarkb | pleia2: for now I lean towards making it not a template and keeping it simple. change the job-template: string to job: and replace instances of {pipeline} with gate and instances of {node} with the label you have configured in nodepool.yaml.erb | 18:33 |
| *** AlexF has quit IRC | 18:33 | |
| pleia2 | clarkb: ok, thanks | 18:33 |
| hub_cap | clarkb: mordred: got some more info about the networks, and its public info... http://www.rackspace.com/blog/network-security-in-the-cloud/ | 18:34 |
| hub_cap | ^ ^ cant spoof between vm's so u cant spoof between hosts or anything etc... | 18:34 |
| uvirtbot | hub_cap: Error: "^" is not a valid command. | 18:34 |
| hub_cap | i hate u u virt bot | 18:34 |
| clarkb | mordred: https://review.openstack.org/#/c/46996/ is it too late for that for havana? (I can approve but don't want to break RCs) | 18:35 |
| *** oubiwann is now known as Guest7950 | 18:36 | |
| mordred | clarkb: it should not matter | 18:38 |
| mordred | clarkb: it changes the version on the non-released dev versions | 18:38 |
| *** Ryan_Lane has joined #openstack-infra | 18:38 | |
| *** CaptTofu has joined #openstack-infra | 18:39 | |
| clarkb | mordred: cool. I gave it a +2 do you want me to approve it as well or should I let you do that? | 18:40 |
| mordred | clarkb: I'll do it - thanks | 18:41 |
| *** sarob has quit IRC | 18:44 | |
| clarkb | jog0: is the elastic-recheck bot stuck again? it has been very quiet | 18:51 |
| *** julim has quit IRC | 18:51 | |
| clarkb | but we are also not running so many tests and we seem to not fail as frequently | 18:51 |
| sdague | ok, tempest user for tests is ready now - https://review.openstack.org/#/c/49021/ | 18:53 |
| clarkb | sdague: it has been starred, will be reviewed after lunch | 18:53 |
| sdague | clarkb: the gate queue is so light today | 18:53 |
| clarkb | sdague: https://review.openstack.org/#/c/49466/ iterested to know what you think about that change | 18:53 |
| sdague | I think we need to start thinking about our analysis in percentages of runs | 18:54 |
| clarkb | sdague: yup we do. I also think jog0's idea to schedule jobs when we are quiet is a good one (to keep the numbers up) | 18:54 |
| *** sarob has joined #openstack-infra | 18:59 | |
| *** garyk has quit IRC | 19:00 | |
| *** DennyZhang has quit IRC | 19:01 | |
| sdague | that was my idea :) | 19:01 |
| *** DinaBelova has joined #openstack-infra | 19:01 | |
| clarkb | oh | 19:03 |
| *** yolanda has quit IRC | 19:03 | |
| clarkb | sdague: well it is a good one :) | 19:03 |
| sdague | honestly we may have had it independently, but mordred / jeblair and I were talking about it over the weekend | 19:03 |
| sdague | the "idle swarm" | 19:03 |
| *** AlexF has joined #openstack-infra | 19:05 | |
| *** AlexF has quit IRC | 19:05 | |
| *** nati_ueno has quit IRC | 19:06 | |
| *** gyee has quit IRC | 19:06 | |
| *** sirushti has quit IRC | 19:07 | |
| *** nati_ueno has joined #openstack-infra | 19:07 | |
| *** dizquierdo has left #openstack-infra | 19:08 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add initial files for tripleo in the gate. https://review.openstack.org/49454 | 19:09 |
| *** davidhadas has quit IRC | 19:11 | |
| *** mrodden1 has joined #openstack-infra | 19:12 | |
| *** davidhadas has joined #openstack-infra | 19:13 | |
| *** rfolco has quit IRC | 19:13 | |
| *** sirushti has joined #openstack-infra | 19:13 | |
| *** mrodden has quit IRC | 19:14 | |
| *** qba73_ has quit IRC | 19:21 | |
| *** marun has quit IRC | 19:21 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Added base_dir parameter to get_projects_info https://review.openstack.org/49518 | 19:21 |
| *** odyssey4me has joined #openstack-infra | 19:27 | |
| openstackgerrit | A change was merged to openstack/requirements: Update sphinxcontrib-pecanwsme https://review.openstack.org/49502 | 19:29 |
| *** odyssey4me2 has quit IRC | 19:29 | |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add initial files for tripleo in the gate. https://review.openstack.org/49454 | 19:32 |
| lifeless | could someone add tripleo-core into the tuskar-core group and remove all the other members? likewise with the tuskar-ptl group ? | 19:32 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Added the server parameter to get_changes https://review.openstack.org/49519 | 19:32 |
| *** marun has joined #openstack-infra | 19:32 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Pep8 compliant https://review.openstack.org/49520 | 19:34 |
| *** vipul is now known as vipul-away | 19:35 | |
| *** _david_ has joined #openstack-infra | 19:35 | |
| *** odyssey4me has quit IRC | 19:36 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Added some optinos to openreviews https://review.openstack.org/49521 | 19:37 |
| *** julim has joined #openstack-infra | 19:39 | |
| *** sarob has quit IRC | 19:40 | |
| *** marun has quit IRC | 19:41 | |
| *** DinaBelova has quit IRC | 19:41 | |
| *** yolanda has joined #openstack-infra | 19:42 | |
| *** vipul-away is now known as vipul | 19:44 | |
| *** vipul is now known as vipul-away | 19:44 | |
| *** _david_ has quit IRC | 19:45 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: TripleO merged with Tuskar. https://review.openstack.org/49606 | 19:46 |
| *** AlexF has joined #openstack-infra | 19:47 | |
| NobodyCam | good afternoon infra just a quick question if I may | 19:48 |
| NobodyCam | should I be getting a forbidden on urls like : http://summit.openstack.org/cfp/topic/7 | 19:49 |
| lifeless | I'd say so, the cfp is over papers were selceted | 19:50 |
| NobodyCam | :) | 19:50 |
| *** marun has joined #openstack-infra | 19:50 | |
| *** yolanda has quit IRC | 19:51 | |
| fungi | lifeless: i've added tripleo-core to tuskar-core but did not remove anyone. i'll leave that to you... https://review.openstack.org/#/admin/groups/195,members | 19:52 |
| fungi | also there were no other tuskar groups (-ptl or otherwise) | 19:52 |
| mgagne | lifeless: is there a way to see all topics including ones that were rejected? | 19:53 |
| lifeless | fungi: cool, thanks | 19:54 |
| lifeless | mgagne: I don't know | 19:54 |
| fungi | mgagne: http://summit.openstack.org/ | 19:54 |
| fungi | is that what you're looking for? | 19:54 |
| mgagne | fungi: thanks! | 19:55 |
| fungi | mgagne: not sure if you were looking for design sessions or conference presentations | 19:55 |
| zul | ttx: ping still no cinder rc1? | 19:55 |
| mgagne | fungi: conferences ^^' | 19:56 |
| openstackgerrit | Pierre Rognant proposed a change to openstack-infra/jenkins-job-builder: Flow project support https://review.openstack.org/49608 | 19:57 |
| sdague | jeblair: https://review.openstack.org/#/c/49021/ when you get a chance, that will let us have the functioning sudo based debug when things go funny | 19:57 |
| fungi | mgagne: as a track chair i can see them, but they're behind a login so i'm not sure what i'm allowed to disclose (if anything) | 19:57 |
| *** dprince has quit IRC | 19:58 | |
| *** AlexF has quit IRC | 19:59 | |
| fungi | mgagne: if you're really interested in a copy of all the abstracts i'm happy to ask the right people and find out what can be released | 20:00 |
| mgagne | fungi: I'm logged in and don't see the one I'm looking for. =) | 20:00 |
| fungi | mgagne: oh, you were a track chair too? go to "browse presentations" in the chairs interface and then click on the track link and change the drop-down to "all categories" i think | 20:01 |
| fungi | i don't see you on the track chairs directory though, so it might be restricted | 20:02 |
| *** julim has quit IRC | 20:02 | |
| *** rnirmal has quit IRC | 20:02 | |
| fungi | note i'm talking about the https://www.openstack.org/track-chairs/ interface | 20:02 |
| mgagne | fungi: ok, I'll look | 20:03 |
| openstackgerrit | Pierre Rognant proposed a change to openstack-infra/jenkins-job-builder: Flow project support https://review.openstack.org/49608 | 20:03 |
| *** adalbas has quit IRC | 20:03 | |
| mgagne | fungi: I don't have access once logged. | 20:03 |
| fungi | if you weren't chairing one of the summit tracks then i don't think it will let you see the list | 20:03 |
| fungi | like i said, i can ask around and find out if any of what's in there is secret or can be disclosed | 20:04 |
| fungi | if you need it | 20:04 |
| mgagne | fungi: more curiosity about one presentation. every links I found was linking to the vote system which is now over. | 20:05 |
| jeblair | back | 20:05 |
| *** rnirmal has joined #openstack-infra | 20:05 | |
| sdague | mgagne: these are design summit sessions, so no voting | 20:05 |
| sdague | the PTLs just accept what they want in their track | 20:05 |
| fungi | sdague: i think he's looking for a list of the conference presentation abstracts instead | 20:06 |
| mgagne | fungi: yes | 20:06 |
| sdague | oh, yeh that's not http://summit.openstack.org/ :) | 20:06 |
| *** CaptTofu has quit IRC | 20:07 | |
| fungi | agreed | 20:07 |
| *** markmcclain has joined #openstack-infra | 20:07 | |
| *** resker has joined #openstack-infra | 20:07 | |
| *** CaptTofu has joined #openstack-infra | 20:07 | |
| *** SergeyLukjanov has quit IRC | 20:09 | |
| *** pcm_ has quit IRC | 20:09 | |
| *** esker has quit IRC | 20:11 | |
| *** resker has quit IRC | 20:11 | |
| jeblair | fungi, mordred: are either of you interested in reviewing https://review.openstack.org/#/c/48517/ ? | 20:13 |
| jeblair | sorry | 20:13 |
| jeblair | clarkb, mordred: ^ | 20:13 |
| jeblair | fungi reviewed it | 20:13 |
| *** gyee has joined #openstack-infra | 20:15 | |
| zaro | clarkb: just wondering, would this be a good thing? https://review.openstack.org/#/c/49448/ | 20:18 |
| *** dkliban has quit IRC | 20:27 | |
| *** marun has quit IRC | 20:29 | |
| *** sarob has joined #openstack-infra | 20:30 | |
| *** jcoufal has joined #openstack-infra | 20:34 | |
| *** CaptTofu has quit IRC | 20:37 | |
| *** CaptTofu has joined #openstack-infra | 20:37 | |
| pleia2 | browsing through reviews, this one got stuck because the dependency is abandoned https://review.openstack.org/#/c/45348/ | 20:39 |
| pleia2 | still want, or should it be abandoned as well? | 20:39 |
| pleia2 | (the other was abandoned due to timeout, rocketships!) | 20:40 |
| pleia2 | so it looks like we still want these | 20:40 |
| clarkb | back from lunch /me reads scrollback | 20:40 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: TripleO merged with Tuskar. https://review.openstack.org/49606 | 20:41 |
| *** tvb has quit IRC | 20:41 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Cleanup tests https://review.openstack.org/48764 | 20:42 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make test_required_files.py real unit tests https://review.openstack.org/49265 | 20:42 |
| clarkb | jeblair: yes very interested. looking now | 20:42 |
| clarkb | zaro: yes, I think that is a good thing. We probably need to think through the security ramifications of clouddbs but in general being configurable is good | 20:43 |
| clarkb | jeblair: did you see the clouddb discussion? any input on that? | 20:43 |
| clarkb | pleia2: I think we need to think through that stuff a bit more | 20:43 |
| zaro | clarkb: cool. just wanted to know if i should rebase my future changes on that. thnx. | 20:43 |
| clarkb | pleia2: eg how to do it without potentially breaking stuff (moving to clouddbs and splitting DB config from service config is a good start) | 20:44 |
| pleia2 | clarkb: ah, ok | 20:44 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Reorganize tests into unit and functional tests https://review.openstack.org/49391 | 20:46 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: run tempest tests under tempest user https://review.openstack.org/49021 | 20:46 |
| mtreinish | mordred, clarkb, jeblair: if you've got a sec can you take a look at: https://review.openstack.org/#/c/49504/ so we can start failing neutron runs again. :) | 20:49 |
| jeblair | clarkb: i'm okay but not thrilled with the arrangement. perhaps we should ask hub_cap nicely for ssl? :) | 20:49 |
| clarkb | it looks like my jeepyb delay for gerrit groups to be created change went into manage_projects but we haven't released a new version of gerritlib with the feature to support that | 20:49 |
| clarkb | are there plans to release a new gerritlib that I am not aware of? | 20:50 |
| jeblair | clarkb: i can cut one real quick | 20:51 |
| clarkb | jeblair: thanks | 20:51 |
| jeblair | clarkb: pushed | 20:52 |
| hub_cap | jeblair: patches welcome ;) As much as id love to do it, itd kill my existing priorities for openstack compatibility... but if u want to convince hp to make it a priority, then it will be so :) | 20:52 |
| jeblair | okay, asking nicely clearly didn't work. lesson learned. :) | 20:53 |
| *** vipul-away is now known as vipul | 20:54 | |
| hub_cap | jeblair: sry :( | 20:54 |
| jeblair | clarkb: can you aprv https://review.openstack.org/#/c/49312/2 ? | 20:55 |
| clarkb | jeblair: ya reviewing | 20:55 |
| clarkb | jeblair: done | 20:56 |
| jeblair | mtreinish: looking; i want to make sure the most recent check ran with the variable set | 20:56 |
| clarkb | jeblair: gerritlib release may not have succeeded. I don't see it on pypi but jobs seem to have finished | 20:56 |
| *** jpeeler has quit IRC | 20:56 | |
| clarkb | twine barfed | 20:56 |
| jeblair | clarkb: you mind looking into that? | 20:57 |
| clarkb | https://jenkins.openstack.org/job/gerritlib-pypi-upload/4/console ya looking now | 20:57 |
| *** rnirmal has quit IRC | 20:57 | |
| openstackgerrit | A change was merged to openstack-infra/config: Handle null in this function https://review.openstack.org/49312 | 20:58 |
| clarkb | dstufft: ^ I think twine is not python2.X compatible | 20:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix zuul status graph with failed change at head https://review.openstack.org/49590 | 20:58 |
| clarkb | dstufft: am I reading that correctly? | 20:58 |
| dstufft | clarkb: I accidently made it not compat on 2.6, I'm gonna be fixing that later tonight | 20:58 |
| dstufft | it should work fine on 2.7 though | 20:58 |
| mtreinish | jeblair: yeah the issue is the variable is only set on the neutron queue | 20:58 |
| clarkb | dstufft: it is importing urllib.parse which is python3 | 20:59 |
| dstufft | hmm, I thought that was guarded by an importerror | 20:59 |
| dstufft | clarkb: sorry i'm sick so i'm a little out of it | 20:59 |
| dstufft | clarkb: one minute | 20:59 |
| jeblair | mtreinish: eek, asymmetric.. | 20:59 |
| clarkb | dstufft: no problem. I can poke at it myself, but figured I would ask to avoid duplicating work | 20:59 |
| dstufft | clarkb: oh I see what happened | 21:00 |
| dstufft | clarkb: https://github.com/dstufft/twine/blob/master/twine/commands/upload.py#L25-L28 | 21:00 |
| dstufft | should be urlparse not urllib | 21:00 |
| dstufft | clarkb: gimme a minute i'll fix this | 21:00 |
| clarkb | dstufft: ty | 21:00 |
| dstufft | clarkb: this is what I get for being "well this is short and I want it now so I won't write tests for it right now" | 21:01 |
| dstufft | guess I'll be writing tests tonight or tomorrow | 21:01 |
| mtreinish | jeblair: yeah it was just to unbreak the gate on the neutron runs. Enabling tenant isolation increased the number of requests a lot for a tempest run which triggered a race in neutron. | 21:01 |
| mtreinish | look at the elastic recheck graph for 1224001 | 21:02 |
| *** alcabrera has quit IRC | 21:02 | |
| mtreinish | but it's a real neutron bug so we're gating on neutron with it turned on | 21:02 |
| mtreinish | just not for the other projects | 21:02 |
| mtreinish | (tenant isolation with neutron that is, tenant isolation is the default for tempest on every other run) | 21:03 |
| jeblair | mtreinish: right, but we're currently in a place where changes to d-g, devstack, and tempest could break gating for neutron | 21:03 |
| *** chris613 has joined #openstack-infra | 21:03 | |
| mtreinish | jeblair: that's sort of the intent | 21:03 |
| *** johnthetubaguy has joined #openstack-infra | 21:04 | |
| jeblair | mtreinish: ok, if i approve this and there's a structural problem with it, will you be around to fix it? :) | 21:04 |
| jeblair | mtreinish: (i don't like making untested changes to devstack-gate, which is what this is if the job that uses it isn't in the d-g job set) | 21:04 |
| mtreinish | jeblair: I should be | 21:04 |
| mtreinish | where else would I be :) | 21:05 |
| jeblair | mtreinish: that's the spirit! | 21:05 |
| jeblair | mtreinish: done | 21:05 |
| mtreinish | jeblair: ok thanks | 21:05 |
| *** rfolco has joined #openstack-infra | 21:05 | |
| clarkb | jeblair: you have implemented a priority based load balancer for nodepool node boots | 21:07 |
| clarkb | jeblair: similar to a heap based priority queue? I am reading docstrings | 21:07 |
| *** dims has quit IRC | 21:08 | |
| *** dims has joined #openstack-infra | 21:09 | |
| *** matty_dubs is now known as matty_dubs|gone | 21:09 | |
| *** rfolco has quit IRC | 21:11 | |
| dstufft | clarkb: try twine >= 1.2.2 | 21:11 |
| dstufft | clarkb: I just uploaded twine using python 2.7 using twine 1.2.2 so it should work | 21:11 |
| clarkb | dstufft: awesome will do (I have to wait for puppet to upgrade twine then I will retrigger the job) | 21:12 |
| *** zojoncj has joined #openstack-infra | 21:12 | |
| dstufft | clarkb: np, I'll get some tests written for it to keep it from happening again :) | 21:12 |
| *** mrodden1 is now known as mrodden | 21:13 | |
| *** marun has joined #openstack-infra | 21:14 | |
| *** boris-42 has quit IRC | 21:15 | |
| *** vipul is now known as vipul-away | 21:17 | |
| clarkb | puppet doesn't ensure twine is latest. I am going to upgrade by hand on pypi.slave. Not sure if want auto updates for that package (I assume yes?) | 21:20 |
| clarkb | rerunning gerritlib pypi push now | 21:23 |
| fungi | seems safe enough | 21:23 |
| fungi | given assurances above ;) | 21:23 |
| clarkb | oh hmm trigger-job.py doesn't have an option for log basepath. Will this copy the log into a funny place? | 21:25 |
| clarkb | I am going to use a local copy that supplies the new arg and will propose a zuul chnage with that fix | 21:26 |
| dstufft | clarkb: I don't plan on breaking compatability of twine (where compatability is the cli) | 21:26 |
| clarkb | dstufft: cool. We should just ensure latest then | 21:27 |
| *** dizquierdo has joined #openstack-infra | 21:29 | |
| *** thomasm has quit IRC | 21:35 | |
| clarkb | dstufft: https://jenkins.openstack.org/job/gerritlib-pypi-upload/5/console that familiar to you at all? | 21:35 |
| dstufft | clarkb: hrm, can you past your ~/.pypirc with the passwords stripped out? | 21:36 |
| dstufft | paste | 21:36 |
| clarkb | yeah | 21:36 |
| clarkb | dstufft: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/templates/pypirc.erb | 21:37 |
| dstufft | clarkb: moment | 21:37 |
| *** chris613 has quit IRC | 21:37 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Ensure devstack has proper tenant isolation setting https://review.openstack.org/49504 | 21:37 |
| *** vipul-away is now known as vipul | 21:38 | |
| dstufft | clarkb: you're missing the [distutils] section | 21:38 |
| dstufft | clarkb: https://gist.github.com/dstufft/6817544 | 21:39 |
| jog0 | clarkb: it is stuck | 21:39 |
| jog0 | (e-r bot) | 21:39 |
| clarkb | dstufft: thanks | 21:39 |
| dstufft | clarkb: I'll probably give twine it's own format at some point, but it'll fall back to ~/.pypirc | 21:40 |
| dstufft | or extend pypirc | 21:40 |
| dstufft | something like that | 21:40 |
| dstufft | to support more options | 21:40 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add missing distutils section to pypirc. https://review.openstack.org/49637 | 21:41 |
| clarkb | dstufft: ^ that should fix it | 21:41 |
| clarkb | fungi: jeblair mordred can get quick reviews on that? | 21:41 |
| clarkb | jog0: thanks. I will wait for the change to add logging before restarting it | 21:41 |
| * jog0 dusts off his patch | 21:42 | |
| dstufft | clarkb: I +1'd it not that it'll help you at all :) | 21:42 |
| *** jcoufal has quit IRC | 21:43 | |
| *** mriedem has quit IRC | 21:45 | |
| pabelanger | well, that was somewhat easy. Managed to get nodepool running and launching instances | 21:45 |
| *** oubiwann_ is now known as oubiwann | 21:46 | |
| *** fbo is now known as fbo_away | 21:49 | |
| *** boris-42 has joined #openstack-infra | 21:49 | |
| *** dizquierdo has quit IRC | 21:50 | |
| *** flaper87|afk is now known as flaper87 | 21:51 | |
| *** davidhadas has quit IRC | 21:51 | |
| clarkb | fungi: thanks for the review. I went ahead and approved it | 21:52 |
| *** rnirmal has joined #openstack-infra | 21:52 | |
| fungi | i'm taking your word for it on that (and the author's obviously). couldn't find it mentioned in the readme and didn't see a manpage | 21:52 |
| clarkb | fungi: http://docs.python.org/2/distutils/packageindex.html#the-pypirc-file | 21:53 |
| fungi | oh, distutils doc mentions it, not twine. that's why | 21:53 |
| dstufft | twine just uses the distutils ~/.pypirc | 21:53 |
| clarkb | ya twine is just being nice and compatible | 21:53 |
| openstackgerrit | A change was merged to openstack-infra/config: Add missing distutils section to pypirc. https://review.openstack.org/49637 | 21:53 |
| dstufft | infact it reaches into distutils to use the same routine | 21:53 |
| dstufft | to ensure compatability :D | 21:54 |
| fungi | aha! neat solution. no code duplication | 21:54 |
| *** dkranz has quit IRC | 21:56 | |
| openstackgerrit | Paul Belanger proposed a change to openstack-infra/nodepool: Don't use max_overflow with sqlite https://review.openstack.org/49639 | 21:56 |
| pabelanger | Trying to decided if nodepool could be used to also setup the master side of the pool. Eg: launch masters, then bring up slaves | 21:57 |
| clarkb | oh neat | 21:58 |
| *** johnthetubaguy has quit IRC | 21:58 | |
| fungi | pabelanger: that could be awesome | 21:58 |
| *** boris-42 has quit IRC | 21:59 | |
| fungi | but setting up a jenkins, even puppeted, still takes a bit of extra monkeying, so it could turn out to be more effort than it's worth | 21:59 |
| fungi | at least insofar as the global config is concerned | 22:00 |
| pabelanger | So, my though process for nodepool is more generic, it's based more on developers. Imagine they each have a tenant in openstack (the cloud). Of course adding logic to nodepool, you specific your master( puppet), then setup each slave(node) with their specific profile. Developer launch nodepool, blam, their tenant reproduces the production environment. | 22:01 |
| pabelanger | Even extending it, add hooks into messagebus, spinning up each specific node that needs more resources | 22:01 |
| clarkb | dstufft: https://jenkins.openstack.org/job/gerritlib-pypi-upload/6/console still doing it (I checked on the slave the new distuitls section is in place) | 22:03 |
| * clarkb looks closer for tyops and the like | 22:03 | |
| *** sarob has quit IRC | 22:03 | |
| dstufft | clarkb: can you ssh into that server and do python -c "from twine.utils import get_distutils_config; print(get_distutils_config('pypi'))" | 22:04 |
| dstufft | that'll print out the password so don't paste the results directly | 22:04 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 22:05 |
| jog0 | clarkb: ^ | 22:05 |
| dstufft | clarkb: sorry this isn't working as well as it could | 22:05 |
| mtreinish | jeblair: it looks like the d-g change works: https://jenkins01.openstack.org/job/check-tempest-devstack-vm-neutron-isolated/53/consoleFull | 22:05 |
| clarkb | dstufft: no problem. The end result should be much better :) | 22:05 |
| clarkb | jog0: thanks | 22:05 |
| *** changbl has joined #openstack-infra | 22:05 | |
| clarkb | dstufft: that spits out a dict | 22:07 |
| dstufft | hrm | 22:07 |
| dstufft | does it have a repository key? | 22:07 |
| clarkb | dstufft: it does | 22:08 |
| clarkb | dstufft: I wonder if the repository arg at https://github.com/dstufft/twine/blob/master/twine/commands/upload.py#L67 isn't correct | 22:08 |
| clarkb | or maybe it is looking in $PWD and not ~/.pypirc | 22:08 |
| * clarkb chdirs and tries again | 22:08 | |
| clarkb | no works fine from the job workspace | 22:09 |
| dstufft | clarkb: what happens if you pass None instead of "pypi" | 22:09 |
| clarkb | dstufft: empty dict | 22:10 |
| dstufft | ah ha | 22:11 |
| dstufft | now why is it doing that.. | 22:11 |
| dstufft | clarkb: what version of python? | 22:11 |
| clarkb | 2.7.3 | 22:11 |
| dstufft | hm | 22:11 |
| dstufft | strange | 22:11 |
| dstufft | on 2.7.5 None gives me a full dict | 22:12 |
| dstufft | clarkb: you can fix this for right now by passing a -r pypi | 22:12 |
| dstufft | to the twine command | 22:12 |
| clarkb | ok I will do that. thanks | 22:12 |
| dstufft | I'll dig into it more, I might end up parsing the config file on my own | 22:12 |
| ttx | zul: tomorrow morning I think | 22:13 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Specific twine repository when publishing packages https://review.openstack.org/49643 | 22:14 |
| clarkb | dstufft: fungi ^ | 22:14 |
| *** datsun180b has quit IRC | 22:15 | |
| dstufft | clarkb: LGTM | 22:15 |
| dstufft | I think i'm going to remove the distutils dependency, stupid piece of crap | 22:15 |
| *** mrda has joined #openstack-infra | 22:16 | |
| clarkb | I am happy to use -r if that is the sanest work around (duplicating code is silly but so are silly bugs) | 22:17 |
| dstufft | well it's not a straight forward use of distutils | 22:17 |
| dstufft | I have to trick it into letting me do it | 22:17 |
| dstufft | https://github.com/dstufft/twine/blob/master/twine/utils.py#L28-L48 | 22:18 |
| dstufft | clarkb: I'm not happy with an api that requires -r, I'm annoyed it requires that for y'all | 22:19 |
| clarkb | wow that is full of trickery | 22:20 |
| *** thedodd has quit IRC | 22:21 | |
| dstufft | Originally I figured i'd hit less edge cases by reusing that code | 22:21 |
| dstufft | but I guess not! | 22:21 |
| *** oubiwann has quit IRC | 22:26 | |
| *** sarob has joined #openstack-infra | 22:27 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add even more debug logs. https://review.openstack.org/49458 | 22:27 |
| *** ryanpetrello has quit IRC | 22:31 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Update trigger-job.py to pass LOG_PATH https://review.openstack.org/49646 | 22:31 |
| clarkb | and that is what I am using to trigger the jobs | 22:32 |
| *** ryanpetrello has joined #openstack-infra | 22:33 | |
| *** prad has quit IRC | 22:33 | |
| *** weshay has quit IRC | 22:36 | |
| *** flaper87 is now known as flaper87|afk | 22:39 | |
| *** jhesketh has joined #openstack-infra | 22:40 | |
| *** changbl has quit IRC | 22:47 | |
| *** changbl has joined #openstack-infra | 22:48 | |
| openstackgerrit | A change was merged to openstack-infra/config: Specific twine repository when publishing packages https://review.openstack.org/49643 | 22:49 |
| *** rnirmal has quit IRC | 22:49 | |
| *** sarob has quit IRC | 22:50 | |
| *** ryanpetrello has quit IRC | 22:53 | |
| openstackgerrit | David Caro proposed a change to openstack-infra/reviewstats: Added some options to openapproved https://review.openstack.org/49522 | 22:58 |
| *** sarob has joined #openstack-infra | 22:58 | |
| *** rcleere has quit IRC | 22:58 | |
| *** rfolco has joined #openstack-infra | 23:02 | |
| clarkb | https://pypi.python.org/pypi/gerritlib/ new gerritlib released | 23:06 |
| *** dims has quit IRC | 23:06 | |
| clarkb | twine made it easy once the problems were sorted | 23:08 |
| jeblair | clarkb: yay! | 23:09 |
| clarkb | jeblair: now back to reviewing the nodepool change | 23:09 |
| *** dkehn has quit IRC | 23:14 | |
| *** sarob_ has joined #openstack-infra | 23:17 | |
| *** jhesketh has quit IRC | 23:17 | |
| *** sarob has quit IRC | 23:20 | |
| *** dims has joined #openstack-infra | 23:23 | |
| lifeless | clarkb: if you're poking at gerritlib, I have patches... | 23:24 |
| lifeless | clarkb: https://review.openstack.org/#/c/49326/ etc | 23:24 |
| mordred | lifeless: you're a patch | 23:25 |
| lifeless | mordred: patch thsi | 23:26 |
| clarkb | -patch thsi\n+patchthis | 23:27 |
| clarkb | wow fail. | 23:27 |
| clarkb | lifeless: I have starred those changes and will attempt to get to them after this nodepool change | 23:28 |
| lifeless | clarkb: https://review.openstack.org/#/dashboard/4190 may interest (or horrify) you | 23:28 |
| lifeless | clarkb: my WIP count is ridiculously high at the moment | 23:29 |
| mordred | lifeless: before I nitpick - is there any specific reason for class GerritWatcher: rather than class GerritWatcher(object): ? | 23:30 |
| lifeless | yes | 23:31 |
| *** jhesketh has joined #openstack-infra | 23:31 | |
| lifeless | (object) is consideref bad form in Python 3.x | 23:31 |
| lifeless | considered | 23:32 |
| jeblair | isn't omitting it considered bad form in python2? | 23:32 |
| lifeless | the canonical thing to do is set __metaclass__=type | 23:32 |
| lifeless | jeblair: it matters in Python 2 if you're doing anything requiring new style classes, which this isn't. | 23:33 |
| mordred | great. I love it that python3 changed that | 23:33 |
| mordred | because I thought what jeblair said | 23:34 |
| mordred | "it matters in Python 2 if you're doing anything requiring new style classes" is a terrible benchmark | 23:34 |
| mordred | since new style classes have been just classes for like 10 years | 23:34 |
| mordred | and I can't remember what's new style and what's not anymore | 23:34 |
| mordred | sigh | 23:34 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Update trigger-job.py to pass LOG_PATH https://review.openstack.org/49646 | 23:38 |
| clarkb | jeblair: the nodepool change seems to be more complicated than it could be. I keep wanting to think of it as a proper datastructure with well known methods and instead need to translate back and forth betwene the various *Request* | 23:41 |
| *** jd__ has quit IRC | 23:42 | |
| jeblair | clarkb: i agree it's complicated. i tried two ways of making it less complicated that didn't work before this one. | 23:42 |
| clarkb | could we have a single priority queue that was updated on each node being created? | 23:42 |
| *** yaguang has joined #openstack-infra | 23:42 | |
| clarkb | then simply grab the item at the head of the queue when booting a node | 23:42 |
| dstufft | clarkb: did that work then? | 23:43 |
| clarkb | dstufft: yup, thanks | 23:43 |
| dstufft | awesome | 23:43 |
| dstufft | sorry for the troubles, glad it works now :D | 23:43 |
| *** jd__ has joined #openstack-infra | 23:44 | |
| jeblair | clarkb: can you explain your idea a bit further? | 23:44 |
| clarkb | jeblair: I think that is essentially what we have ended up with, but the state of the queue is managed in many places | 23:44 |
| *** sarob has joined #openstack-infra | 23:45 | |
| clarkb | jeblair: a single http://docs.python.org/2/library/queue.html#Queue.PriorityQueue that is recreated each time external state changes eg building a node or a test starting. To spin up a node you recalculate priority then grab the head of that queue and grant the request, continue until all available resources are used | 23:48 |
| *** sarob_ has quit IRC | 23:49 | |
| jeblair | clarkb: so calculating the priority is likely to be just as complex as the current logic | 23:49 |
| clarkb | jeblair: from my understanding of the current code this is basically what you have but the operations aren't explicit from a data structure perspective. Instead there are several cooperating objects that together form that queue | 23:49 |
| clarkb | jeblair: right, but we can do it in one place I think | 23:49 |
| jeblair | clarkb: it's managed in several places because it's solving several problems -- which providers to use for which images, how much of each provider to use for each image, and where to attach each node | 23:50 |
| jeblair | clarkb: the current system ends up with a count of nodes for each provider-image-target, and loops over that to spin them up. it sounds like the major change would be that it would remove items from a queue instead. but all the logic to figure out what items need to be supplied/attached would still need to exist, and likely similarly complicated. | 23:53 |
| clarkb | jeblair: ok. I will keep thinking about how to reduce complexity as I finish up the review | 23:53 |
| jeblair | clarkb: note that it accomplishes those tasks in 3 phases, so it's taking a somewhat holistic view -- i think that's part of the complexity. | 23:53 |
| *** dkehn has joined #openstack-infra | 23:54 | |
| clarkb | jeblair: I think what I want is the ability to rely on preexisting semantic operations, but they may not help too much here (difficulty for me is keeping track of what each of the discrete thing is supposed to do) | 23:54 |
| *** che-arne has joined #openstack-infra | 23:54 | |
| *** dkehn has quit IRC | 23:54 | |
| zaro | clarkb: which puppet module creates the cgit machines (git01.o.o, git02.o.o, etc.)? | 23:55 |
| clarkb | zaro: cgit | 23:55 |
| *** dkehn has joined #openstack-infra | 23:56 | |
| *** jhesketh has quit IRC | 23:56 | |
| clarkb | jeblair: the docstring at the top of the allocation file is great, but mapping the pieces onto objects makes my brain tired :) | 23:57 |
| *** jhesketh has joined #openstack-infra | 23:57 | |
| jeblair | clarkb: specific cases that drive this complexity are things like: image A can be supplied only by provider 1, image B can be supplied by providers 1, 2, and 3; requests for image A and B each exceed the available capacity of all providers | 23:57 |
| *** jcoufal has joined #openstack-infra | 23:58 | |
| clarkb | jeblair: you are ensuring that B doesn't starve A | 23:58 |
| jeblair | clarkb: yep | 23:58 |
| jeblair | clarkb: and then on top of that, there's taking nodes of image A from provider 1 and mapping them to target X in the right proportions (because the target is specified in terms of image _and_ provider) | 23:59 |
| jeblair | clarkb: i admit, i kinda drew a blank on succinct names. so the class names got a bit long. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!