| IAX | hi! I'm having an issue while using the logstash::watchdog class, in my Ubuntu 12.04 box puppet returns an error that the package 'jq' was not found, doing some research the package is availabe in backports repo which is not enabled by default.... have anyone had the same issue? | 00:00 |
|---|---|---|
| *** sdake has joined #openstack-infra | 00:00 | |
| IAX | i think the easiest solution is to uncomment the backport repo line in apt, but i'm wondering if it's there a more elegant solution. | 00:00 |
| mordred | jeblair: I am now testing the script along with the puppet changes | 00:01 |
| mordred | so if this time it works, I'll set dns and keep the server | 00:01 |
| *** iax7 has joined #openstack-infra | 00:04 | |
| clarkb | IAX: hrm our images have it they may already allow backports | 00:05 |
| clarkb | IAX: though I don't think you need to use the watchdog anymore if using our current setup | 00:05 |
| *** dkehnx1 has quit IRC | 00:05 | |
| clarkb | IAX: which may be a better fix | 00:05 |
| *** sballe has quit IRC | 00:06 | |
| mordred | clarkb: who is playing in the footballs tonight/ | 00:06 |
| clarkb | seattle and green bay | 00:06 |
| adam_g | heya! we've got a rejig to the current ironic job layout that could use some eyes if anyone has a chance. it eliminates some redundancy and paves the way for additional agent CI that needs to start running soon https://review.openstack.org/#/c/112134/ | 00:06 |
| clarkb | opening game of season should be good | 00:06 |
| mordred | clarkb: ah! why yes | 00:06 |
| *** marcoemorais has quit IRC | 00:07 | |
| * jroll boooooooo seattle | 00:07 | |
| *** marcoemorais has joined #openstack-infra | 00:07 | |
| jroll | :P | 00:07 |
| jogo | I have another gate bug that was just fixed: https://review.openstack.org/#/c/116354/ | 00:07 |
| *** mfainberg_phone has joined #openstack-infra | 00:08 | |
| *** dkehnx1 has joined #openstack-infra | 00:08 | |
| jogo | 65 fails in 24 hours | 00:08 |
| jogo | in all queues | 00:08 |
| jogo | fungi jeblair: ^ | 00:08 |
| IAX | clarkb, i'm using openstack_project::logstash_worker which calls that class, it makes sure that the cron job is absent but there is still declared | 00:08 |
| IAX | that's where it fails | 00:09 |
| clarkb | IAX: oh in that case lets more properly remove it | 00:09 |
| clarkb | jogo: you want to do what with it? | 00:09 |
| jogo | clarkb: get it promoted if there is an opening | 00:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add entry for new PyPI mirrors https://review.openstack.org/119129 | 00:09 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split pypi_mirror into its own class https://review.openstack.org/119124 | 00:09 |
| clarkb | jeblair: fungi mordred I just realized hpcloud security groups need opening up doing that now | 00:10 |
| mordred | clarkb: ossum | 00:10 |
| *** jergerber has quit IRC | 00:10 | |
| notmyname | mordred: jeblair: with gertty, do I have to enter a password in the config file? | 00:10 |
| IAX | great! then is not required! thanks! | 00:11 |
| *** mfink has joined #openstack-infra | 00:11 | |
| clarkb | notmyname: you do, it is your gerrit http key thing though not your lp password | 00:11 |
| *** mfainberg_phone has quit IRC | 00:11 | |
| mordred | notmyname: you need the password from here: https://review.openstack.org/#/settings/http-password | 00:11 |
| mordred | notmyname: if you do not have one, click "generate password" | 00:12 |
| jogo | clarkb: if possible | 00:13 |
| clarkb | it would be nice if there was a way to just say allow all traffic but security groups blacklist by default right? | 00:13 |
| clarkb | so removing the entries blocks everything? | 00:14 |
| clarkb | jogo: do we want to see it pass check testing before we put it at the head of the queue? | 00:14 |
| jogo | clarkb: probably a good idea | 00:14 |
| *** ianw has quit IRC | 00:15 | |
| *** camunoz_ has joined #openstack-infra | 00:15 | |
| *** amcrn has quit IRC | 00:16 | |
| *** ianw_ has joined #openstack-infra | 00:16 | |
| asselin__ | anyone see this error before? tips to isolate/debug would be welcomed. http://15.126.198.151/17/119017/1/silent/dsvm-tempest-hp-3par-fc/3dd141c/console.html | 00:16 |
| clarkb | mordred: I think security group is wide open now | 00:17 |
| *** gokrokve has quit IRC | 00:17 | |
| clarkb | asselin__: looks like gcc failed to build the python lxml bindings | 00:18 |
| *** gokrokve has joined #openstack-infra | 00:18 | |
| mordred | clarkb: awesome | 00:18 |
| clarkb | asselin__: you might double check your versions of those things and that they are all compatible | 00:18 |
| *** ianw_ has quit IRC | 00:18 | |
| *** ianw_ has joined #openstack-infra | 00:19 | |
| *** ianw_ is now known as ianw | 00:20 | |
| *** marcoemorais has quit IRC | 00:21 | |
| clarkb | mordred: you put the site.pp and pypi puppet changes on the very top of the launch changes which includes the thing yo udon't want to fix | 00:21 |
| clarkb | mordred: maybe you can branch at the commit before that? | 00:21 |
| *** marcoemorais has joined #openstack-infra | 00:21 | |
| clarkb | looking good otherwise though | 00:21 |
| mordred | clarkb: wait, really? | 00:22 |
| mordred | dammit | 00:22 |
| mordred | clarkb: thanks | 00:22 |
| *** gokrokve has quit IRC | 00:22 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add entry for new PyPI mirrors https://review.openstack.org/119129 | 00:22 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split pypi_mirror into its own class https://review.openstack.org/119124 | 00:22 |
| *** zhiwei has quit IRC | 00:23 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures https://review.openstack.org/118944 | 00:23 |
| jeblair | notmyname: oh, but if you were asking if you could avoid actually saving it to disk, i believe it does support that | 00:24 |
| jeblair | notmyname: i think i completely forgot to document that, but harlowja added that a while back | 00:24 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures https://review.openstack.org/118944 | 00:24 |
| harlowja | what i do, lol | 00:24 |
| jeblair | notmyname: so i think if you omit it from the config file, it will prompt you on startup | 00:25 |
| fungi | however the passwords generated by gerrit are not entirely memorable, so unless you have an excellent memory, some sort of password manager is probably in order | 00:25 |
| anteaya | asselin__: the problems seems to start here: 00:04:58 Found existing installation: lxml 3.3.3 | 00:26 |
| anteaya | 00:04:58 Not uninstalling lxml at /usr/lib/python2.7/dist-packages, outside environment /opt/stack/new/tempest/.tox/all | 00:26 |
| mika | hm, i'm getting "sqlalchemy.exc.OperationalError: (OperationalError) no such table: revision u'ALTER TABLE revision ADD COLUMN fetch_auth BOOLEAN' ()" when testing gertty, any ideas what's going wrong here? :) | 00:26 |
| mordred | jeblair: now we're probably about to find out what the load is on gerrit of 1000 people all syncing projecs using gertty at the same time | 00:26 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Don't retry forever when a provider is stopped https://review.openstack.org/118710 | 00:26 |
| notmyname | jeblair: ok, thanks | 00:27 |
| asselin__ | clarkb, anteaya thanks....looking into that | 00:27 |
| jeblair | mordred: yeah, also, we might need to have a conversation about gertty's default polling interval at some point too :) | 00:27 |
| mordred | mika: I'm looking now, but jeblair might be more useful on that | 00:27 |
| mika | mordred: thanks! i'm using a *custom* gerrit installation, if that matters | 00:28 |
| mordred | mika: well, I would't expect that to affect the alembic migrations | 00:28 |
| lifeless | jeblair: gertty shouldn't poll. gerrit should push :). | 00:28 |
| * mordred waits for lifeless to write that patch to gerrit ... | 00:29 | |
| *** ianw has quit IRC | 00:29 | |
| jeblair | mika: could you rm ~/.gertty.db; then run gertty with the '-d' flag and pastebin the output? | 00:29 |
| *** camunoz__ has joined #openstack-infra | 00:29 | |
| jeblair | sorry not output | 00:29 |
| jeblair | mika: contents of ~/.gertty.log | 00:29 |
| lifeless | mordred: stranger things have happened | 00:29 |
| mika | sure, will do | 00:29 |
| mordred | lifeless: many many stranger things :) | 00:29 |
| jeblair | lifeless: it _could_ use stream-events, but i imagine we'd hit some connection limits at some point | 00:29 |
| *** prad has joined #openstack-infra | 00:30 | |
| *** SumitNaiksatam has quit IRC | 00:30 | |
| lifeless | jeblair: we might have to up the fd count, but there's little reason a modern machine with a modern java stack can't handle 10K concurrent sockets | 00:30 |
| *** camunoz has joined #openstack-infra | 00:30 | |
| lifeless | jeblair: which we're well below; and the work per event should be very low | 00:30 |
| lifeless | should. <- HAHAHA. | 00:30 |
| clarkb | jenkins can do about 200 | 00:31 |
| clarkb | but I guess those sockets are actually pumping data | 00:31 |
| mika | jeblair: ha, managed to spot it already - if git-root is pointing to a directory which already includes git repos i'm getting this problem, i'm getting into the UI if i point it to an empty dir (after removing ~/.gertty.db and rerunning it again) | 00:31 |
| lifeless | clarkb: jenkins has a tonne of problems | 00:31 |
| jeblair | lifeless: yeah, i say that mostly because we've seen a couple of problems with gerrit and lots of stream-events with poorly behaved clients on the other end. this is all vague and handwavey, i don't know what the actual problems may have been. | 00:31 |
| mordred | it would be interesting to benchmark and understand better | 00:32 |
| mordred | it would also be stellar if we could scale gerrit out ... | 00:32 |
| *** camunoz_ has quit IRC | 00:32 | |
| mordred | like, add another stream-events emitter machine, for instance | 00:32 |
| jeblair | mika: hrm, that is weird. i'm pretty sure most of my tests of first-runs have been with existing git repos | 00:32 |
| *** oanufriev has quit IRC | 00:33 | |
| jeblair | mika: and the specific error (that it could not alter a table that should have already existed) seems strange to be related to that | 00:33 |
| *** camunoz__ has quit IRC | 00:33 | |
| jeblair | mika: any chance you had an earlier aborted run or something that left the db in a weird state? | 00:33 |
| mika | jeblair: *could* be, i can't reproduce it right now - will let you know if i notice anything further, thanks for your help | 00:34 |
| *** dane_leblanc has quit IRC | 00:34 | |
| jeblair | mika: thanks for yours! | 00:34 |
| mika | jeblair: http://paste.grml.org/hidden/94ca2f28/ - hm, is it trying to gather a list of projects *before* authenticating? | 00:38 |
| *** prad has quit IRC | 00:39 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added robust message publisher and subscriber https://review.openstack.org/119266 | 00:40 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added robust message publisher https://review.openstack.org/119266 | 00:41 |
| *** lcheng_ has quit IRC | 00:41 | |
| jeblair | mika: it should do everything authenticated; i wonder if your gerrit doesn't support the expected authentication mechanism | 00:42 |
| clarkb | oh ya you need gerrit >2.4 iirc | 00:42 |
| clarkb | for the rest api | 00:42 |
| *** stevemar has joined #openstack-infra | 00:42 | |
| mika | i have gerrit 2.9 | 00:42 |
| krotscheck | jeblair: ^^ That should take care of the error from the webserver side of things. Basically it was caused whenever the MQ server either dropped a connection, restarted, or cleaned up its connection pool. | 00:42 |
| jeblair | clarkb, mika: i think gertty assumes http digest auth | 00:43 |
| jeblair | i'm not sure if gerrit has other choices | 00:43 |
| mika | jeblair: oh, i'm using ldap | 00:44 |
| mika | [auth] | 00:44 |
| mika | type = LDAP | 00:44 |
| *** ianw has joined #openstack-infra | 00:45 | |
| jeblair | mika: did you generate an http password in gerrit? | 00:46 |
| jeblair | mika: i think that's it | 00:47 |
| jeblair | mika: at least, i get the same behavior when i put in an invalid password | 00:47 |
| notmyname | jeblair: what is the git-root config variable in gerrty used for? | 00:48 |
| *** bdpayne has quit IRC | 00:48 | |
| jeblair | mika: so if you go to the gerrit webui, then "Settings" -> "HTTP Password" you should be able to generate one and use that with gertty. it should be completely separate from the ldap auth. | 00:48 |
| mika | jeblair: oh that sounds nice, thanks for the hint, will try | 00:48 |
| *** liusheng has joined #openstack-infra | 00:49 | |
| mordred | notmyname: that is the location underwhich gertty will clone your git repos | 00:49 |
| jeblair | notmyname: gertty will clone or re-use git repos that are there; eg mine is ~/git and i end up with ~/git/openstack/foo | 00:49 |
| mordred | mine is set to ~/src | 00:49 |
| notmyname | ok | 00:49 |
| jeblair | notmyname: the intent is that you can use those for your own hacking too | 00:49 |
| fungi | i use ~/work/openstack | 00:50 |
| *** chuckC has quit IRC | 00:50 | |
| mika | jeblair: seems to work, thanks for the hint (and gertty of course :)) | 00:50 |
| mordred | fungi: does that get you ~/work/openstack/openstack-infra/config ? | 00:50 |
| jeblair | notmyname: gertty won't manipulate the index or working tree in those repos unless you tell it to, it just fetches and diffs behind the scenes | 00:50 |
| fungi | mordred: yep | 00:50 |
| mordred | fungi: nice | 00:50 |
| *** yaguang has joined #openstack-infra | 00:50 | |
| fungi | mordred: i have other non-openstack subtrees under ~/work too | 00:50 |
| *** GheRivero has quit IRC | 00:50 | |
| fungi | mordred: useful to be able to treat it separately for things like backups and synchronization | 00:51 |
| *** yjiang5 has joined #openstack-infra | 00:51 | |
| jeblair | fungi: like the sekrit private fork of openstack the foundation has been working on? | 00:51 |
| *** GheRivero has joined #openstack-infra | 00:51 | |
| fungi | jeblair: yes, we'll be releasing it under an open-core subscription-based fee service any day now | 00:51 |
| *** marcoemorais has quit IRC | 00:51 | |
| jeblair | that's the spirit! | 00:52 |
| *** jgriffith has quit IRC | 00:52 | |
| fungi | a "fee software license" if you will ;) | 00:52 |
| jeblair | ba dum | 00:52 |
| *** jgriffith has joined #openstack-infra | 00:52 | |
| *** zhiwei has joined #openstack-infra | 00:52 | |
| mordred | fungi: you could call it "Stackzilla" | 00:52 |
| jeblair | so, lesson learned. for 1.0.1, talk more about gerrit passwords :) | 00:53 |
| clarkb | mordred: not dopenstack? | 00:53 |
| *** yamamoto has joined #openstack-infra | 00:53 | |
| fungi | mordred: and then rename it "firestack" a few years later | 00:54 |
| mordred | fungi: otoh - moztack might be snappier | 00:55 |
| mordred | ok. I'm going to go to the store and try to find food for my mouthhole | 00:56 |
| *** dims has joined #openstack-infra | 00:59 | |
| *** arnaud has quit IRC | 01:01 | |
| *** lakshmiS has quit IRC | 01:02 | |
| *** dims has quit IRC | 01:03 | |
| *** dims has joined #openstack-infra | 01:04 | |
| jeblair | okay, sent followup email to the list that addresses password issues | 01:05 |
| *** david-lyle has joined #openstack-infra | 01:05 | |
| jeblair | notmyname, mika: thanks for reminding me about those :) | 01:06 |
| *** dims_ has joined #openstack-infra | 01:06 | |
| *** yamamoto has quit IRC | 01:07 | |
| *** dims has quit IRC | 01:08 | |
| *** tomoe__ has joined #openstack-infra | 01:11 | |
| *** tomoe_ has quit IRC | 01:12 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/nodepool: Be more atomic when counting nodes https://review.openstack.org/119256 | 01:12 |
| fungi | nodepool has finally just caught up with the zuul waiting worker demand | 01:14 |
| fungi | for the first time in a couple days i think | 01:14 |
| *** dims_ has quit IRC | 01:14 | |
| *** dims has joined #openstack-infra | 01:14 | |
| *** david-lyle has quit IRC | 01:16 | |
| *** dims has quit IRC | 01:19 | |
| *** gokrokve has joined #openstack-infra | 01:19 | |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Fixed gate-trove-functional-dsvm jobs https://review.openstack.org/105692 | 01:20 |
| *** dims has joined #openstack-infra | 01:20 | |
| *** arnaud has joined #openstack-infra | 01:20 | |
| *** baohua has joined #openstack-infra | 01:27 | |
| *** baoli has joined #openstack-infra | 01:28 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:29 | |
| jesusaurus | ianw: i noticed that you abandoned 117425, i think that change is a good thing. i think that complex file discovery in bashate is better than massive find commands in tox.ini files | 01:31 |
| clarkb | jesusaurus: there is a ml thread about it iirc | 01:31 |
| * jesusaurus looks through the ml | 01:31 | |
| *** yamamoto has joined #openstack-infra | 01:32 | |
| *** arnaud__ has joined #openstack-infra | 01:32 | |
| *** salv-orlando has quit IRC | 01:33 | |
| *** dkliban is now known as dkliban_afk | 01:37 | |
| ianw | jesusaurus: i think shell files are used in so many ways it's a loosing battle. the worst is when you miss something that should be checked (as happened last night and broke devstack). better to just let projects decide what they want to match in their test harness. | 01:38 |
| *** _buzz has quit IRC | 01:40 | |
| openstackgerrit | Isaias proposed a change to openstack-infra/config: Watchdog class will not be installed in logstash_worker class anymore https://review.openstack.org/119272 | 01:40 |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Be more atomic when counting nodes https://review.openstack.org/119256 | 01:40 |
| *** _buzz has joined #openstack-infra | 01:41 | |
| *** mestery has quit IRC | 01:41 | |
| *** camunoz has quit IRC | 01:41 | |
| jesusaurus | hm, then maybe a good solution to my aversion to having a massive find in tox.ini is to stub it out into a tiny script in tools/ | 01:41 |
| *** ianw has quit IRC | 01:41 | |
| *** camunoz has joined #openstack-infra | 01:41 | |
| *** yamamoto has quit IRC | 01:41 | |
| *** ianw has joined #openstack-infra | 01:41 | |
| *** mestery has joined #openstack-infra | 01:42 | |
| *** yamamoto has joined #openstack-infra | 01:42 | |
| *** hdd has joined #openstack-infra | 01:44 | |
| *** tomoe_ has joined #openstack-infra | 01:45 | |
| *** dims has quit IRC | 01:46 | |
| *** dims has joined #openstack-infra | 01:47 | |
| *** yamahata has joined #openstack-infra | 01:47 | |
| *** nosnos has joined #openstack-infra | 01:49 | |
| *** tomoe__ has quit IRC | 01:49 | |
| *** dims_ has joined #openstack-infra | 01:49 | |
| openstackgerrit | Isaias proposed a change to openstack-infra/config: Watchdog class will not be installed in logstash_worker class anymore https://review.openstack.org/119272 | 01:49 |
| *** dims has quit IRC | 01:52 | |
| iax7 | clarkb, I made the change that you suggested about Watchdog class, could you take a look at it if that's what you had in mind? | 01:53 |
| clarkb | yup | 01:53 |
| *** Ryan_Lane has quit IRC | 01:54 | |
| *** sarob has joined #openstack-infra | 01:56 | |
| *** ryanpetrello has quit IRC | 01:56 | |
| *** david-lyle has joined #openstack-infra | 01:56 | |
| *** flaviof_zzz has joined #openstack-infra | 01:57 | |
| *** jyuso has joined #openstack-infra | 01:58 | |
| *** ryanpetrello has joined #openstack-infra | 02:00 | |
| *** praneshp has quit IRC | 02:01 | |
| *** david-lyle has quit IRC | 02:02 | |
| *** praneshp has joined #openstack-infra | 02:06 | |
| *** mfer has joined #openstack-infra | 02:07 | |
| StevenK | Once https://review.openstack.org/#/c/118090/ do the nodepool images need to be refreshed? | 02:10 |
| clarkb | no | 02:12 |
| clarkb | that change is in devstack so applies right away | 02:12 |
| clarkb | it doesnt fix anything other than a confusing error message though | 02:12 |
| *** mmaglana has quit IRC | 02:13 | |
| anteaya | the number of nodes in delete state has changed significantly for the better | 02:19 |
| *** koolhead17 has joined #openstack-infra | 02:21 | |
| *** dims_ has quit IRC | 02:25 | |
| *** dims has joined #openstack-infra | 02:25 | |
| *** psedlak has quit IRC | 02:27 | |
| *** harlowja is now known as harlowja_away | 02:27 | |
| *** psedlak has joined #openstack-infra | 02:29 | |
| *** dims_ has joined #openstack-infra | 02:30 | |
| *** dims has quit IRC | 02:30 | |
| *** nelsnels_ has joined #openstack-infra | 02:30 | |
| *** nelsnelson has quit IRC | 02:33 | |
| openstackgerrit | Jeffrey Zhang proposed a change to openstack-infra/config: add openstack-rating channel to eavesdrop https://review.openstack.org/117057 | 02:33 |
| *** wayne__ has quit IRC | 02:34 | |
| *** nelsnels_ has quit IRC | 02:35 | |
| *** dkehnx has quit IRC | 02:37 | |
| *** doug-fish has quit IRC | 02:39 | |
| *** doug-fish has joined #openstack-infra | 02:40 | |
| *** unicell has quit IRC | 02:43 | |
| *** adalbas has quit IRC | 02:45 | |
| *** Ryan_Lane has joined #openstack-infra | 02:46 | |
| *** Daisy has joined #openstack-infra | 02:49 | |
| *** arnaud__ has quit IRC | 02:55 | |
| *** arnaud has quit IRC | 02:55 | |
| *** baohua has quit IRC | 02:59 | |
| *** baohua has joined #openstack-infra | 03:00 | |
| *** mfer has quit IRC | 03:01 | |
| *** zz_naotok is now known as naotok | 03:03 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: add experimental job for nova with f20 virt-preview https://review.openstack.org/113350 | 03:04 |
| *** dims_ has quit IRC | 03:04 | |
| *** dims has joined #openstack-infra | 03:05 | |
| *** _buzz has quit IRC | 03:06 | |
| *** _buzz has joined #openstack-infra | 03:07 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Clean up bashate failures https://review.openstack.org/118944 | 03:08 |
| *** bdpayne has joined #openstack-infra | 03:09 | |
| *** dims has quit IRC | 03:10 | |
| *** yamamoto has quit IRC | 03:11 | |
| *** yamamoto has joined #openstack-infra | 03:12 | |
| *** yamamot__ has joined #openstack-infra | 03:13 | |
| *** yamamoto has quit IRC | 03:17 | |
| *** radez is now known as radez_g0n3 | 03:27 | |
| *** jerryz has quit IRC | 03:29 | |
| *** TravT has quit IRC | 03:29 | |
| *** praneshp has quit IRC | 03:32 | |
| *** zz_dimtruck is now known as dimtruck | 03:39 | |
| *** baohua_ has joined #openstack-infra | 03:44 | |
| *** baohua_ has quit IRC | 03:47 | |
| *** baohua has quit IRC | 03:47 | |
| *** baohua has joined #openstack-infra | 03:48 | |
| *** homeless has quit IRC | 03:48 | |
| *** daya_k has joined #openstack-infra | 03:49 | |
| yamamot__ | why official jenkins does not suffer https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1348830 ? | 03:51 |
| *** jyuso has quit IRC | 03:51 | |
| uvirtbot | Launchpad bug 1348830 in nova "nova interface attach command fails in 14.04 with 2014.1.1: error: "libvirtError: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS" (dup-of: 1322568)" [Undecided,Confirmed] | 03:51 |
| uvirtbot | Launchpad bug 1322568 in libvirt "nova interface-attach fails" [High,Fix committed] | 03:51 |
| *** baoli has quit IRC | 03:54 | |
| *** tomoe_ has quit IRC | 03:57 | |
| *** jamespage has quit IRC | 04:00 | |
| *** gnuoy has quit IRC | 04:00 | |
| *** gnuoy has joined #openstack-infra | 04:00 | |
| *** jamespage has joined #openstack-infra | 04:00 | |
| *** pcrews has quit IRC | 04:01 | |
| *** camunoz has quit IRC | 04:04 | |
| *** ianw has quit IRC | 04:04 | |
| *** ianw has joined #openstack-infra | 04:04 | |
| *** camunoz has joined #openstack-infra | 04:04 | |
| *** torandu has joined #openstack-infra | 04:07 | |
| *** Ryan_Lane has quit IRC | 04:07 | |
| *** camunoz has quit IRC | 04:10 | |
| *** camunoz has joined #openstack-infra | 04:10 | |
| *** ianw has quit IRC | 04:10 | |
| *** baohua has quit IRC | 04:11 | |
| *** ianw has joined #openstack-infra | 04:11 | |
| *** arnaud has joined #openstack-infra | 04:16 | |
| *** harlowja_at_home has joined #openstack-infra | 04:18 | |
| *** praneshp has joined #openstack-infra | 04:20 | |
| *** miqui has quit IRC | 04:23 | |
| *** praneshp_ has joined #openstack-infra | 04:24 | |
| *** praneshp has quit IRC | 04:27 | |
| *** praneshp_ is now known as praneshp | 04:27 | |
| *** SumitNaiksatam_ has joined #openstack-infra | 04:32 | |
| *** bdpayne has quit IRC | 04:33 | |
| *** tomoe_ has joined #openstack-infra | 04:35 | |
| *** SumitNaiksatam has quit IRC | 04:35 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 04:35 | |
| *** Ryan_Lane has joined #openstack-infra | 04:38 | |
| *** Ryan_Lane has joined #openstack-infra | 04:39 | |
| *** pcrews has joined #openstack-infra | 04:41 | |
| *** tomoe_ has quit IRC | 04:41 | |
| *** tomoe_ has joined #openstack-infra | 04:42 | |
| *** dimtruck is now known as zz_dimtruck | 04:46 | |
| *** rushiagr_away is now known as rushiagr | 04:47 | |
| *** pcrews has quit IRC | 04:48 | |
| *** hdd has quit IRC | 04:49 | |
| *** baohua has joined #openstack-infra | 04:52 | |
| *** _nadya_ has joined #openstack-infra | 04:52 | |
| *** Ryan_Lane has quit IRC | 05:02 | |
| *** Ryan_Lane has joined #openstack-infra | 05:03 | |
| *** kevinbenton has quit IRC | 05:03 | |
| *** yjiang5 has quit IRC | 05:05 | |
| *** _nadya_ has quit IRC | 05:06 | |
| *** kevinbenton has joined #openstack-infra | 05:06 | |
| *** gokrokve_ has joined #openstack-infra | 05:09 | |
| *** gokrokve has quit IRC | 05:11 | |
| *** krtaylor has quit IRC | 05:12 | |
| *** salv-orlando has joined #openstack-infra | 05:15 | |
| *** lcheng_ has joined #openstack-infra | 05:17 | |
| *** krtaylor has joined #openstack-infra | 05:20 | |
| *** salv-orlando has quit IRC | 05:22 | |
| *** ppai has joined #openstack-infra | 05:24 | |
| *** SumitNaiksatam has quit IRC | 05:24 | |
| *** TravT has joined #openstack-infra | 05:27 | |
| *** camunoz has quit IRC | 05:27 | |
| *** ianw has quit IRC | 05:27 | |
| *** camunoz has joined #openstack-infra | 05:27 | |
| *** ianw has joined #openstack-infra | 05:27 | |
| *** loki184 has joined #openstack-infra | 05:27 | |
| *** gargola has quit IRC | 05:29 | |
| *** flaper87|afk is now known as flaper87 | 05:29 | |
| *** Ryan_Lane has quit IRC | 05:34 | |
| *** penguinRaider has quit IRC | 05:37 | |
| *** penguinRaider has joined #openstack-infra | 05:38 | |
| *** hdd has joined #openstack-infra | 05:39 | |
| *** Ryan_Lane has joined #openstack-infra | 05:39 | |
| *** alkari has quit IRC | 05:41 | |
| *** alkari has joined #openstack-infra | 05:42 | |
| *** hdd has quit IRC | 05:42 | |
| *** amotoki has joined #openstack-infra | 05:52 | |
| kevinbenton | hello folks. how do i turn this into a recheck bug? | 05:52 |
| kevinbenton | http://logstash.openstack.org/#eyJzZWFyY2giOiIgbWVzc2FnZTpcIkZBSUw6IG5ldXRyb24udGVzdHMudW5pdC50ZXN0X3NlY3VyaXR5X2dyb3Vwc19ycGMuVGVzdFNlY3VyaXR5R3JvdXBBZ2VudEVuaGFuY2VkUnBjV2l0aElwdGFibGVzLnRlc3RfcHJlcGFyZV9yZW1vdmVfcG9ydFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIiwic3RhbXAiOjE0MDk4OTYyNzg1NzV9 | 05:52 |
| jesusaurus | you dont need to recheck against a bug anymore, you can just use "recheck" | 05:54 |
| *** frank_zhou has joined #openstack-infra | 05:56 | |
| *** unicell has joined #openstack-infra | 05:58 | |
| *** sabeen has joined #openstack-infra | 05:58 | |
| kevinbenton | jesusaurus: i was just wondering how that process is done | 06:02 |
| kevinbenton | jesusaurus: to make an elastic recheck detector | 06:03 |
| *** stevemar has quit IRC | 06:03 | |
| *** harlowja_at_home has quit IRC | 06:04 | |
| adam_g | kevinbenton, you create a query and propose it to the elastic-recheck repo. see https://wiki.openstack.org/wiki/ElasticRecheck | 06:05 |
| *** oomichi has quit IRC | 06:05 | |
| *** yamamot__ has quit IRC | 06:05 | |
| *** yamamoto has joined #openstack-infra | 06:05 | |
| kevinbenton | adam_g: oh okay. well the fix is already in review so probably not worth it now. | 06:05 |
| openstackgerrit | Kevin Benton proposed a change to openstack-infra/elastic-recheck: Add recheck query for bug 1365829 https://review.openstack.org/119299 | 06:11 |
| uvirtbot | Launchpad bug 1365829 in neutron "ipv6 tests leave ipv6 flag disabled" [High,In progress] https://launchpad.net/bugs/1365829 | 06:11 |
| *** SumitNaiksatam has joined #openstack-infra | 06:11 | |
| kevinbenton | adam_g: like ^^ https://review.openstack.org/#/c/119299/? | 06:11 |
| adam_g | kevinbenton, yeah, tho you probably want to make your query a bit more specific than just the fail string | 06:13 |
| *** pkoniszewski has joined #openstack-infra | 06:13 | |
| *** Daisy has quit IRC | 06:15 | |
| *** koolhead17 has quit IRC | 06:16 | |
| openstackgerrit | Kevin Benton proposed a change to openstack-infra/elastic-recheck: Add recheck query for bug 1365829 https://review.openstack.org/119299 | 06:16 |
| uvirtbot | Launchpad bug 1365829 in neutron "ipv6 tests leave ipv6 flag disabled" [High,In progress] https://launchpad.net/bugs/1365829 | 06:16 |
| kevinbenton | adam_g: how about that one ^^? | 06:16 |
| *** HeOS has quit IRC | 06:17 | |
| *** salv-orlando has joined #openstack-infra | 06:20 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Update sideways Ironic job https://review.openstack.org/119166 | 06:23 |
| *** cipcosma has joined #openstack-infra | 06:24 | |
| *** k4n0 has joined #openstack-infra | 06:25 | |
| adam_g | kevinbenton, i was thinking more about targeting a string somewhere that identifies the specific failure case you're looking for. as is, it would be possible for that test to fail for another reason and be incorrectly identified as your bug | 06:25 |
| *** gyee has quit IRC | 06:25 | |
| adam_g | ie, 'MismatchError: 8 != 4' | 06:25 |
| kevinbenton | adam_g: oh, i see | 06:26 |
| *** rajesht has joined #openstack-infra | 06:27 | |
| *** camunoz has quit IRC | 06:27 | |
| *** liam_ has joined #openstack-infra | 06:27 | |
| *** liam_ is now known as Guest22638 | 06:28 | |
| openstackgerrit | Kevin Benton proposed a change to openstack-infra/elastic-recheck: Add recheck query for bug 1365829 https://review.openstack.org/119299 | 06:30 |
| uvirtbot | Launchpad bug 1365829 in neutron "ipv6 tests leave ipv6 flag disabled" [High,In progress] https://launchpad.net/bugs/1365829 | 06:30 |
| kevinbenton | adam_g: good call. looks like it was occuring in a couple of other unit tests anyway | 06:31 |
| *** openstack_ has joined #openstack-infra | 06:36 | |
| *** JadnG has quit IRC | 06:39 | |
| *** frank_zhou has quit IRC | 06:46 | |
| *** arnaud has quit IRC | 06:47 | |
| *** doude has joined #openstack-infra | 06:47 | |
| *** Guest22638 has quit IRC | 06:48 | |
| *** Daisy has joined #openstack-infra | 06:59 | |
| *** sabeen has quit IRC | 07:02 | |
| *** salv-orlando has quit IRC | 07:04 | |
| *** arxcruz has joined #openstack-infra | 07:07 | |
| *** tsg has joined #openstack-infra | 07:08 | |
| *** yolanda has joined #openstack-infra | 07:12 | |
| *** luqas has joined #openstack-infra | 07:14 | |
| *** jistr has joined #openstack-infra | 07:15 | |
| *** daya_k has quit IRC | 07:19 | |
| *** marun_afk has quit IRC | 07:22 | |
| *** tsg has quit IRC | 07:25 | |
| *** pblaho has joined #openstack-infra | 07:29 | |
| *** ppai has quit IRC | 07:29 | |
| *** jp_at_hp has joined #openstack-infra | 07:29 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/jenkins-job-builder: Add hipchat section to jjb sample https://review.openstack.org/119305 | 07:31 |
| *** HeOS has joined #openstack-infra | 07:31 | |
| *** jcoufal has joined #openstack-infra | 07:32 | |
| *** KanagarajM has joined #openstack-infra | 07:34 | |
| *** MaxV has joined #openstack-infra | 07:37 | |
| *** tsg has joined #openstack-infra | 07:38 | |
| *** lcheng_ has quit IRC | 07:39 | |
| *** _nadya_ has joined #openstack-infra | 07:41 | |
| *** tsg has quit IRC | 07:42 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/jenkins-job-builder: Enable hipchat section into sample file https://review.openstack.org/119307 | 07:42 |
| *** ppai has joined #openstack-infra | 07:42 | |
| *** Guest42286 has joined #openstack-infra | 07:43 | |
| *** ajo|call is now known as ajo | 07:44 | |
| *** andreykurilin_ has joined #openstack-infra | 07:46 | |
| *** yamamoto has quit IRC | 07:47 | |
| *** YorikSar has quit IRC | 07:48 | |
| *** mpaolino has joined #openstack-infra | 07:49 | |
| *** skraynev has quit IRC | 07:50 | |
| *** YorikSar has joined #openstack-infra | 07:50 | |
| *** derekh has joined #openstack-infra | 07:54 | |
| *** YorikSar has quit IRC | 07:56 | |
| *** Longgeek has joined #openstack-infra | 07:57 | |
| *** ilyashakhat has quit IRC | 07:58 | |
| *** enikanorov has quit IRC | 07:58 | |
| *** naotok is now known as zz_naotok | 07:58 | |
| *** _nadya_ has quit IRC | 08:03 | |
| *** YorikSar has joined #openstack-infra | 08:06 | |
| *** afazekas has joined #openstack-infra | 08:09 | |
| *** jlibosva has joined #openstack-infra | 08:09 | |
| *** rlandy has joined #openstack-infra | 08:11 | |
| *** ilyashakhat has joined #openstack-infra | 08:11 | |
| *** yamamoto has joined #openstack-infra | 08:12 | |
| *** YorikSar_ has joined #openstack-infra | 08:12 | |
| *** rajesht_ has joined #openstack-infra | 08:13 | |
| *** ilyashakhat has quit IRC | 08:14 | |
| *** e0ne has joined #openstack-infra | 08:14 | |
| *** Longgeek has quit IRC | 08:14 | |
| *** TravT has quit IRC | 08:15 | |
| openstackgerrit | Longgeek proposed a change to openstack-infra/puppet-yum: Add .gitreview Rakefile files and update puppet coding style https://review.openstack.org/102876 | 08:15 |
| *** rajesht has quit IRC | 08:15 | |
| *** dtantsur|afk is now known as dtantsur | 08:15 | |
| *** Longgeek has joined #openstack-infra | 08:15 | |
| *** dkehn has quit IRC | 08:15 | |
| *** medieval1 has quit IRC | 08:16 | |
| *** dkehnx1 has quit IRC | 08:16 | |
| *** Longgeek_ has joined #openstack-infra | 08:16 | |
| *** YorikSar has quit IRC | 08:16 | |
| *** ZZelle has quit IRC | 08:19 | |
| *** ZZelle has joined #openstack-infra | 08:19 | |
| *** Longgeek has quit IRC | 08:20 | |
| *** pblaho is now known as pblaho|afk | 08:24 | |
| *** e0ne has quit IRC | 08:24 | |
| *** ociuhandu has quit IRC | 08:25 | |
| *** medieval1 has joined #openstack-infra | 08:25 | |
| *** e0ne has joined #openstack-infra | 08:26 | |
| *** dkehn has joined #openstack-infra | 08:28 | |
| *** dkehnx1 has joined #openstack-infra | 08:28 | |
| *** yamamoto has quit IRC | 08:30 | |
| *** arxcruz has quit IRC | 08:32 | |
| *** medieval1 has quit IRC | 08:33 | |
| *** praneshp has quit IRC | 08:33 | |
| *** arxcruz has joined #openstack-infra | 08:34 | |
| *** arxcruz has quit IRC | 08:37 | |
| *** derekh has quit IRC | 08:37 | |
| *** pblaho|afk is now known as pblaho | 08:38 | |
| *** arxcruz has joined #openstack-infra | 08:39 | |
| *** marun_afk has joined #openstack-infra | 08:40 | |
| *** medieval1 has joined #openstack-infra | 08:41 | |
| *** KanagarajM has quit IRC | 08:46 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:47 | |
| *** yongli has quit IRC | 08:48 | |
| *** gokrokve has joined #openstack-infra | 08:49 | |
| *** alexpilotti has joined #openstack-infra | 08:50 | |
| *** gokrokve_ has quit IRC | 08:53 | |
| *** gokrokve has quit IRC | 08:53 | |
| *** ppai has quit IRC | 08:54 | |
| *** skraynev has joined #openstack-infra | 08:55 | |
| *** andreykurilin_ has quit IRC | 09:00 | |
| *** rajesht has joined #openstack-infra | 09:05 | |
| *** salv-orlando has joined #openstack-infra | 09:05 | |
| *** rajesht_ has quit IRC | 09:08 | |
| *** linkid has joined #openstack-infra | 09:08 | |
| *** marun_afk is now known as marun | 09:09 | |
| *** dtantsur is now known as dtantsur|brb | 09:10 | |
| linkid | hi | 09:10 |
| *** Daisy has quit IRC | 09:10 | |
| linkid | I had the same bug after 2 rechecks, on almost every tests | 09:10 |
| linkid | http://logs.openstack.org/98/116498/1/check/gate-cloudkitty-dashboard-python26/e140e6f/console.html | 09:10 |
| *** ppai has joined #openstack-infra | 09:10 | |
| linkid | is it a known issue or do I have to report it on Launchpad ? | 09:11 |
| *** yamamoto has joined #openstack-infra | 09:12 | |
| *** penguinRaider has quit IRC | 09:15 | |
| *** daya_k has joined #openstack-infra | 09:16 | |
| *** k4n0 has quit IRC | 09:16 | |
| *** yamamoto_ has joined #openstack-infra | 09:16 | |
| *** yamamoto has quit IRC | 09:17 | |
| *** isviridov_away is now known as isviridov | 09:18 | |
| *** andreaf has joined #openstack-infra | 09:19 | |
| *** ildikov has joined #openstack-infra | 09:20 | |
| *** ppai has quit IRC | 09:23 | |
| *** marun is now known as marun_afk | 09:24 | |
| *** marun_afk is now known as marun_ | 09:24 | |
| *** marun_ is now known as marun | 09:24 | |
| *** gokrokve has joined #openstack-infra | 09:25 | |
| *** mrmartin has joined #openstack-infra | 09:26 | |
| *** pkoniszewski has quit IRC | 09:30 | |
| *** gokrokve has quit IRC | 09:30 | |
| *** pkoniszewski has joined #openstack-infra | 09:31 | |
| *** YorikSar_ has quit IRC | 09:31 | |
| *** zz_naotok is now known as naotok | 09:31 | |
| *** penguinRaider has joined #openstack-infra | 09:31 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Add neutron full postgres job to integrated gate https://review.openstack.org/119332 | 09:32 |
| *** YorikSar has joined #openstack-infra | 09:32 | |
| *** KanagarajM has joined #openstack-infra | 09:33 | |
| *** GheRivero is now known as GheRivero_pto | 09:34 | |
| *** ppai has joined #openstack-infra | 09:35 | |
| *** KanagarajM2 has joined #openstack-infra | 09:36 | |
| *** KanagarajM has quit IRC | 09:38 | |
| *** daya_k has quit IRC | 09:38 | |
| *** pelix has joined #openstack-infra | 09:39 | |
| *** ociuhandu has joined #openstack-infra | 09:41 | |
| tchaypo | huh. no derek. | 09:42 |
| *** daya_k has joined #openstack-infra | 09:43 | |
| *** yamamoto_ has quit IRC | 09:45 | |
| *** yamamoto has joined #openstack-infra | 09:48 | |
| *** bradjones has quit IRC | 09:49 | |
| *** baohua has quit IRC | 09:54 | |
| *** rajesht has quit IRC | 09:57 | |
| *** rajesht_ has joined #openstack-infra | 09:57 | |
| rajesht_ | Hi | 09:58 |
| rajesht_ | Jenkins is failing on following patches because of environment issues. | 09:58 |
| rajesht_ | https://review.openstack.org/#/c/118672/ | 09:59 |
| rajesht_ | https://review.openstack.org/#/c/118674 | 09:59 |
| rajesht_ | I am not able to reproduce the same in my environment. | 09:59 |
| rajesht_ | Could anybody please help me understand what is the reason jenkins gets failed randomly. | 10:00 |
| *** _nadya_ has joined #openstack-infra | 10:00 | |
| *** pblaho is now known as pblaho|afk | 10:01 | |
| *** _nadya_ has quit IRC | 10:01 | |
| *** vhoward has joined #openstack-infra | 10:02 | |
| *** zhiwei has quit IRC | 10:03 | |
| *** bookwar has quit IRC | 10:04 | |
| *** rajesht has joined #openstack-infra | 10:09 | |
| *** Guest42286 has quit IRC | 10:11 | |
| *** rajesht_ has quit IRC | 10:11 | |
| *** pcm_ has joined #openstack-infra | 10:14 | |
| *** _nadya_ has joined #openstack-infra | 10:15 | |
| *** pcm_ has quit IRC | 10:16 | |
| *** jcoufal has quit IRC | 10:16 | |
| *** pcm_ has joined #openstack-infra | 10:16 | |
| *** bradjones has joined #openstack-infra | 10:17 | |
| *** bradjones has joined #openstack-infra | 10:17 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Accept capitalized Sem-Ver headers https://review.openstack.org/118648 | 10:18 |
| *** yfried_ has quit IRC | 10:21 | |
| *** andreaf has quit IRC | 10:23 | |
| *** derekh has joined #openstack-infra | 10:23 | |
| *** andreaf has joined #openstack-infra | 10:23 | |
| rajesht | could anybody please help me understand why jenkins gets failed randomly ? | 10:25 |
| *** Longgeek_ has quit IRC | 10:25 | |
| *** gokrokve has joined #openstack-infra | 10:25 | |
| *** salv-orlando has quit IRC | 10:28 | |
| *** gokrokve has quit IRC | 10:30 | |
| *** yaguang has quit IRC | 10:31 | |
| *** pblaho|afk is now known as pblaho | 10:31 | |
| *** yfried_ has joined #openstack-infra | 10:38 | |
| *** andreaf_ has quit IRC | 10:39 | |
| *** andreaf_ has joined #openstack-infra | 10:40 | |
| *** koolhead17 has joined #openstack-infra | 10:40 | |
| *** jcoufal has joined #openstack-infra | 10:41 | |
| *** salv-orlando has joined #openstack-infra | 10:45 | |
| *** yamahata has quit IRC | 10:46 | |
| *** baoli has joined #openstack-infra | 10:52 | |
| *** salv-orlando_ has joined #openstack-infra | 10:53 | |
| *** salv-orlando_ has quit IRC | 10:53 | |
| *** dims has joined #openstack-infra | 10:54 | |
| *** salv-orlando has quit IRC | 10:55 | |
| *** dims has quit IRC | 10:56 | |
| *** dims_ has joined #openstack-infra | 10:57 | |
| *** dims has joined #openstack-infra | 10:57 | |
| *** KanagarajM2 has quit IRC | 10:57 | |
| *** ildikov_ has joined #openstack-infra | 11:00 | |
| *** dims_ has quit IRC | 11:01 | |
| *** dtantsur|brb is now known as dtantsur | 11:01 | |
| *** ildikov has quit IRC | 11:03 | |
| *** pkoniszewski has quit IRC | 11:06 | |
| *** luqas has quit IRC | 11:07 | |
| *** pkoniszewski has joined #openstack-infra | 11:07 | |
| *** e0ne has quit IRC | 11:08 | |
| *** loki184 has quit IRC | 11:09 | |
| *** andreaf has quit IRC | 11:09 | |
| *** andreaf has joined #openstack-infra | 11:10 | |
| *** flaviof_zzz is now known as flaviof | 11:20 | |
| *** Longgeek has joined #openstack-infra | 11:25 | |
| *** gokrokve has joined #openstack-infra | 11:25 | |
| *** mmedvede has joined #openstack-infra | 11:28 | |
| *** Longgeek has quit IRC | 11:30 | |
| *** gokrokve has quit IRC | 11:30 | |
| dims | SergeyLukjanov: ping. was there any concerns about this one? https://review.openstack.org/#/c/117309/ i saw a +A followed by a -A | 11:31 |
| *** tomoe__ has joined #openstack-infra | 11:31 | |
| dims | SergeyLukjanov: please let me know and i'll try to take care of it :) | 11:32 |
| SergeyLukjanov | dims, I've removed approve to avoid irc bots restart during the working day / irc meetings | 11:33 |
| dims | SergeyLukjanov: ah cool | 11:33 |
| dims | did not realize that | 11:34 |
| *** tomoe_ has quit IRC | 11:34 | |
| *** mpaolino has quit IRC | 11:36 | |
| *** e0ne has joined #openstack-infra | 11:36 | |
| *** tomoe__ has quit IRC | 11:45 | |
| *** Longgeek has joined #openstack-infra | 11:45 | |
| *** Guest42286 has joined #openstack-infra | 11:45 | |
| *** naotok is now known as zz_naotok | 11:46 | |
| *** Longgeek has quit IRC | 11:46 | |
| *** mbacchi has joined #openstack-infra | 11:47 | |
| *** Longgeek has joined #openstack-infra | 11:48 | |
| *** lttrl has joined #openstack-infra | 11:52 | |
| *** dmsimard_away is now known as dmsimard | 11:53 | |
| *** k4n0 has joined #openstack-infra | 11:55 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add missing oslo projects to gerritbot config https://review.openstack.org/117309 | 11:57 |
| *** mrmartin has quit IRC | 11:58 | |
| *** mfink has quit IRC | 11:59 | |
| *** openstackgerrit has quit IRC | 12:01 | |
| *** openstackgerrit has joined #openstack-infra | 12:02 | |
| *** andreaf has quit IRC | 12:07 | |
| *** andreaf has joined #openstack-infra | 12:07 | |
| *** weshay has joined #openstack-infra | 12:09 | |
| *** dprince has joined #openstack-infra | 12:10 | |
| *** mwagner_lap has joined #openstack-infra | 12:10 | |
| *** luqas has joined #openstack-infra | 12:14 | |
| *** tomoe_ has joined #openstack-infra | 12:17 | |
| *** koolhead17 has quit IRC | 12:19 | |
| *** crinkle has quit IRC | 12:19 | |
| *** dims has quit IRC | 12:19 | |
| *** nosnos has quit IRC | 12:19 | |
| *** dims has joined #openstack-infra | 12:20 | |
| *** nosnos has joined #openstack-infra | 12:20 | |
| *** lttrl has quit IRC | 12:23 | |
| *** lcheng_ has joined #openstack-infra | 12:24 | |
| *** aysyd has joined #openstack-infra | 12:24 | |
| *** nosnos has quit IRC | 12:24 | |
| *** gokrokve has joined #openstack-infra | 12:25 | |
| *** adalbas has joined #openstack-infra | 12:26 | |
| *** YorikSar has quit IRC | 12:26 | |
| *** Longgeek has quit IRC | 12:28 | |
| *** Longgeek has joined #openstack-infra | 12:29 | |
| *** gokrokve has quit IRC | 12:30 | |
| *** Ajaeger1 has joined #openstack-infra | 12:30 | |
| *** bswartz has quit IRC | 12:31 | |
| *** ttx has quit IRC | 12:31 | |
| *** jaypipes is now known as leakypipes | 12:32 | |
| Ajaeger1 | Hi, any idea why I get a failure of "No distributions matching the version for oslotest>=1.1.0.0a1" for a python 3.3 job? It seems to be available at http://pypi.openstack.org/simple/oslotest/ but fails in https://review.openstack.org/115606 and other oslo requirements test in a reliable way | 12:33 |
| *** Longgeek_ has joined #openstack-infra | 12:33 | |
| *** YorikSar has joined #openstack-infra | 12:33 | |
| *** ttx has joined #openstack-infra | 12:34 | |
| *** ttx has quit IRC | 12:34 | |
| *** ttx has joined #openstack-infra | 12:34 | |
| *** linkid has left #openstack-infra | 12:35 | |
| *** Longgeek has quit IRC | 12:37 | |
| *** fungi has quit IRC | 12:37 | |
| *** dims has quit IRC | 12:39 | |
| *** dims has joined #openstack-infra | 12:40 | |
| *** ppai has quit IRC | 12:41 | |
| *** jbryce has quit IRC | 12:41 | |
| *** lbragstad has quit IRC | 12:43 | |
| *** lbragstad has joined #openstack-infra | 12:43 | |
| *** jbryce has joined #openstack-infra | 12:43 | |
| *** d34dh0r53 has quit IRC | 12:44 | |
| *** d34dh0r53 has joined #openstack-infra | 12:44 | |
| *** pabelanger has quit IRC | 12:45 | |
| *** dims has quit IRC | 12:45 | |
| *** sarob has quit IRC | 12:45 | |
| *** amotoki has quit IRC | 12:45 | |
| *** dims_ has joined #openstack-infra | 12:46 | |
| *** pabelanger has joined #openstack-infra | 12:47 | |
| *** derekh has quit IRC | 12:48 | |
| *** flaviof has quit IRC | 12:48 | |
| *** dims_ has quit IRC | 12:49 | |
| *** miqui has joined #openstack-infra | 12:50 | |
| *** dims has joined #openstack-infra | 12:50 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid.org openid instance https://review.openstack.org/119364 | 12:52 |
| *** morganfainberg has quit IRC | 12:54 | |
| *** paul-- has quit IRC | 12:54 | |
| *** jhesketh has quit IRC | 12:54 | |
| *** daya_k has quit IRC | 12:57 | |
| *** morganfainberg has joined #openstack-infra | 12:58 | |
| *** paul-- has joined #openstack-infra | 12:58 | |
| *** jhesketh has joined #openstack-infra | 12:58 | |
| *** mfink_ has joined #openstack-infra | 12:59 | |
| *** lcheng_ has quit IRC | 13:04 | |
| *** Clabbe has joined #openstack-infra | 13:04 | |
| *** fungi has joined #openstack-infra | 13:04 | |
| *** cdent has joined #openstack-infra | 13:05 | |
| *** hdd has joined #openstack-infra | 13:06 | |
| dhellmann | Ajaeger1: we don't have a python 3 wheel of oslotest. clarkb had some ideas about how to fix that, but during the rush up to feature freeze I wasn't worried about making the infra changes needed, but now I think we need to look at it more closely. | 13:08 |
| Ajaeger1 | dhellmann: So, should we make the python33 jobs non-voting for now? Or fix the wheel? | 13:09 |
| dhellmann | Ajaeger1: I need to talk to clarkb and fungi about that this morning to decide the best course of action | 13:09 |
| Ajaeger1 | dhellmann: btw check https://review.openstack.org/#/q/status:open++branch:master+topic:openstack/requirements,n,z - there're quite a few failures due to this. | 13:09 |
| Ajaeger1 | dhellmann: Great, thanks! | 13:10 |
| fungi | dhellmann: did the conversion to mox3 get stalled? | 13:10 |
| fungi | dhellmann: i vaguely recall seeing a patch proposed to oslotest for that last week | 13:10 |
| *** bswartz has joined #openstack-infra | 13:10 | |
| dhellmann | fungi: yeah, I didn't want to risk that breaking something right before ff, so I WIP-ed it | 13:11 |
| *** derekh has joined #openstack-infra | 13:11 | |
| dhellmann | fungi: approving https://review.openstack.org/#/c/118551/ now | 13:12 |
| *** _nadya_ has quit IRC | 13:13 | |
| dhellmann | fungi: after that, I have 2 others I'd like to land before we cut a release. What other changes are needed to the release tools/processes to build the right kinds of wheels? | 13:13 |
| *** pelix has quit IRC | 13:13 | |
| *** _nadya_ has joined #openstack-infra | 13:14 | |
| *** andreaf has quit IRC | 13:14 | |
| fungi | dhellmann: i believe you need to set universal=true in setup.cfg, but i think that's all? | 13:14 |
| *** andreaf has joined #openstack-infra | 13:14 | |
| fungi | looks like 118551 already does that | 13:14 |
| dhellmann | fungi: ok, I thought we had a python 2 and universal wheel already, is it just python 2? | 13:14 |
| dhellmann | oh, or maybe we only had a python 2 wheel then | 13:15 |
| *** sarob has joined #openstack-infra | 13:15 | |
| fungi | dhellmann: so in theory what you have there should "just work" but we can try to do a 'tox -e venv python setup.py bdist_wheel' and then examine/install the results | 13:15 |
| *** flaviof_zzz has joined #openstack-infra | 13:16 | |
| dhellmann | ok | 13:16 |
| fungi | e.g. preinstall it into a -p python3.3 virtualenv and tell tox -e py33 to use that | 13:16 |
| fungi | for whatever projects currently claim a test-requirement on it | 13:17 |
| *** mrmartin has joined #openstack-infra | 13:20 | |
| *** dustins has joined #openstack-infra | 13:22 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid.org openid instance https://review.openstack.org/119364 | 13:24 |
| *** mriedem has joined #openstack-infra | 13:24 | |
| *** morganfainberg has quit IRC | 13:25 | |
| *** paul-- has quit IRC | 13:25 | |
| *** jhesketh has quit IRC | 13:25 | |
| *** gokrokve has joined #openstack-infra | 13:25 | |
| *** morganfainberg has joined #openstack-infra | 13:27 | |
| *** paul-- has joined #openstack-infra | 13:27 | |
| *** jhesketh has joined #openstack-infra | 13:27 | |
| *** marun is now known as maroon | 13:28 | |
| dhellmann | fungi: ok, that patch merged and now I get only a "py2.py3-none-any" wheel file when I run bdist_wheel | 13:29 |
| *** gokrokve has quit IRC | 13:30 | |
| dhellmann | fungi: we have 2 more patches I'd like to land, and then I'll cut a release and see if that fixes the installation issue | 13:30 |
| *** bknudson has quit IRC | 13:30 | |
| fungi | dhellmann: keen | 13:32 |
| *** pcrews has joined #openstack-infra | 13:33 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 13:33 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 13:33 | |
| *** d34dh0r53 has quit IRC | 13:34 | |
| *** d34dh0r53 has joined #openstack-infra | 13:35 | |
| *** craigbr has joined #openstack-infra | 13:36 | |
| *** julim has joined #openstack-infra | 13:37 | |
| *** dhellmann is now known as dhellmann_ | 13:40 | |
| *** dustins has quit IRC | 13:43 | |
| *** pblaho has quit IRC | 13:43 | |
| *** homeless has joined #openstack-infra | 13:44 | |
| Ajaeger1 | fungi, I think we have a problem with the merge check. When a patch is currently in the check queue and then a merge happens, it does not get checked. See https://review.openstack.org/#/c/110129 which is exactly the same as https://review.openstack.org/#/c/116055/ but does not have a merge conflict | 13:44 |
| *** dustins has joined #openstack-infra | 13:45 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add some notes to the readme about queries that don't hit https://review.openstack.org/119184 | 13:47 |
| *** trinaths has joined #openstack-infra | 13:47 | |
| trinaths | Hi, I get this error in my devstack. "ERROR: cliff.app Authorization Failed: Unable to sign token. " How to handle this error? | 13:48 |
| fungi | Ajaeger1: i think that's a tradeoff for design simplicity. a more complicated alternative would be to have zuul abort currently running jobs and report the merge conflict which crept up after that change's jobs started but before they finished | 13:49 |
| *** mfer has joined #openstack-infra | 13:49 | |
| fungi | trinaths: you might want to ask questions about devstack, tempest, et cetera in #openstack-qa instead. that's where they generally get discussed | 13:50 |
| Ajaeger1 | fungi: wouldn't a later merge figure this conflict out? | 13:50 |
| fungi | Ajaeger1: it would | 13:51 |
| *** bknudson has joined #openstack-infra | 13:51 | |
| *** bodepd_ has quit IRC | 13:51 | |
| trinaths | fungi: okay thanks | 13:51 |
| Ajaeger1 | Strange but there were merges to nova after the 116055 merged... | 13:51 |
| fungi | Ajaeger1: so any effort we put into finding those slightly sooner may not be particularly beneficial | 13:51 |
| *** trinaths has left #openstack-infra | 13:51 | |
| *** bodepd has joined #openstack-infra | 13:52 | |
| Ajaeger1 | fungi: I agree with the non-aborting simplification but then why it's still not marked as conflicting? Let me double check that it really conflicts... | 13:52 |
| *** r-daneel has joined #openstack-infra | 13:53 | |
| Ajaeger1 | Ah, I get: | 13:53 |
| Ajaeger1 | Applying: Don't use __add__ with oslo.i18n Message object | 13:53 |
| Ajaeger1 | Using index info to reconstruct a base tree... | 13:53 |
| Ajaeger1 | Mnova/exception.py | 13:53 |
| Ajaeger1 | Falling back to patching base and 3-way merge... | 13:53 |
| Ajaeger1 | No changes -- Patch already applied. | 13:53 |
| *** dangers_away has quit IRC | 13:53 | |
| *** blkperl has quit IRC | 13:53 | |
| *** acfleury has joined #openstack-infra | 13:54 | |
| fungi | that may just be a corner case then | 13:54 |
| fungi | Ajaeger1: i assumed the hole you identified is that zuul doesn't immediately report merge conflicts it discovers in the merge-check pipeline for changes which are already in the process of running jobs in the check pipeline (but subsequent merges, or check runs, or approval into the gate, would spot and report it) | 13:55 |
| *** amuller has joined #openstack-infra | 13:55 | |
| fungi | wow, freenode seems to be struggling again today | 13:55 |
| Ajaeger1 | fungi: That was my first impression, I needed to think this through with you. I didn't expect it would just say "Patch applied" | 13:56 |
| *** bodepd has quit IRC | 13:57 | |
| *** bodepd has joined #openstack-infra | 13:58 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Move ordereddict to requirements https://review.openstack.org/119071 | 13:58 |
| *** SpamapS has quit IRC | 13:59 | |
| *** adalbas has quit IRC | 14:02 | |
| *** eharney has joined #openstack-infra | 14:02 | |
| *** eharney has quit IRC | 14:02 | |
| *** eharney has joined #openstack-infra | 14:02 | |
| *** ildikov has joined #openstack-infra | 14:02 | |
| *** nelsnelson has joined #openstack-infra | 14:02 | |
| *** paulrad has joined #openstack-infra | 14:02 | |
| *** ildikov_ has quit IRC | 14:02 | |
| *** blkperl_ has joined #openstack-infra | 14:02 | |
| *** paulrad has quit IRC | 14:02 | |
| *** bnemec is now known as beekneemech | 14:02 | |
| *** gokrokve has joined #openstack-infra | 14:02 | |
| *** lxsli is now known as thelexx | 14:02 | |
| *** SpamapS_ has joined #openstack-infra | 14:02 | |
| *** dangers_` has joined #openstack-infra | 14:02 | |
| *** dangers_` is now known as dangers | 14:02 | |
| *** dangers has quit IRC | 14:02 | |
| *** dangers has joined #openstack-infra | 14:02 | |
| *** dangers is now known as dangers_away | 14:05 | |
| *** dangers_away is now known as dangers | 14:06 | |
| *** pkoniszewski has quit IRC | 14:06 | |
| *** dims is now known as dimsum_ | 14:12 | |
| *** kgiusti has joined #openstack-infra | 14:13 | |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Update release-tools doc to match Juno tools https://review.openstack.org/117289 | 14:15 |
| openstackgerrit | A change was merged to openstack-infra/release-tools: Support new location for oslo-incubator https://review.openstack.org/117733 | 14:15 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Openstackid.org openid instance https://review.openstack.org/119364 | 14:15 |
| sdague | fungi: so whatever new mirroring solution seems to be in place, it's definitely not working out. pip download really represents the bulk of the fails right now. | 14:17 |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Add a gate job for zaqar-redis https://review.openstack.org/119382 | 14:17 |
| *** morganfainberg has quit IRC | 14:18 | |
| *** paul-- has quit IRC | 14:18 | |
| *** jhesketh has quit IRC | 14:18 | |
| *** lcheng_ has joined #openstack-infra | 14:18 | |
| fungi | sdague: there is no new mirroring solution in place yet afaik. work is still in progress | 14:19 |
| sdague | ok | 14:19 |
| sdague | guess I missed a piece of that | 14:19 |
| sdague | my bad | 14:19 |
| *** dansmith is now known as superdan | 14:19 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Add a gate job for zaqar-redis https://review.openstack.org/119382 | 14:19 |
| fungi | sdague: also no guarantees it will help either, if network problems are internal to the providers/regions | 14:19 |
| fungi | sdague: the outstanding patches for it, which i'm starting back through now, are https://review.openstack.org/#/q/status:open+topic:mirror,n,z | 14:20 |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Uninstall puppet-common for puppet jobs https://review.openstack.org/117044 | 14:21 |
| fungi | (where "starting back through" means making new patchsets to address current test failures for them" for the most part) | 14:21 |
| *** gargola has joined #openstack-infra | 14:22 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Add a gate job for zaqar-redis https://review.openstack.org/119382 | 14:22 |
| *** zz_dimtruck is now known as dimtruck | 14:24 | |
| rajesht | Could anybody please help me understand why jenkins is failing on following patches. | 14:25 |
| *** tonytan4ever has joined #openstack-infra | 14:25 | |
| *** crinkle has joined #openstack-infra | 14:25 | |
| rajesht | I have checked and found that it is not because of changes I have made in both patches. | 14:25 |
| rajesht | https://review.openstack.org/#/c/118672/ | 14:25 |
| rajesht | https://review.openstack.org/#/c/118674 | 14:25 |
| *** david-lyle has joined #openstack-infra | 14:26 | |
| *** salv-orlando has joined #openstack-infra | 14:26 | |
| fungi | rajesht: most often, if your change doesn't introduce the issue, it's either because tests are relying on access to things over the internet (which is of course unreliable at the best of times) or because of myriad bugs in openstack | 14:26 |
| *** morganfainberg has joined #openstack-infra | 14:27 | |
| *** paul-- has joined #openstack-infra | 14:27 | |
| *** jhesketh has joined #openstack-infra | 14:27 | |
| *** mpaolino has joined #openstack-infra | 14:27 | |
| rajesht | fungi: I have checked the failed test case, it is also failing randomly on latest master also. | 14:28 |
| fungi | rajesht: also, the elastic recheck bot has posted comments on 118674 explaining the bugs it saw in the test logs for that change | 14:28 |
| rajesht | please try to check this unit test on latest master | 14:29 |
| rajesht | nova.tests.integrated.test_multiprocess_api | 14:29 |
| *** julim has quit IRC | 14:29 | |
| rajesht | fungi: I have noticed that it gets failed randomly for test_restart_sighup unit test. | 14:29 |
| rajesht | It throws the mismatch error for pids before and after sending SIGHUP signal. | 14:30 |
| fungi | rajesht: oh, if nova's unit tests are randomly failing now, then it sounds like maybe someone merged a recent patch to it which introduced a race condition or other nondeterministic bug. you might do better to compare notes with other nova developers (i'm just one of the lowly drudges who keeps the servers running and the lights on) | 14:30 |
| fungi | rajesht: possibly a discussion better fitted to #openstack-nova | 14:31 |
| rajesht | yes, I will discuss this in #openstack-nova channel. | 14:32 |
| rajesht | fungi: | 14:32 |
| rajesht | Thanks for your time. | 14:32 |
| *** paulrad has joined #openstack-infra | 14:32 | |
| mordred | sdague: I'm still working on getting the new mirroring up ... I'm failing at getting pip installed right now - so I'm about to do a bunch of things by hand | 14:33 |
| *** andreaf has quit IRC | 14:33 | |
| *** andreaf has joined #openstack-infra | 14:33 | |
| fungi | oh, if mordred is awake now and continuing to iterate on that stack of changes, i'll switch back to python 3.4 stuff | 14:35 |
| *** radez_g0n3 is now known as radez | 14:35 | |
| *** pkoniszewski has joined #openstack-infra | 14:37 | |
| *** habib has joined #openstack-infra | 14:37 | |
| *** signed8bit has joined #openstack-infra | 14:39 | |
| *** datsun180b has joined #openstack-infra | 14:42 | |
| mordred | fungi: yeah. I'm on it | 14:43 |
| *** dhellmann_ is now known as dhellmann | 14:43 | |
| *** vhoward has left #openstack-infra | 14:43 | |
| *** Ajaeger1 has quit IRC | 14:44 | |
| gilliard_ | lowly drudge! | 14:45 |
| *** jergerber has joined #openstack-infra | 14:46 | |
| fungi | gilliard_: yep, it's my lot in life ;) | 14:46 |
| *** zz_jgrimm is now known as jgrimm | 14:47 | |
| gilliard_ | I aspire to drudgery | 14:48 |
| *** pkoniszewski has joined #openstack-infra | 14:50 | |
| fungi | it definitely has its plusses | 14:53 |
| homeless | hi guys, Im having a hard time installing nodepool, I get an error related to the novaclient, for some reason its missing the tenant_networks.py and some other scripts, Im installing pip nodepool from master, so I dont know what is wrong. any thoughts? http://paste.forj.io/show/46/ | 14:53 |
| *** mrmartin has quit IRC | 14:54 | |
| anteaya | you can't go down any further than the drudge | 14:54 |
| anteaya | so falling just keeps you where you are | 14:54 |
| *** rlandy has quit IRC | 14:54 | |
| *** amotoki has joined #openstack-infra | 14:54 | |
| *** flaper87 is now known as flaper87|afk | 14:55 | |
| jeblair | homeless: we use novaclient from pypi. i wonder if the one you have might be too old? (maybe installed by the os?) | 14:55 |
| *** jlibosva has quit IRC | 14:56 | |
| homeless | could be, let me check.. thanks jeblair | 14:56 |
| *** k4n0 has quit IRC | 14:57 | |
| *** stevemar has joined #openstack-infra | 14:57 | |
| jeblair | fungi: i'm approving the image delete parallelization change to nodepool. i think when that lands, we can restart nodepool to pick up all those changes and start checking that the provider file it installs looks correct | 14:58 |
| fungi | jeblair: agreed | 14:58 |
| chmouel | mordred: was wondering if you missed to commit this file http://paste.openstack.org/raw/G11JOGr3ntF6QbO5tyHU/ | 14:59 |
| jeblair | also, it looks like there's less green on the graph which i guess is good | 14:59 |
| mordred | chmouel: yup | 15:00 |
| mordred | chmouel: pushed | 15:00 |
| chmouel | mordred: cool thanks! | 15:00 |
| *** beekneemech has quit IRC | 15:01 | |
| *** bnemec has joined #openstack-infra | 15:02 | |
| mordred | chmouel: I've also got a dox.yml for that and a dockerfile ... let me add them too | 15:02 |
| *** bnemec is now known as beekneemech | 15:03 | |
| openstackgerrit | A change was merged to openstack-infra/nodepool: Paralellize image deletes https://review.openstack.org/119208 | 15:03 |
| mordred | chmouel: I'm not sure where the 'right' place to put and organize such dockerfiles are - but for now, I just shoved them into a dockerfiles dir :) | 15:03 |
| chmouel | nice, yeah i was wondering about an example | 15:03 |
| chmouel | mordred: i like the idea, i think i have seen some other project trying to implement docker and tox together | 15:04 |
| chmouel | but it never worked | 15:04 |
| *** tsg has joined #openstack-infra | 15:05 | |
| mordred | chmouel: my thoughts so far have been that we might want a dockerfile/make thing that can take a directory tree of dockerfiles and make corresponding docker images | 15:05 |
| mordred | because right now Dockerfile assumes a one-image kinda thing, which works GREAT for their intended use case - but not for how we use tox | 15:05 |
| mordred | jeblair, fungi: best I can tell, "nova volume-attach" is async, but there is no status taht can be checked | 15:06 |
| mordred | just to add to the levels of insanity | 15:06 |
| chmouel | mordred: like having multiple layers building in dockerile? | 15:06 |
| *** acfleury has quit IRC | 15:06 | |
| *** adalbas has joined #openstack-infra | 15:07 | |
| mordred | chmouel: kinda. more that we're not really using our Dockerfile to express a _deployment_ of a thing, but rather to encapsulate one or more different possibilities that we want to test | 15:07 |
| *** esker has joined #openstack-infra | 15:07 | |
| *** yamahata has joined #openstack-infra | 15:08 | |
| *** zhiwei has joined #openstack-infra | 15:08 | |
| mordred | for instance - it would be nice to express "nova needs: apt-get install libvirt-bin zookeeper" ... but it turns out that same command is used in the py26 and the py27 containers, so we don't need a full new dockerfile for nova/py27 and nova/py26 | 15:08 |
| mordred | BUT | 15:08 |
| mordred | I might be overthinking it | 15:08 |
| *** paulrad has quit IRC | 15:09 | |
| chmouel | mordred: i think that would be very cool to be able to have that, but at first just having running unittests in a base dockerfile it would be nice to get the thing going | 15:09 |
| jroll | mordred: fwiw, I would also love to be able to deploy services on docker | 15:09 |
| mordred | also, unlike tox, we should be able to say "pep8 should run $commands, and they should run in the py27 container" | 15:09 |
| mordred | jroll: yah - the iscsi deficiency makes that harder because cinder | 15:10 |
| jroll | and I think every service should have some sort of base container | 15:10 |
| jroll | right | 15:10 |
| jroll | mordred: so things that use iscsi maybe don't run in a container | 15:10 |
| jroll | or like, one day we actually fix iscsi | 15:10 |
| mordred | yah. and then it starts getting complex :) - but yeah, I'm with you in theory | 15:10 |
| chmouel | jroll: i think that's an another work that would need to be done to get devstack intalling services directly in docker containers | 15:11 |
| chmouel | (if the iscsi issues or others get fixed) | 15:11 |
| mordred | yeah. it's a pretty big effort, and right now devstack is doing its job pretty well | 15:11 |
| mordred | so I figured just keeping the problem space focused on what tox does might be achievable | 15:11 |
| chmouel | yeah agreed! | 15:11 |
| jroll | chmouel: to be clear, I'm talking about a real deployment, not devstack :) | 15:11 |
| jroll | I do agree that tox is the first step | 15:12 |
| chmouel | jroll: ++ | 15:12 |
| *** bdpayne has joined #openstack-infra | 15:15 | |
| *** Guest18621 is now known as annegentle | 15:16 | |
| *** blkperl_ is now known as blkperl | 15:16 | |
| mordred | jeblair, fungi: any idea why a fresh node in hp might not be able to perform curl -O https://bootstrap.pypa.io/get-pip.py | 15:16 |
| mordred | btw | 15:17 |
| *** pelix has joined #openstack-infra | 15:17 | |
| mordred | why the HELL would I ever want this in a resolv.conf: | 15:17 |
| mordred | search openstacklocal | 15:17 |
| clarkb | same reason we fail at hitting our mirror? if pypa.io is hosted in rax or across the same routes it may fail | 15:17 |
| fungi | mordred: ... um ... what tenant? can it get to other things? | 15:17 |
| mordred | it can't resolv bootstrap.pypa.io best I can check | 15:18 |
| fungi | mordred: intermittent or consistent? | 15:18 |
| mordred | clarkb: this is me still trying to create mirror nodes | 15:18 |
| fungi | ahh, dns problems? | 15:18 |
| mordred | fungi: consistent since last night | 15:18 |
| fungi | weird | 15:18 |
| mordred | clarkb: OH | 15:18 |
| mordred | clarkb: you did security-groups there ... | 15:18 |
| fungi | have you tried comparing dns responses it gets from the default nameservers vs others? | 15:18 |
| clarkb | oh did I faul | 15:18 |
| mordred | clarkb: is it possible you didn't do egress rules? | 15:18 |
| clarkb | mordred it is possible. actually I may have derped udp :( sorry | 15:19 |
| clarkb | why cant we just do allow everything? | 15:19 |
| *** Sincler has joined #openstack-infra | 15:19 | |
| clarkb | every other firewall in the world has an allow ip any any equivalent | 15:20 |
| *** zhiwei has left #openstack-infra | 15:20 | |
| jeblair | clarkb: if we send traffic outbound, haxors will know where we are | 15:20 |
| fungi | because that would be "unsafe" or some such nonsense, is i'm sure the questionable reasoning behind it | 15:20 |
| * mordred stabs openstack in the face | 15:20 | |
| clarkb | mordred you fixing or should I really wake up? | 15:21 |
| jeblair | mordred: are you the one who sliced up the openstack logo? | 15:21 |
| jroll | fungi: whyyy | 15:21 |
| jroll | who is it that wants these firewalled | 15:21 |
| jroll | ? | 15:21 |
| mordred | jroll: I have NO IDAE | 15:21 |
| mordred | clarkb: I'm fixing | 15:21 |
| clarkb | jroll neutron | 15:21 |
| jroll | wat | 15:22 |
| clarkb | jroll you get a security group called default | 15:22 |
| clarkb | whether you want it or not | 15:22 |
| jroll | oh, so this is just HP's cloud? | 15:22 |
| mordred | clarkb: did you see that nova volume-attach is not immedaite, but that there is no status I can see to track | 15:22 |
| clarkb | and it is default deny best I can tell | 15:22 |
| *** pkoniszewski has quit IRC | 15:23 | |
| clarkb | mordred no, is thst why you cant list name? | 15:23 |
| jroll | clarkb: so uh, what's wrong with using a security group that is default allow | 15:23 |
| clarkb | thats :( | 15:23 |
| clarkb | jroll nothing but you need like 20 rules for that | 15:23 |
| mordred | clarkb: well, it means in rackspace that when I get to mount volume, the volume still isn't there because $bonghits | 15:23 |
| anteaya | this email poster is having problems: jranjan@gmail.com | 15:23 |
| jroll | clarkb: ughhh | 15:23 |
| anteaya | either he can't figure out his email client or his is spamming the list | 15:23 |
| clarkb | one each for ingress and egress udp tcp over ipv4 then ipv6 and finaly icmp | 15:24 |
| jroll | clarkb: this is the first time not having security groups at rackspace makes me happy | 15:24 |
| clarkb | jroll and I derped udp | 15:24 |
| jroll | heh | 15:24 |
| jroll | gotcha | 15:24 |
| jroll | I just thought like... someone was telling us we need to firewall nodepool instances for some silly reason | 15:24 |
| *** tomoe_ has quit IRC | 15:24 | |
| *** gokrokve_ has joined #openstack-infra | 15:24 | |
| clarkb | they are firewalled with local iptables rules | 15:25 |
| fungi | clarkb: i suspect the default security group nonsense is feature parity with hpcloud 1.0 | 15:25 |
| fungi | clarkb: which, if memory serves, also blocked everything by default and required you to add a slew of rules to allow all traffic | 15:26 |
| clarkb | yup you basically got ssh inbound | 15:26 |
| *** Longgeek_ has quit IRC | 15:27 | |
| mordred | ok. | 15:27 |
| mordred | new strategy | 15:28 |
| mordred | every time I hit something like this, I'm emailing the mailing list and I'm not going to be polite | 15:28 |
| mordred | security groups have now been covered | 15:28 |
| *** gokrokve has quit IRC | 15:28 | |
| *** gokrokve_ has quit IRC | 15:29 | |
| *** bdpayne has quit IRC | 15:29 | |
| *** alexpilotti has quit IRC | 15:30 | |
| *** koolhead17 has joined #openstack-infra | 15:30 | |
| *** koolhead17 has joined #openstack-infra | 15:30 | |
| *** Longgeek has joined #openstack-infra | 15:31 | |
| jroll | heh | 15:31 |
| *** mmaglana has joined #openstack-infra | 15:31 | |
| *** esker has quit IRC | 15:35 | |
| chmouel | mordred: what kind of docker version do you have? I am gttign that with the namespace you are setting by default in dox http://paste.openstack.org/raw/Vw34qo5rkBJK9ZHhTJU8/ | 15:35 |
| *** habib has quit IRC | 15:35 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 15:36 | |
| mordred | chmouel: docker 0.9.1 ? | 15:36 |
| chmouel | mordred: ah i have 1.1.2 | 15:36 |
| *** dtantsur is now known as dtantsur|afk | 15:36 | |
| mordred | oh! excellent. I should upgrade then :) | 15:36 |
| chmouel | heh | 15:36 |
| mordred | WOW | 15:37 |
| mordred | rackdns is a pile of crap | 15:37 |
| * jroll hides | 15:37 | |
| mordred | why is server-href required? | 15:38 |
| *** packet has joined #openstack-infra | 15:38 | |
| mordred | turns out I might want to point DNS at an IP address that is not a server in that cloud | 15:38 |
| *** gokrokve has joined #openstack-infra | 15:38 | |
| jroll | ffs | 15:38 |
| jroll | really? | 15:38 |
| mordred | yah | 15:38 |
| chmouel | mordred: that comes from slicehost if i recall correctly | 15:38 |
| * jroll grumbles | 15:38 | |
| jroll | mordred: great neutron email, though | 15:38 |
| chmouel | they got the api from slicehost on rax cloud straight | 15:38 |
| mordred | also, it does not like https://region-b.geo-1.compute.hpcloudsvc.com/v2/10871533237286/servers/03c614e6-855a-4d0d-9128-d7e9d7d3adb9 as a server-href :) | 15:39 |
| jroll | lol | 15:39 |
| mordred | which is silly | 15:39 |
| mordred | since that's a URL | 15:39 |
| jroll | it doesn't have rackspace.com in the name, duh | 15:39 |
| mordred | oh, right. I'm sorry. I forgot. I'm using the rackspace Open Cloud ... not OpenStack | 15:39 |
| jroll | I need to stay out of this channel right now, going to end up flipping tables :| | 15:39 |
| * mordred is VERY snarky this morning | 15:39 | |
| jeblair | jroll: mordred has already flipped them upside down, you'll just be righting them | 15:40 |
| jroll | jeblair: heh, or booting them across the room | 15:40 |
| jroll | mordred: happy to pass things along internally | 15:40 |
| jeblair | mordred: i guess we get to use the web interface for that? | 15:40 |
| *** maroon has quit IRC | 15:41 | |
| *** slagle_ is now known as slagle | 15:41 | |
| jeblair | mordred: oh, wait, you're doing rdns | 15:41 |
| jeblair | mordred: you don't need server-href for forward dns | 15:41 |
| jroll | I can make dns entries in our UI without a server-href | 15:41 |
| jroll | or without pointing them at a rackspace server | 15:41 |
| *** markmcclain has joined #openstack-infra | 15:41 | |
| * jroll looks at api calls | 15:41 | |
| jeblair | jroll: ^ i'm pretty sure he was looking at the reverse dns commands | 15:42 |
| jroll | ah | 15:42 |
| mordred | OOOOH | 15:42 |
| *** yamahata has quit IRC | 15:42 | |
| mordred | wait | 15:42 |
| mordred | that was reverse | 15:42 |
| * mordred hides | 15:42 | |
| *** yamahata has joined #openstack-infra | 15:42 | |
| jeblair | mordred: instead you could get mad at hpcloud for having no facility to set reverse dns :) | 15:42 |
| jroll | oh | 15:42 |
| fungi | mordred: rackdns record-create --name zm04.openstack.org --type AAAA --data 2001:4800:7817:101:be76:4eff:fe04:45ba --ttl 3600 openstack.org | 15:43 |
| jroll | yeah, thsoe are tied to servers | 15:43 |
| fungi | mordred: no server needed | 15:43 |
| jroll | (dunno why) | 15:43 |
| mordred | Kiall: ^^ when am I getting reverse dns in HP Cloud? | 15:43 |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 15:43 | |
| * mordred throws wet cat at Kiall | 15:43 | |
| *** flaper87|afk is now known as flaper87 | 15:43 | |
| clarkb | except dont AAAA hpcloud IPs | 15:43 |
| mordred | Kiall: when am I getting ipv6 in hpcloud? | 15:43 |
| jroll | mordred: I'm still happy to yell at our dns team | 15:43 |
| fungi | clarkb: yeah, fair point ;) | 15:43 |
| mordred | jroll: yell at the for good measure | 15:43 |
| Kiall | mordred: pull requests welcome ;) | 15:43 |
| mordred | jroll: and ask when I'm getting designate at rackspace | 15:44 |
| Kiall | Kidding.. Not currently sure when we plan to deploy those... | 15:44 |
| jroll | mordred: you have no idea how much good measure I have | 15:44 |
| mordred | hehe | 15:44 |
| fungi | jroll: if you just feel like yelling at people over dns stupidity, i have a list of poor implementation decisions i'm happy to dig back up and let you pass along | 15:44 |
| *** lttrl has joined #openstack-infra | 15:44 | |
| Kiall | mordred: That said... If you have a need today, ping me on HipChat and we can more than likely work something out ;) | 15:44 |
| fungi | (as someone who designed systems to manage mass dns hosting at an isp for years) | 15:44 |
| *** amuller has quit IRC | 15:45 | |
| jroll | fungi: jim@jimrollenhagen.com | 15:45 |
| clarkb | jroll did you watch the game last night? | 15:45 |
| mordred | Kiall: nah. just grousing about the fact taht I can't do the same things in both clouds | 15:45 |
| jroll | clarkb: did not, haven't activated cable yet | 15:45 |
| jroll | clarkb: I did hear the result :P | 15:45 |
| *** markmcclain1 has joined #openstack-infra | 15:45 | |
| Kiall | mordred: yea, I figured as much... ;) | 15:45 |
| *** r1chardj0n3s has left #openstack-infra | 15:45 | |
| mordred | jeblair, clarkb, fungi: pypi.region-b.geo-1.openstack.org exists and is in dns. it's currently running initial bandersnatch sync | 15:46 |
| *** markmcclain has quit IRC | 15:46 | |
| mordred | which would go smoother if we had AFS | 15:46 |
| mordred | clarkb: I fixed the cinder race by adding a sleep 120, btw | 15:46 |
| fungi | mordred: indeed. exciting nevertheless | 15:46 |
| clarkb | mordred awesome | 15:47 |
| * fungi sighs at arbitrary sleep durations... | 15:47 | |
| mordred | pypi.dfw.openstack.org exists, bandersnatch is not yet running | 15:47 |
| fungi | i like that novaclient has a "block until it's ready" option for boot. clearly attach needs the same | 15:47 |
| mordred | I wonder, should we just rsync instead? | 15:47 |
| mordred | fungi: EVERYTHING needs a synchronous mode | 15:47 |
| clarkb | mordred I think exsrcising bandersnatch is good | 15:48 |
| mordred | because sometimes what you're doing makes more sense | 15:48 |
| mordred | clarkb: kk | 15:48 |
| *** Longgeek has quit IRC | 15:48 | |
| clarkb | basically proves it should continue to work | 15:48 |
| jeblair | i'm going to restart nodepool now | 15:48 |
| fungi | mordred: i'm with clarkb: having bandersnatch in one place plus rsync all over probably means longer periods of out-of-sync than running separate bandersnatches (modulo hang bugs we've been running into so far) | 15:48 |
| *** gokrokve has quit IRC | 15:49 | |
| *** gyee has joined #openstack-infra | 15:49 | |
| *** jistr has quit IRC | 15:49 | |
| mordred | fungi: oh, I just meant the initial sync | 15:49 |
| *** gokrokve has joined #openstack-infra | 15:50 | |
| fungi | mordred: ahh, well it's probably the same bandwidth utilization either way since it's basically a full copy. also you'd need to stop bandersnatch from running at the source side and preserve the journal | 15:50 |
| fungi | or i guess multiple rsyncs | 15:51 |
| fungi | meh | 15:51 |
| mordred | yeah. I'll just do clarkb's thing | 15:52 |
| fungi | 2014-09-05 15:49:16,044 DEBUG nodepool.NodePool: Loading configuration | 15:53 |
| fungi | no exceptions thrown at least | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add entry for new PyPI mirrors https://review.openstack.org/119129 | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add option to keep failed server https://review.openstack.org/119233 | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove references to old volume from fstab https://review.openstack.org/119230 | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add a floating IP if needed https://review.openstack.org/119231 | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Split pypi_mirror into its own class https://review.openstack.org/119124 | 15:53 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Add support to launch node for attaching volumes https://review.openstack.org/119143 | 15:53 |
| mordred | ok. the stack has been successful in rax and hp now | 15:53 |
| *** unicell has quit IRC | 15:53 | |
| fungi | hrm, it reloaded configuration 10 seconds later... | 15:54 |
| *** gokrokve has quit IRC | 15:54 | |
| fungi | jeblair: did you modify nodepool.yaml just before restarting it? | 15:55 |
| *** yfried_ has quit IRC | 15:55 | |
| fungi | :q | 15:55 |
| clarkb | mordred iad and ord too? | 15:55 |
| fungi | heh | 15:55 |
| jeblair | fungi: no, it loads its configuration every time through the loop | 15:55 |
| mordred | clarkb: doing iad now | 15:55 |
| mordred | clarkb: ord next | 15:55 |
| jeblair | fungi: it's pretty dumb :) | 15:55 |
| fungi | jeblair: heh | 15:55 |
| openstackgerrit | Craig Bryant proposed a change to openstack-infra/config: Add new project monasca-statsd https://review.openstack.org/119403 | 15:55 |
| jeblair | fungi: but then it compares to the previous cycle and only shuts down/creates providers, etc, if something changed | 15:56 |
| fungi | anyway, i guess the restart was actually 15:49:26,335 in the debug log in that case | 15:56 |
| fungi | since it's immediately followed by creating new provider managers | 15:56 |
| clarkb | we should inotify that file and make it smart if we worry about it | 15:57 |
| fungi | anyway, the thread creation and demand calculations look sane | 15:58 |
| *** arxcruz has quit IRC | 15:58 | |
| reed | jeblair, gertty makes doing reviews a lot more fun. I didn't realize how much gerrit UI is confusing until I tried doing some reviews on gertty. thank you | 15:59 |
| pleia2 | yay | 15:59 |
| fungi | reed: it's making me wish i hadn't recycled that truck load of wyse 60 serial terminals now ;) | 16:00 |
| reed | fungi, I'm sure they're still where you left them :) | 16:01 |
| fungi | amber text was so easy on the eyes, even if it didn't have the ti-928 greenscreen chic | 16:01 |
| *** Guest42286 has quit IRC | 16:01 | |
| * krotscheck is having trouble finding the correct backwards-compatible way to invoke super.__init__, anyone have a pointer that google doesn’t have? | 16:04 | |
| mordred | krotscheck: how backwards compatible? | 16:04 |
| *** bdpayne has joined #openstack-infra | 16:04 | |
| krotscheck | mordred: Python 2/3 | 16:05 |
| *** habib has joined #openstack-infra | 16:05 | |
| mordred | krotscheck: it should go something like: class Foo(ParentClass): def __init__(self): super(Foo, self).__init__() | 16:05 |
| clarkb | super(MyClass, self).__init___() is probably what you want unless you are mordred and hacking on distutils internals | 16:05 |
| reed | oh, I crashed it :) | 16:05 |
| reed | UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128) | 16:05 |
| krotscheck | Awesome | 16:05 |
| krotscheck | Thanks | 16:05 |
| *** Longgeek has joined #openstack-infra | 16:06 | |
| clarkb | reed: was that on a specific change? with potential utf8 characters in the diff or commit message? | 16:06 |
| reed | clarkb, yes, Iet me pull it | 16:06 |
| clarkb | reed: having a specific example may help debug where the bad string conversion is | 16:06 |
| reed | clarkb, this one: https://review.openstack.org/#/c/119302 | 16:07 |
| *** Longgeek has quit IRC | 16:07 | |
| jeblair | reed: confirmed! :) | 16:07 |
| reed | oh, firefox is choking on it too Script: https://review.openstack.org/gerrit_ui/89B6B1C2255AB537598BF6FC8F36BB0D.cache.html line 149 > eval:16 | 16:07 |
| krotscheck | Ahh, had an old style class in my inheritance, that’s why it was yelling at me. | 16:08 |
| reed | I can't see the diff even on firefox | 16:08 |
| jeblair | i was really sloppy with unicode stuff in gertty; it wouldn't surprise me if there are real problems there | 16:09 |
| clarkb | yup firefox doesn't like it here either | 16:09 |
| clarkb | chromium seemed to do ok (not surprising as I think gerrit is mostly tested with chrom* | 16:09 |
| *** packet has quit IRC | 16:09 | |
| *** dims_ has joined #openstack-infra | 16:09 | |
| *** julim has joined #openstack-infra | 16:09 | |
| *** dims_ has quit IRC | 16:09 | |
| *** e0ne has quit IRC | 16:10 | |
| fungi | i'm able to pull it up in ff eventually, just takes it a little while | 16:10 |
| *** dims_ has joined #openstack-infra | 16:10 | |
| *** dims_ has quit IRC | 16:10 | |
| mordred | jeblair: ok. I'm VERY confused now | 16:10 |
| *** isviridov is now known as isviridov_away | 16:10 | |
| fungi | i can actually render https://review.openstack.org/#/c/119302/1/etc/default_data.json in ff24 (well, iceweasel 24.4.0) | 16:11 |
| mordred | jeblair: I created all three rackspace servers | 16:11 |
| *** david-lyle has quit IRC | 16:11 | |
| mordred | but only the dfw one shows up in nova list | 16:11 |
| *** yjiang5 has joined #openstack-infra | 16:11 | |
| mordred | nova list in IAD is blank | 16:11 |
| mordred | as is nova list in ORD | 16:11 |
| mordred | BUT | 16:11 |
| mordred | I can ssh into them | 16:11 |
| clarkb | mordred: did you change your region in your env var when listing properly? | 16:11 |
| mordred | yes | 16:11 |
| *** dims has joined #openstack-infra | 16:11 | |
| *** dims has quit IRC | 16:11 | |
| jeblair | wow | 16:11 |
| *** alexpilotti has joined #openstack-infra | 16:11 | |
| clarkb | kk that trips me up all the time | 16:12 |
| mordred | root@ci-puppetmaster:~# export OS_REGION_NAME=IAD | 16:12 |
| mordred | root@ci-puppetmaster:~# nova list | 16:12 |
| mordred | +----+------+--------+------------+-------------+----------+ | 16:12 |
| mordred | | ID | Name | Status | Task State | Power State | Networks | | 16:12 |
| mordred | +----+------+--------+------------+-------------+----------+ | 16:12 |
| mordred | +----+------+--------+------------+-------------+----------+ | 16:12 |
| *** jpeeler has quit IRC | 16:12 | |
| mordred | I cannot find the servers in nova list in any region on any account we ahve | 16:12 |
| *** dims_ has joined #openstack-infra | 16:13 | |
| fungi | mordred: not for me, on ci-puppetmaster sourcing ~root/ci-launch/openstackjenkins-rax-iad.sh and using the novaclient 2.18.1 installed there | 16:13 |
| clarkb | oh they shouldn't be in the jenkins account | 16:13 |
| clarkb | right? | 16:13 |
| mordred | fungi: do you see pypi*openstack.org in that account? | 16:13 |
| fungi | ahh, yeah openstackci | 16:13 |
| * mordred has been using the openstackci account | 16:13 | |
| *** atiwari has joined #openstack-infra | 16:13 | |
| *** dimsum_ has quit IRC | 16:14 | |
| jeblair | it's the unicode quotes in the commit message that are causing it | 16:14 |
| *** dims_ is now known as dimsum_ | 16:14 | |
| clarkb | jeblair: in the author field? | 16:14 |
| jeblair | clarkb: yeah | 16:14 |
| clarkb | that is an interesting place for them | 16:14 |
| fungi | mordred: confirmed, empty | 16:14 |
| *** dimsum_ has quit IRC | 16:14 | |
| zaro | morning | 16:14 |
| jeblair | yeah, not sure that's rfc822 kosher :) | 16:15 |
| *** jpeeler has joined #openstack-infra | 16:15 | |
| *** jpeeler has quit IRC | 16:15 | |
| *** jpeeler has joined #openstack-infra | 16:15 | |
| fungi | jeblair: possibly not even rfc2822 kosher | 16:15 |
| jeblair | fungi: yeah but anything goes in 5322 so | 16:16 |
| *** dimsum_ has joined #openstack-infra | 16:16 | |
| clarkb | mordred: did the reverse ptr dns creation succeed? | 16:17 |
| clarkb | mordred: was dns able to assert the server existed for that? :) | 16:17 |
| mordred | chmouel, jroll: I just made #dox | 16:17 |
| clarkb | definitely weird | 16:17 |
| *** dimsum_ has quit IRC | 16:17 | |
| *** esker has joined #openstack-infra | 16:17 | |
| mordred | clarkb: yes, it succeeded | 16:17 |
| clarkb | ya dig confirms | 16:17 |
| *** habib has quit IRC | 16:18 | |
| *** mordred has quit IRC | 16:18 | |
| *** mordred has joined #openstack-infra | 16:18 | |
| clarkb | and there are packages there | 16:18 |
| clarkb | odd | 16:18 |
| fungi | mordred: the rackspace dashboard knows of no servers for our openstackci tenant in iad | 16:19 |
| fungi | and lists ci-backup-rs-ord as the only instance in ord | 16:19 |
| *** rajesht has quit IRC | 16:19 | |
| jeblair | so i think python git may be storing unicode strings in str() objects | 16:19 |
| jeblair | i think i can run unicode(author.email, 'utf8') on them and get real unicode strings out of them | 16:19 |
| *** packet has joined #openstack-infra | 16:19 | |
| fungi | mordred: which is also oddly not showing up in nova list | 16:19 |
| *** marcoemorais has joined #openstack-infra | 16:20 | |
| mordred | fungi: wow. that's AMAZING | 16:20 |
| mordred | jroll: you want to be our new person to bother about rackspace cloud servers ... pvo doesn't hang out in chanel much anymore | 16:20 |
| fungi | jroll: ^ any chance you have visibility into what's going wrong there? | 16:20 |
| fungi | well, phschwartz has been excellent, but he's on a much-deserved vacation this week | 16:21 |
| fungi | from what i gather | 16:21 |
| jeblair | oh, actually gitpython returns unicode strings for _names_ just not for _emails_, which to be honest, is entirely sensible | 16:21 |
| clarkb | at some disney * think | 16:21 |
| clarkb | *thing | 16:22 |
| fungi | temple of the rodent, yes | 16:22 |
| clarkb | jeblair: huh | 16:22 |
| fungi | mordred: best wild guess is that openstackci was created when there could be only one region, and never got correctly expanded to the new regions which followed? | 16:22 |
| clarkb | jeblair: does it make sense to force to unicode on all of those fields regardless? | 16:22 |
| *** praneshp has joined #openstack-infra | 16:22 | |
| clarkb | jeblair: tricky bit is that you don't have to put utf8 unicode in git | 16:23 |
| clarkb | so doing that may not fix all the problems without more smarts | 16:23 |
| mordred | fungi: maybe so - I mean, it's VERY strange | 16:23 |
| fungi | clarkb: jeblair: from a belt-and-suspenders "be liberal in what you accept" perspective, it should probably work around it even if it's not 100% legit | 16:23 |
| JayF | mordred: fungi: Those of us in OnMetal don't have a lot of insight into the Compute side... but I'll do anything I can to help? | 16:23 |
| clarkb | and we are sure it isn't in eg the openstack account or openstack jenkins? | 16:23 |
| jeblair | clarkb: well, i think at the moment we're assuming ascii, so assuming utf8 is probably no worse and slightly better | 16:23 |
| mordred | JayF: well, tl;dr - I created a server in iad and one in ord | 16:24 |
| *** leakypipes has quit IRC | 16:24 | |
| mordred | they both exist, but neither show up in nova list | 16:24 |
| jeblair | clarkb: but agreed; a complete solution is probably "try a bunch of things and then fall back on some error-prone replacement" | 16:24 |
| *** pballand has joined #openstack-infra | 16:24 | |
| fungi | JayF: well, we can make with the clicky-clicky and open a trouble ticket about it. just sometimes people in irc here happen to have faster answers and are mildly less fanatical ;) | 16:24 |
| *** mpaolino has quit IRC | 16:24 | |
| JayF | mordred: do you have the instance uuids? | 16:24 |
| mordred | jeblair: maybe fall back to the string "Very Weird Username" | 16:24 |
| *** dims has joined #openstack-infra | 16:24 | |
| *** dims has quit IRC | 16:24 | |
| clarkb | mordred: should be in your scrollback near where launch node tells you how to dns | 16:25 |
| mordred | JayF: the one in ord is a3066c4f-bc4e-4a63-83f3-25e952f33e39 and the one in iad is f17d5307-d2cd-42d6-90a2-9e7c8a9783fe | 16:25 |
| clarkb | oh what if you nova show that uuid? does that pop up? | 16:25 |
| mordred | JayF: rackdns rdns-create worked properly | 16:25 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Handle unicode emails in git commits https://review.openstack.org/119410 | 16:25 |
| jeblair | reed: ^ fixes that | 16:25 |
| mordred | clarkb: nope. | 16:25 |
| mordred | JayF: at least, I'm assuming the uuid at the end of the server href is the uuid | 16:26 |
| mordred | https://iad.servers.api.rackspacecloud.com/v2/506635/servers/f17d5307-d2cd-42d6-90a2-9e7c8a9783fe | 16:26 |
| mordred | for instance | 16:26 |
| fungi | ERROR (CommandError): No server with a name or ID of 'a3066c4f-bc4e-4a63-83f3-25e952f33e39' exists. | 16:26 |
| fungi | yeah, no such luck | 16:26 |
| clarkb | free servers for everyone! :) | 16:27 |
| mordred | fwiw, there are also a cinder volume in each place which also don't show up | 16:27 |
| JayF | yeah sadly there appears to be an issue with our internal tool to troubleshoot this stuff, so give me a sec to continue deeper into the rabbithole | 16:27 |
| clarkb | mordred: and it really isn't in one of the other two accounts? | 16:27 |
| fungi | mordred: ahh, yeah you're right. no block devices showing for those | 16:27 |
| mordred | clarkb: I checked. nothing | 16:28 |
| fungi | mordred: i double-checked, you're wrong | 16:28 |
| fungi | a3066c4f-bc4e-4a63-83f3-25e952f33e39 | pypi.ord.openstack.org | ACTIVE | - | Running | public=162.242.166.23, 2001:4801:7825:104:be76:4eff:fe11:b12f; private=10.210.6.210 | 16:28 |
| *** flaper87 is now known as flaper87|afk | 16:28 | |
| fungi | mordred: it landed in the openstack tenant, not openstackci | 16:28 |
| * JayF relieved | 16:29 | |
| mordred | fungi: wow. really? | 16:29 |
| *** mwagner_lap has quit IRC | 16:29 | |
| fungi | mordred: yep. doesn't rule out a bug, but occam would argue it's a likely case of pebcak | 16:29 |
| mordred | fungi: there is nothing in that shell's history that sources that file | 16:29 |
| mordred | AHHHHHHHHHHHHH | 16:30 |
| mordred | the dns | 16:30 |
| mordred | it sets things | 16:30 |
| fungi | yeah, since our dns is tied to one common tenant which controls the domain | 16:31 |
| *** hdd has quit IRC | 16:31 | |
| clarkb | oh ya always do dns last and don't reuse shells | 16:31 |
| mordred | yup | 16:31 |
| mordred | ok. at least the mystery is solved | 16:31 |
| fungi | i spawn multiple screen instances in my ssh session and do one launch in each. makes it easy to copy between then | 16:31 |
| fungi | them | 16:31 |
| *** flaper87|afk is now known as flaper87 | 16:31 | |
| *** marcoemorais has quit IRC | 16:32 | |
| mordred | fungi: but wait- why isn't it showing them to me in that tenantn? | 16:32 |
| *** marcoemorais has joined #openstack-infra | 16:32 | |
| mordred | oh - I resourced. nevermind | 16:32 |
| * mordred starts over | 16:32 | |
| *** dims has joined #openstack-infra | 16:33 | |
| clarkb | it would be cool if you could reparent resources between users/tenants/projects/whatervertheabstraction is | 16:33 |
| JayF | mordred: fungi: glad to know that didn't happen. That's pretty close to the max nasty fail case | 16:33 |
| *** packet has quit IRC | 16:33 | |
| *** yamahata has quit IRC | 16:33 | |
| *** packet has joined #openstack-infra | 16:34 | |
| *** mpaolino has joined #openstack-infra | 16:35 | |
| *** luqas has quit IRC | 16:36 | |
| *** MaxV has quit IRC | 16:38 | |
| *** jp_at_hp has quit IRC | 16:41 | |
| jeblair | corvus@bare-trusty-1409926504:/etc/nodepool$ cat /etc/nodepool/provider | 16:41 |
| jeblair | NODEPOOL_PROVIDER=hpcloud-b1 | 16:41 |
| jeblair | NODEPOOL_REGION=region-b.geo-1 | 16:41 |
| jeblair | NODEPOOL_AZ=az3 | 16:41 |
| jeblair | corvus@py3k-precise-rax-iad-1954810:~$ cat /etc/nodepool/provider | 16:43 |
| jeblair | NODEPOOL_PROVIDER=rax-iad | 16:43 |
| jeblair | NODEPOOL_REGION=IAD | 16:43 |
| jeblair | NODEPOOL_AZ= | 16:43 |
| *** emagana has joined #openstack-infra | 16:44 | |
| jeblair | so i think we're ready to merge the mirror switch code as soon as all the mirrors have entries in dns | 16:44 |
| *** derekh has quit IRC | 16:45 | |
| fungi | yep, i think so. rereviewing that series now | 16:46 |
| *** Drop_Kick has joined #openstack-infra | 16:46 | |
| *** Drop_Kick has left #openstack-infra | 16:46 | |
| anteaya | ttx so at some point I will request a tag for http://git.openstack.org/cgit/openstack/governance/tree/ for the purposes of the election | 16:47 |
| anteaya | ttx when do you think the right date for that would be? | 16:47 |
| *** rushiagr is now known as rushiagr_away | 16:47 | |
| *** mpaolino has quit IRC | 16:47 | |
| fungi | anteaya: ttx: similarly, i need to tweak http://git.openstack.org/cgit/openstack-infra/config/tree/tools/atc/email_stats.py to allow for a tag parameter, so give me a good day lead before you need the rolls so i can implement and test thoroughly | 16:48 |
| *** e0ne has joined #openstack-infra | 16:48 | |
| jesusaurus | jeblair: fungi clarkb what size instances do you use for zuul and nodepool? are they primarily io-bound or are there any important cpu or memory requirements? | 16:49 |
| clarkb | jesusaurus: I think they are currently far oversided. we went with bigger nodes to tmpfs the git repos | 16:49 |
| jeblair | yeah, our zuul is too big | 16:49 |
| clarkb | jesusaurus: a 4cpu ish 8gb node is probably more than plenty | 16:50 |
| jeblair | jesusaurus: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=23 | 16:50 |
| jesusaurus | clarkb: huh, tmpfs isnt a terrible thing to do there | 16:50 |
| anteaya | fungi: sounds good | 16:50 |
| jesusaurus | jeblair: awesome thanks, i should explore those graphs more | 16:50 |
| jeblair | jesusaurus: it doesn't need it anymore because of the zuul mergers | 16:50 |
| clarkb | jesusaurus: why is that? | 16:50 |
| clarkb | jeblair: I mean with the zuul emrgers we don't need it | 16:51 |
| clarkb | gah tab complete | 16:51 |
| anteaya | fungi: I'll be calling for the rolls on September 26, 2014 after 05:59 UTC | 16:51 |
| jeblair | jesusaurus: yeah, i agree with clarkb -- we should be fine on an 8g node | 16:51 |
| clarkb | jesusaurus: it doesn't hurt but is no longer necessary. There I englished properly | 16:51 |
| fungi | jesusaurus: yeah, tmpfs doesn't really buy you anything in this case as long as the kernel is allowed to cache in available ram like it would normally want to | 16:51 |
| *** jogo is now known as flashgordon | 16:52 | |
| jeblair | jesusaurus: also, our mergers are too big; we want to put them in 4g nodes | 16:52 |
| fungi | (well, the most recent one is 4gb) | 16:52 |
| jeblair | yeah this one: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=135 | 16:52 |
| jeblair | maybe we should consider 2g nodes :) | 16:53 |
| fungi | jeblair: well, to be fair, it's making good use of the cache there | 16:53 |
| jroll | mordred: I don't really want to be the new person to bug, but I can be :P | 16:54 |
| fungi | jeblair: after all, it is serving up a lot of arbitrary stuff off disk, and having the more frequently-accessed files cached into ram could provide some performance boost | 16:54 |
| clarkb | jroll: you did so well when we found that funky hypervisor though :) | 16:54 |
| clarkb | jroll: so now tag you're it | 16:54 |
| fungi | jeblair: but yeah the response delta for that between a 2gb and 4gb instance is likely trivial | 16:54 |
| jeblair | fungi: yeah. the others are using almost 2g of memory for stack/heap; zm04 may yet grow | 16:54 |
| jroll | clarkb: :) | 16:55 |
| jeblair | fungi: lets give it some more time | 16:55 |
| jroll | pvo is in my org chain, I'll just push everything up to him :P | 16:55 |
| JayF | I think the technical term is 'grandboss' | 16:56 |
| jroll | heh | 16:56 |
| * jroll bbiab | 16:56 | |
| clarkb | mordred: any update on redoing the rax mirrors? | 16:57 |
| *** gokrokve has joined #openstack-infra | 16:58 | |
| *** _nadya_ has quit IRC | 16:58 | |
| *** david-lyle has joined #openstack-infra | 16:58 | |
| *** tonytan4ever has quit IRC | 16:59 | |
| fungi | unrelated, but do any of the former-mysqlites here know whether it notices it's using an nfs backend and changes its behavior intentionally? i've got some reports of nfs-backed virtual machines seeing significantly better performance than mounting nfs directly to the guest (with as identical of paramaters/tuning as can be managed) | 16:59 |
| mordred | clarkb: working on it | 17:00 |
| *** dguerri is now known as dguerri`afk | 17:00 | |
| jeblair | fungi: i find the lower amount of green on the node graph pleasing | 17:00 |
| fungi | jeblair: as do i | 17:00 |
| mordred | fungi: no, mysql does not behave differently | 17:00 |
| clarkb | fungi: perhaps a difference in nfs drivers? those things can be cranky particularly on linux | 17:01 |
| *** melwitt has joined #openstack-infra | 17:01 | |
| jeblair | (there's a joke in here somewhere) | 17:01 |
| fungi | mordred: okay, then it's down to blaming nfs file locking in guest rather than in host, non-monotonic clock issues, big backing caches in the hypervisor, et cetera | 17:01 |
| mordred | fungi: or network drivers in the guest | 17:01 |
| fungi | or different nfsclient implementations as clarkb suggests, yeah | 17:01 |
| *** loki184 has joined #openstack-infra | 17:01 | |
| mordred | it's a vnic, which may have somethign to do with it | 17:01 |
| mordred | but yeah | 17:01 |
| *** BobBall is now known as BobBall_AWOL | 17:02 | |
| clarkb | jeblair: use afs right? | 17:04 |
| *** bhuvan has joined #openstack-infra | 17:04 | |
| *** marcoemorais has quit IRC | 17:04 | |
| mordred | clarkb: ord done. doing iad | 17:04 |
| *** openstackgerrit has quit IRC | 17:04 | |
| *** marcoemorais has joined #openstack-infra | 17:04 | |
| fungi | clarkb: i'll pass that suggestion along ;) | 17:05 |
| *** marcoemorais has quit IRC | 17:05 | |
| *** marcoemorais has joined #openstack-infra | 17:05 | |
| *** harlowja_away is now known as harlowja | 17:05 | |
| *** jcoufal has quit IRC | 17:05 | |
| clarkb | so I must admit to having never really used afs. perhaps I should afs my home network | 17:06 |
| mordred | clarkb: afs is amazing | 17:06 |
| mordred | clarkb: but you'll need to learn some new words | 17:07 |
| *** marcoemorais has quit IRC | 17:07 | |
| clarkb | mordred: well its truly global right? | 17:07 |
| clarkb | so afs my home network is not quite right | 17:07 |
| *** marcoemorais has joined #openstack-infra | 17:07 | |
| *** cdent has quit IRC | 17:08 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:08 | |
| fungi | clarkb: join a cell in your home network to afs? | 17:08 |
| clarkb | there we go | 17:08 |
| mordred | yup | 17:09 |
| dhellmann | fungi: it looks like the updates to oslotest fixed the installation issue for python 3 locally; now to see if the jenkins slaves are happy | 17:10 |
| *** HeOS has quit IRC | 17:10 | |
| *** markmcclain1 has quit IRC | 17:10 | |
| clarkb | dhellmann: that is a new release with mox3? | 17:12 |
| clarkb | oh I see email now | 17:12 |
| *** HenryG is now known as HenryThe8th | 17:12 | |
| *** dprince has quit IRC | 17:12 | |
| dhellmann | clarkb: yep | 17:12 |
| *** ildikov_ has joined #openstack-infra | 17:12 | |
| *** jasondotstar has joined #openstack-infra | 17:14 | |
| jasondotstar | hi all | 17:14 |
| *** salv-orlando has quit IRC | 17:14 | |
| *** ildikov_ has quit IRC | 17:14 | |
| jasondotstar | is there someone here that can help me understand something re: the puppet-glance repo? | 17:14 |
| clarkb | jasondotstar: sure | 17:15 |
| *** ildikov_ has joined #openstack-infra | 17:15 | |
| *** ildikov has quit IRC | 17:15 | |
| jasondotstar | so, I'm trying to understand something | 17:15 |
| jasondotstar | I've cloned the puppet-glance module | 17:16 |
| *** loki184 has quit IRC | 17:16 | |
| jasondotstar | but I need to create a new resource | 17:16 |
| jasondotstar | to add a new variable to the glance-api.conf file | 17:16 |
| *** pelix has quit IRC | 17:17 | |
| jasondotstar | how should I use the glance_api_config resource? | 17:17 |
| jasondotstar | is it simply name => value ? | 17:17 |
| fungi | jasondotstar: you also may find better support for those modules in the #puppet-openstack channel according to https://wiki.openstack.org/wiki/Puppet-openstack#IRC | 17:17 |
| clarkb | jasondotstar: ah ok, so this is a specific qusetion about how the openstack project puppet modules do things. ^ ya that is probably what you want | 17:18 |
| jasondotstar | ya | 17:18 |
| jasondotstar | it's about how to use what looks to be a custom provider | 17:18 |
| fungi | jasondotstar: we in here run the developer community infrastructure systems, so don't do a lot of deploying openstack components themselves (devstack aside) | 17:19 |
| clarkb | jasondotstar: ya you will want to checkout the #puppet-openstack channel. We don't actually hack on puppet-glance. We support the developer infrastructure that puppet-glance consumes | 17:19 |
| clarkb | (so I don't know the answer to your question) | 17:19 |
| jasondotstar | i see. | 17:21 |
| jasondotstar | ok | 17:21 |
| jasondotstar | thx | 17:21 |
| *** shardy is now known as shardy_afk | 17:22 | |
| *** SumitNaiksatam has quit IRC | 17:22 | |
| *** dhellmann is now known as dhellmann_ | 17:23 | |
| *** tomoe_ has joined #openstack-infra | 17:25 | |
| *** esker has quit IRC | 17:25 | |
| *** tomoe_ has quit IRC | 17:29 | |
| *** baoli has quit IRC | 17:31 | |
| *** ZZelle_ has joined #openstack-infra | 17:38 | |
| *** emagana has quit IRC | 17:39 | |
| *** wenlock has joined #openstack-infra | 17:39 | |
| *** emagana has joined #openstack-infra | 17:39 | |
| flashgordon | looking for a review for https://review.openstack.org/#/c/113658/1/functions.sh which is needed to debug some gate issues | 17:40 |
| *** e0ne has quit IRC | 17:40 | |
| flashgordon | gate bug 1365625 | 17:40 |
| uvirtbot | Launchpad bug 1365625 in grenade "javelin failing in gate-grenade-dsvm-neutron gate runs with no information" [Undecided,New] https://launchpad.net/bugs/1365625 | 17:40 |
| clarkb | flashgordon: I stilll don't see the javelin log in the check tests for that change | 17:41 |
| *** e0ne has joined #openstack-infra | 17:41 | |
| jeblair | clarkb: looks like the last run was before the grenade change merged | 17:41 |
| jeblair | rechecking | 17:42 |
| flashgordon | jeblair: thanks guess I need a recheck first | 17:42 |
| flashgordon | jeblair: thanks, now I just need to wait to see if it works | 17:42 |
| *** robemcallister has joined #openstack-infra | 17:43 | |
| *** yamamoto has quit IRC | 17:43 | |
| *** emagana has quit IRC | 17:43 | |
| *** mestery is now known as man_of_mystery | 17:43 | |
| *** rushiagr_away is now known as rushiagr | 17:44 | |
| jeblair | flashgordon: i left a +2 for if the check tests come back ok | 17:44 |
| mordred | jeblair, clarkb, fungi: bandersnatch is running on the new hosts | 17:44 |
| clarkb | mordred: woot | 17:44 |
| mordred | it's on django at hp ... so I'm expecting several more hours | 17:44 |
| fungi | oops, i was gonna review those and got sidetracked by the nova-is-doomed thread | 17:45 |
| clarkb | mordred: I can hit ord and dfw but not iad yet fwiw | 17:45 |
| clarkb | fungi: you are doing the email sidetrack thing too? | 17:45 |
| *** emagana has joined #openstack-infra | 17:46 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:46 | |
| mordred | clarkb: oh! well - rackdns sure does let me add multiple entries ... | 17:46 |
| fungi | clarkb: sometimes i can't help myself | 17:46 |
| mordred | clarkb: let me go clean those up | 17:46 |
| flashgordon | jeblair: thanks | 17:46 |
| *** penguinRaider has quit IRC | 17:47 | |
| *** tonytan4ever has joined #openstack-infra | 17:47 | |
| fungi | mordred: yeah, round robin dns is a fairly basic use case, so they sort of need to | 17:47 |
| clarkb | mordred: oh that will do it (it is a valid thing to get dns round robining) | 17:47 |
| *** mwagner_lap has joined #openstack-infra | 17:47 | |
| *** andreaf has quit IRC | 17:47 | |
| *** andreaf has joined #openstack-infra | 17:48 | |
| *** david-lyle is now known as david-lyle_afk | 17:49 | |
| mordred | clarkb: okie. fixed | 17:50 |
| mordred | should propagate | 17:50 |
| *** radez is now known as radez_g0n3 | 17:51 | |
| *** e0ne has quit IRC | 17:51 | |
| *** jaypipes has joined #openstack-infra | 17:53 | |
| *** yamamoto has joined #openstack-infra | 17:54 | |
| *** robemcallister has quit IRC | 17:55 | |
| flashgordon | https://review.openstack.org/#/c/119288/ <-- fixes a gate bug in neutron unit tests and already in gate queue | 17:56 |
| flashgordon | is it possible to get that prioritized at the next reset | 17:56 |
| jeblair | flashgordon: just fyi; if you ask us to do it at the next reset, you're asking us to sit around and watch the status screen. i'd be happy to promote it, but it's going to happen fairly immediately. | 17:59 |
| *** hdd has joined #openstack-infra | 17:59 | |
| notmyname | mordred: jeblair: with gertty, all I'm seeing in a diff is the commit message. not sure if I'm doing something wrong or what | 17:59 |
| jeblair | flashgordon: i can do that one now or 40 minutes from now | 17:59 |
| flashgordon | jeblair: now is fine | 17:59 |
| clarkb | notmyname: it should be all on that single page. try scrolling down if the commit message is long? | 17:59 |
| *** yolanda_ has joined #openstack-infra | 17:59 | |
| flashgordon | jeblair: as it is passing check queue. In the future when I ask, I would be happy to babysit the status page myself | 18:00 |
| flashgordon | man_of_mystery: ^ | 18:00 |
| flashgordon | man_of_mystery: we are discussing promoting the neutrion unit test fix | 18:00 |
| man_of_mystery | flashgordon: ack | 18:00 |
| fungi | mordred: clarkb: were one of you planning to fix the pep8 issue in the mirror stack, or shall i? | 18:01 |
| jeblair | flashgordon: no problem, and we can wait sometimes. just wanted to make sure you knew the tool only promotes immediately, not on reset :) | 18:01 |
| notmyname | clarkb: yeah. I found one that works. trying to find a pattern | 18:01 |
| clarkb | fungi: I hadn't noticed yet so go for it? | 18:01 |
| man_of_mystery | flashgordon: I'm happy to babysit as well, let me know | 18:01 |
| flashgordon | jeblair: ack | 18:01 |
| mordred | fungi: oh - I didn't notice - go for it if you're in a position to | 18:01 |
| clarkb | notmyname: another thing to check is which patches yo uare diffing. hit 'p' to bring that selection up | 18:01 |
| fungi | mordred: clarkb: already had the stack checked out, so happy to. just didn't want to step on toes | 18:01 |
| *** emagana has quit IRC | 18:01 | |
| mordred | all yours! | 18:02 |
| dhellmann_ | fungi: Is there something I can read to understand the timing for when you want/need us to land those py34 changes you're proposed? I've poked at the ML archives but didn't find anything obviously related. | 18:02 |
| jeblair | notmyname: change number? | 18:02 |
| *** dhellmann_ is now known as dhellmann | 18:02 | |
| jeblair | flashgordon: man_of_mystery: done | 18:02 |
| fungi | other than that it's testing clean, just didn't want my votes to get tossed when someone eventually fixed that minor issue | 18:02 |
| *** yolanda has quit IRC | 18:02 | |
| man_of_mystery | jeblair: thanks! | 18:02 |
| *** emagana has joined #openstack-infra | 18:02 | |
| flashgordon | jeblair: thanks | 18:02 |
| notmyname | jeblair: 37781 doesn't work. 39844 does. 118812 doesn't work (yes I know the first two are old, but they came up at the top) | 18:03 |
| jeblair | notmyname: huh, both of those show code diffs for me | 18:04 |
| notmyname | jeblair: hmm | 18:04 |
| asselin__ | did the review.opensack.org key change? /usr/local/lib/python2.7/dist-packages/paramiko/client.py:517: UserWarning: Unknown ssh-rsa host key for [review.openstack.org]:29418: 28c642b744d24864c13f31d81b6e3b63 | 18:05 |
| *** julim has quit IRC | 18:05 | |
| clarkb | asselin__: not since ~april/may whenever that was | 18:06 |
| asselin__ | ok | 18:06 |
| jeblair | notmyname: the '-d' flag output copious debug info to the log file. however, there's very little debug output related to diffs, so that may not be useful. | 18:06 |
| *** che-arne has quit IRC | 18:07 | |
| notmyname | jeblair: ok thanks. I'll try that. if that doesn't work, I'll be asking how to reset it to force a clean grap from the server | 18:07 |
| *** emagana has quit IRC | 18:07 | |
| *** emagana has joined #openstack-infra | 18:07 | |
| fungi | openstackgerrit seems to have gone on walkabout. i'll see if i can bring it back to us | 18:08 |
| jeblair | notmyname: there are many commented out debug statements in gitrepo.py if you felt like spelunking. i probably should put those in and then set up log filters to scope it. (so you can do "-d diff" "-d db" etc) | 18:08 |
| *** koolhead17 has quit IRC | 18:09 | |
| fungi | seems to have exited an hour ago | 18:09 |
| fungi | 2014-09-05 17:04:52 <-- openstackgerrit (~openstack@review.openstack.org) has quit (Ping timeout: 240 seconds) | 18:09 |
| fungi | freenode has been in bad shape off and on today though | 18:09 |
| *** jgrimm is now known as zz_jgrimm | 18:09 | |
| pleia2 | early weekend for openstackgerrit | 18:09 |
| jeblair | i have to run, biab. | 18:09 |
| *** jaypipes is now known as leakypipes | 18:09 | |
| fungi | k | 18:09 |
| *** openstackgerrit has joined #openstack-infra | 18:11 | |
| fungi | welcome back, openstackgerrit | 18:11 |
| *** _nadya_ has joined #openstack-infra | 18:14 | |
| krotscheck | clarkb: Earlier this week you mentioned a pythony way of scanning for installed plugin-thingies. Is there a phrase I can google? | 18:14 |
| clarkb | krotscheck: entry points | 18:14 |
| clarkb | krotscheck: also stevedore | 18:14 |
| fungi | krotscheck: look in jenkins-job-builder source for a (massive) plugin registration implementation using entry points | 18:16 |
| notmyname | jeblair: I find it humorous that gertty's code wouldn't pass the openstack pep8/flake8 tests ;-) | 18:16 |
| krotscheck | fungi clarkb: Awesome, that’ll give me a good starting point | 18:16 |
| * krotscheck wants to make the API event stream handlers pluggable :) | 18:17 | |
| fungi | notmyname: i don't find it that humorous. it's mostly written by one author, so coding style is "whatever jeblair did" | 18:17 |
| fungi | a lot of which is probably "whatever emacs python mode did" in reality | 18:18 |
| clarkb | krotscheck: note that while those entry points are all internal to the package you can install them from outside whcih may be what you want | 18:18 |
| *** hemna__ is now known as hemna | 18:18 | |
| notmyname | fungi: sure sure. I don't mind it at all. I'm certainly not complaining | 18:18 |
| clarkb | krotscheck: it all basically works the same and you install to where you want them to go | 18:18 |
| krotscheck | clarkb: So noted. | 18:19 |
| clarkb | so I might finally be mostly caught up on mail \o/ | 18:19 |
| clarkb | one thing I noticed is that there are a lot of feature freeze exceptions... seems odd to be doing that | 18:20 |
| clarkb | but I am not on the hook for them so carry on | 18:21 |
| *** sabeen has joined #openstack-infra | 18:21 | |
| clarkb | krotscheck: JJB doesn't use stevedore but stevedore aims to make these things easy. dhellmann can probably speak more to that | 18:23 |
| *** bradjones has quit IRC | 18:24 | |
| *** man_of_mystery is now known as mestery | 18:25 | |
| *** _nadya_ has quit IRC | 18:26 | |
| *** baoli has joined #openstack-infra | 18:26 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/infra-manual: Add Working with Bugs section to Dev guide https://review.openstack.org/107741 | 18:28 |
| *** mrmartin has joined #openstack-infra | 18:29 | |
| *** Sincler has quit IRC | 18:30 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added robust message publisher and subscriber https://review.openstack.org/119266 | 18:30 |
| mordred | krotscheck: also, I believe devananda used it in Ironic - so there should be good examples there | 18:32 |
| jeblair | notmyname i will have more debug steps in a few mins. dont destroy your context. | 18:33 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added robust message publisher and subscriber https://review.openstack.org/119266 | 18:33 |
| *** ildikov_ has quit IRC | 18:33 | |
| *** tsg has quit IRC | 18:37 | |
| *** loki184 has joined #openstack-infra | 18:37 | |
| loki184 | fungi: can you please review https://review.openstack.org/#/c/118245/ | 18:39 |
| *** dkliban_afk is now known as dkliban | 18:40 | |
| fungi | loki184: looks like a repeat of 110562 which you abandoned and subsequently lost all review context. why didn't you just ask jeblair and SergeyLukjanov to confirm your updates addressed their concerns so they would lift their -2 votes on it? | 18:42 |
| *** andreaf has quit IRC | 18:44 | |
| fungi | loki184: anyway, i've approved it now | 18:44 |
| *** andreaf has joined #openstack-infra | 18:45 | |
| clarkb | mordred: I am reviewing the mirror stack again post pep8 fix. Where is the ask cinder/noav for volume device name happening or was that dropped because of the sleep? | 18:46 |
| notmyname | jeblair: ack | 18:47 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 18:47 |
| clarkb | mordred: I still don't see it but thats fine stack looks good as is but I didn't approve just in case you are still tweaking | 18:48 |
| *** e0ne has joined #openstack-infra | 18:48 | |
| jeblair | notmyname: sorry, was on phone while afk. back now. | 18:49 |
| jeblair | notmyname: https://etherpad.openstack.org/p/UtR3MCQxdH | 18:49 |
| *** e0ne has quit IRC | 18:49 | |
| notmyname | jeblair: no worries. I ate a burger :-) | 18:49 |
| jeblair | notmyname: can you run that and paste into the etherpad? | 18:49 |
| *** flaviof_zzz is now known as flaviof | 18:49 | |
| notmyname | jeblair: done | 18:50 |
| clarkb | mordred: also the rax nodes seem to have overtaken the hp node | 18:50 |
| clarkb | unless bandersnatch syncs out of order | 18:51 |
| *** bradjones has joined #openstack-infra | 18:51 | |
| *** bradjones has quit IRC | 18:52 | |
| *** bradjones has joined #openstack-infra | 18:52 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 18:52 |
| fungi | clarkb: mordred: jeblair: looks like we're at 2x +2 (or 3x, or 4x...) on the mirror stack, so assuming the initial bandersnatch runs complete some time this afternoon do we want to approve the stack and pull the switch before the weekend or save it for monday-fun-day? | 18:52 |
| clarkb | fungi: I would like to pull the switch and am happy to push/help/dowhatever | 18:52 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/infra-manual: Add Working with Bugs section to Dev guide https://review.openstack.org/107741 | 18:52 |
| fungi | clarkb: that's my feeling as well. sooner the better | 18:53 |
| jeblair | notmyname: the important values there all match mine; what do you get from "git diff a8e161ba7056cd23c1823de1c048e8ee0b16617d..fe7206cdc0089d6b451390dd1662d088a5409847" in that repo? | 18:53 |
| notmyname | jeblair: when I do a local checkout? | 18:53 |
| clarkb | fungi: and ya I think that if the mirrors sync and are happy we should merge the stack | 18:53 |
| jeblair | notmyname: no need; those gitshas should already be in the repo... sorry, i meant do that on the repo that gertty is using | 18:54 |
| jeblair | notmyname: (and if they are not there, that tells us something) | 18:54 |
| notmyname | jeblair: hmm...actually can we try a different patch set? that was the only open review on that repo and I ended up using the web UI to merge it. | 18:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Adding windows disk image builder tool for tripleo on stackforge https://review.openstack.org/118245 | 18:54 |
| clarkb | jeblair: I think the response to your question on https://review.openstack.org/#/c/106043/ confirms we don't need the fixed ips | 18:55 |
| clarkb | we just need an accessible ip | 18:55 |
| jeblair | fungi, clarkb: i think we can merge mordreds changes at will; only my chaneg to flip the switch needs to wait till all is done, right? | 18:55 |
| jeblair | notmyname: sure; 37781? | 18:55 |
| clarkb | jeblair: yup. the reason I was waiting on mordred's changes is I am not sure if he had another edit for the get volume by name thing | 18:55 |
| clarkb | we can always push that up as a new patch later too | 18:56 |
| *** amotoki has quit IRC | 18:56 | |
| jeblair | notmyname: so in swift: git diff f48465457f6bc0341818fcd50d3fbaa4381f7311..8a8499805b8b8fddd0caa88ed8a2fafb98795fff | 18:56 |
| jeblair | notmyname: and can you double check those are the values in the gertty db with that query? | 18:56 |
| jeblair | notmyname: oh that has lots of patchsets; let's go with: select r.* from revision r, change c where r.change_key=c.key and c.number=37781 and r.number=12; | 18:57 |
| mordred | clarkb: I do not. all of my edits are up | 18:57 |
| loki184 | fungi: Thank you :) | 18:58 |
| notmyname | jeblair: and yes, the git diff in the local repo has stuff (looks right) | 18:58 |
| jeblair | notmyname: how about: da5c3c6441067ee7b8b126c58c1a88a3085e36c1..84f494c0e74a1659d8ca2c1a4f83f23d18974450 | 18:58 |
| jeblair | notmyname: (sorry, my earlier git diff was for a random revision) | 18:59 |
| notmyname | jeblair: https://gist.github.com/notmyname/7ae9c298a01bc532b93c | 19:00 |
| mordred | that looks like a diff | 19:00 |
| notmyname | yup | 19:01 |
| *** HeOS has joined #openstack-infra | 19:01 | |
| jeblair | notmyname: and can you paste the output of "pip freeze" in the venv? | 19:01 |
| *** flaper87 is now known as flaper87|afk | 19:01 | |
| notmyname | jeblair: which venv? I'm not using any (unless gertty is doing something) | 19:01 |
| *** openstackgerrit has quit IRC | 19:01 | |
| jeblair | notmyname: nope; then how about: pip show gitpython gitdb | 19:01 |
| jeblair | (that will be shorter than a system-wide pip freeze and is all i really need) | 19:02 |
| *** openstackgerrit has joined #openstack-infra | 19:02 | |
| notmyname | jeblair: at the bottom of the etherpad | 19:02 |
| *** Ajaeger has joined #openstack-infra | 19:03 | |
| jeblair | notmyname: one last thing: git --version | 19:03 |
| *** packet has quit IRC | 19:03 | |
| notmyname | jeblair: 1.9.2 | 19:04 |
| *** loki184 has quit IRC | 19:04 | |
| *** loki184 has joined #openstack-infra | 19:05 | |
| *** rushiagr is now known as rushiagr_away | 19:05 | |
| jeblair | notmyname: the values in your gertty db match mine and gerrit, your local git repo has the needed commits, and you're using the same versions of the git python modules as i am. we have different versions of git (i'm on 1.7.9.5 in precise) | 19:05 |
| jeblair | anyone else happen to have git 1.9.2 installed? | 19:05 |
| mordred | jeblair: I've got 1.9.1 | 19:05 |
| fungi | jeblair: i do (debian testing) | 19:06 |
| jeblair | mordred/fungi: how does 37781 look for you? | 19:06 |
| fungi | checking | 19:06 |
| jeblair | ctrl-o then enter 37781 in case this isn't a normal workflow for you | 19:06 |
| fungi | it is, and it's fine. i can select the (unified since that's what i'm configured for) diff and scroll down through it with pgdn just fine | 19:07 |
| fungi | many, many files of diffing | 19:07 |
| mordred | notmyname: are you on a mac? (/me grasping at straws) | 19:07 |
| notmyname | mordred: yes. mac host machine for text editing, guest vms for testing | 19:08 |
| fungi | the unified diff display does get a little wonky-looking on line numbers in the thousands since it no longer has any buffer space, but other than that no observable issues | 19:08 |
| jeblair | i tried to not do anything that would break macs, but don't have one for testing | 19:08 |
| mordred | jeblair: ooh, I got a crash when I hit p on the change page taht I'd jumped to with ctrl-o, btw | 19:09 |
| jeblair | mordred: confirmed! | 19:09 |
| notmyname | jeblair: mordred: FWIW I'm also using iTerm. ctrl-o doesn't seem to do anything for me | 19:09 |
| *** packet has joined #openstack-infra | 19:09 | |
| mordred | jeblair: I've got a mac mini laying around here that I could probably put on the network and give you access to if that would be useful - it won't be in the next 10 minutes though | 19:10 |
| fungi | notmyname: command-o? O_o | 19:10 |
| jeblair | mordred: i will take you up on that offer at your convenience :) | 19:10 |
| mordred | notmyname: ctrl-o should open a text dialog and ask for you to enter a change number | 19:10 |
| mordred | or, as notmyname, maybe it's command-o or open-apple-o | 19:10 |
| notmyname | other ctrl commands for gerrty work. | 19:10 |
| notmyname | I wonder if there's somehting else intercepting it | 19:11 |
| mordred | is ctrl-o something mac or iterm normally uses? | 19:11 |
| fungi | sonuds like maybe some ui layer is trapping ctrl-o for other (nefarious) porpoises | 19:11 |
| jeblair | notmyname: hrm. well, you can rebind them in .gertty.yaml if you need to | 19:11 |
| jeblair | notmyname: see http://git.openstack.org/cgit/stackforge/gertty/tree/examples/reference-gertty.yaml#n84 | 19:12 |
| jeblair | notmyname: that would be 'change-search' for the command | 19:12 |
| notmyname | jeblair: I'm looking at patch set 74845 now. I can see a diff on patch sets 1 and 3, but not 2,4,5,6 | 19:13 |
| notmyname | jeblair: the difference is that patch sets 1 and 3 have inline comments | 19:13 |
| * mordred thinks notmyname may wind up winning the prize for most interesting gertty bug to date | 19:13 | |
| *** david-lyle_afk is now known as david-lyle | 19:14 | |
| jeblair | notmyname: i'm out of immediate ideas without more debugging info; so i think i need one or more of the following to proceed: (a) i need to make better debug logging in the diff generation and display area; (b) access to a mac to see if it something mac-ish that's causing this | 19:18 |
| Ajaeger | infra team, stevemar and myself are looking into why heat-translator is not shown up on readthedocs. | 19:18 |
| Ajaeger | https://review.openstack.org/#/c/117533/ has a patch that merges it in | 19:18 |
| notmyname | jeblair: ok, no worries. thanks for working through it with me | 19:18 |
| Ajaeger | Looking at the last commit, I see it not used in the post job: http://logs.openstack.org/72/72474adf4fd76c4f843d1d73d66e1e19654a9b35/ | 19:19 |
| Ajaeger | But it was run on jenkins02: | 19:19 |
| Ajaeger | https://jenkins02.openstack.org/view/All/job/hook-heat-translator-rtfd/1/console - without downloading anything | 19:19 |
| jeblair | notmyname: or of course if playing around in someone else's spaghetti code sounds fun, you're welcome to look into that and i'd help out where i can. gitrepo.py isn't pretty though, so i don't blame you if you have better things to do on a friday afternoon :) | 19:19 |
| notmyname | :-) | 19:19 |
| Ajaeger | do you know what's wrong here? | 19:19 |
| stevemar | thanks for recapping Ajaeger | 19:20 |
| jeblair | notmyname: that last thing you mentioned suggestes the possibility though that it's actually producing a diff but choosing not to display it for some reason (there's code that specifically ensures that diffs with comments display the sections with comments). very strange. | 19:20 |
| notmyname | jeblair: indeed | 19:20 |
| *** mwagner_lap has quit IRC | 19:20 | |
| jeblair | notmyname: on the patchsets that you do see diffs, are they correctly color-coded, etc? | 19:20 |
| clarkb | Ajaeger: it is just supposed to hit the build url which kicks it off | 19:21 |
| clarkb | Ajaeger: is the project registered and in read the docs? | 19:21 |
| jeblair | notmyname: and do you see "Expand previous 10" "Expand x lines of context" and "expand next 10" buttons? | 19:21 |
| *** robemcallister has joined #openstack-infra | 19:21 | |
| Ajaeger | stevemar: did you register the prject in read the docs? | 19:21 |
| *** mfer has quit IRC | 19:21 | |
| *** emagana has quit IRC | 19:22 | |
| stevemar | Ajaeger, nope! i wasn't aware i had to | 19:22 |
| jeblair | Ajaeger: yeah, the hook isn't even necessary; readthedocs will build projects nightly regardless of whether the hook is there; this just speeds it up | 19:22 |
| *** emagana has joined #openstack-infra | 19:22 | |
| *** stevemar has quit IRC | 19:23 | |
| Ajaeger | jeblair, clarkb : so, there's a manual step needed on readthedocs? That's not documented on http://ci.openstack.org/stackforge.html ;( | 19:23 |
| *** mrmartin has quit IRC | 19:24 | |
| *** stevemar has joined #openstack-infra | 19:24 | |
| notmyname | jeblair: interesting. no. the colors don't see correct (ie no colors). and I only see the context buttons after I scroll to the very end of the diff | 19:24 |
| stevemar | bah, just got disconnect for a minute there | 19:24 |
| jeblair | Ajaeger: readthedocs is not part of our system or workflow | 19:24 |
| Ajaeger | jeblair, clarkb: So, stevemar should register himself heat-translator on readthedocs? | 19:24 |
| notmyname | jeblair: so it's probably a rendering issue, not a logic error | 19:25 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/infra-manual: Add Working with Bugs section to Dev guide https://review.openstack.org/107741 | 19:25 |
| clarkb | Ajaeger: correct | 19:25 |
| jeblair | Ajaeger: if he wants to use it, yes. i don't have an opinion on whether he should. :) | 19:25 |
| Ajaeger | jeblair: understood, still many non-integrated projects use it... | 19:26 |
| Ajaeger | jeblair: ;) | 19:26 |
| jeblair | Ajaeger: yes, but not because we tell them to :) | 19:26 |
| *** tomoe_ has joined #openstack-infra | 19:26 | |
| jeblair | notmyname: that's starting to make me think it's a bug parsing git output | 19:26 |
| Ajaeger | stevemar: so, if you want to use it, register it and wait for a day to have it running... | 19:26 |
| Ajaeger | jeblair, clarkb: Thanks! | 19:26 |
| notmyname | jeblair: ya | 19:26 |
| stevemar | Ajaeger, alright, i'll give it a shot | 19:27 |
| *** emagana has quit IRC | 19:27 | |
| stevemar | Ajaeger, signed up, well that was painless | 19:27 |
| stevemar | thanks clarkb | 19:28 |
| Ajaeger | stevemar: never used readthedocs - so no clue ;/ | 19:28 |
| stevemar | Ajaeger, same, thanks a lot for getting back to me. | 19:28 |
| Ajaeger | stevemar: you're welcome - I learned something from it as well ;) | 19:28 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: lsb_release is missing for debian setup https://review.openstack.org/119245 | 19:29 |
| *** _nadya_ has joined #openstack-infra | 19:29 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: lsb_release is missing for debian setup https://review.openstack.org/119245 | 19:29 |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: provide an option in plugin to pre-update a plugin with puppet https://review.openstack.org/104652 | 19:30 |
| *** tomoe_ has quit IRC | 19:30 | |
| *** zz_gondoi is now known as gondoi | 19:31 | |
| *** tsg has joined #openstack-infra | 19:31 | |
| *** rfolco has quit IRC | 19:36 | |
| *** andreykurilin_ has joined #openstack-infra | 19:36 | |
| clarkb | jeblair: updated https://review.openstack.org/#/c/118924/ with a short note on the testing I did locally. I will probably do a phased rollout of that change at some point if you think it is reasonable | 19:38 |
| *** koolhead17 has joined #openstack-infra | 19:38 | |
| mordred | clarkb: you know - if I'd been forward thinking AT ALL ... I would have called these hosts mirror.$region.o.o | 19:40 |
| mordred | clarkb: and made a plan for putting apt/yum/npm mirrors on them too | 19:40 |
| mordred | clarkb: but clearly I am _NOT_ forward thinking | 19:40 |
| clarkb | mordred: thats an easy enough change when we get there :) | 19:40 |
| fungi | mordred: there's nothing stopping us from adding additional dns entries | 19:40 |
| mordred | fungi: dns is pricey | 19:40 |
| *** otherwiseguy has joined #openstack-infra | 19:41 | |
| fungi | only if you get the gold-plated ones | 19:41 |
| jroll | clarkb, mordred, looping back around, y'all can always poke me for rackspace issues and whatnot if needed | 19:41 |
| mordred | clarkb, fungi: iad seems to be finished | 19:41 |
| jeblair | notmyname: do you have any global or local git settings or env vars that alter the display at all? when you run git diff, does it colorize anything for you? | 19:41 |
| *** salv-orlando has joined #openstack-infra | 19:41 | |
| fungi | jroll: thanks for the offer! we will likely pester you incessantly about random stuff for no good reason | 19:42 |
| fungi | (at least i know i probably will) | 19:42 |
| jroll | yay, I love it when people do that! | 19:42 |
| jeblair | mordred, fungi: i'm wondering if i should go look in git-review for any "disable fancy git output stuff" we may need to apply to gertty | 19:42 |
| mordred | jeblair: ++ | 19:42 |
| notmyname | jeblair: ah, interesting. I have "color.ui=always" set. so I have to do `git diff --no-color` when eg copying to a clipboard | 19:42 |
| jroll | fungi: it's ok, I'll pester y'all back for other random things :) | 19:42 |
| fungi | jroll: we expect it | 19:42 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Add translation jobs for swift https://review.openstack.org/119454 | 19:43 |
| mordred | I'm going to guess that's it ... line 270 in gertty/gitrepo.py | 19:43 |
| clarkb | notmyname: I think 'true' is "safer | 19:44 |
| clarkb | notmyname: it will only color when writing to a terminal | 19:44 |
| fungi | i'm skimming git_review/cmd.py for any helpful ideas | 19:44 |
| mriedem | gate-oslo.serialization-python33 seems to be completely hosed, should it be made non-voting? | 19:44 |
| *** homeless has quit IRC | 19:44 | |
| mriedem | until oslotest gets it's packaging in order? | 19:44 |
| clarkb | mriedem: oslotest should have its packaging in order | 19:44 |
| clarkb | as of this morning pacific time | 19:45 |
| *** chuckC has joined #openstack-infra | 19:45 | |
| dhellmann | yeah, we thought that was fixed is it still failling? | 19:45 |
| Ajaeger | mriedem: dhellmann is on i... | 19:45 |
| mriedem | http://logs.openstack.org/54/117854/5/check/gate-oslo.serialization-python33/825493e/console.html | 19:45 |
| *** homeless has joined #openstack-infra | 19:45 | |
| mriedem | another recheck going through now | 19:45 |
| *** Ryan_Lane has quit IRC | 19:45 | |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Have git not colorize output for diffs https://review.openstack.org/119455 | 19:45 |
| clarkb | 1447utc was probably too early | 19:45 |
| mordred | jeblair, notmyname: maybe somethign liek that ^^ | 19:45 |
| mriedem | checked zuul, it passed | 19:45 |
| mriedem | clarkb: dhellmann: ^ | 19:45 |
| *** andreykurilin_ has quit IRC | 19:45 | |
| dhellmann | mriedem: so you're good? | 19:46 |
| mriedem | yeah | 19:46 |
| dhellmann | ok, good | 19:46 |
| Ajaeger | cool! | 19:46 |
| *** craigbr has quit IRC | 19:46 | |
| mriedem | clarkb: btw, over/under on harvin, 6 games - what say you? :) | 19:46 |
| *** andreykurilin_ has joined #openstack-infra | 19:46 | |
| *** craigbr has joined #openstack-infra | 19:46 | |
| clarkb | mriedem: after last night? over | 19:46 |
| mordred | oh - or git-review does --color=never | 19:46 |
| mriedem | clarkb: he's just teasing you | 19:46 |
| fungi | jeblair: mordred: --color=never | 19:46 |
| fungi | d'oh, i read too slowly | 19:47 |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Have git not colorize output for diffs https://review.openstack.org/119455 | 19:47 |
| jeblair | mordred: that'll take care of the diffstat (which appears on the change page), but we nneed to do something about line 365 for the actual diff | 19:47 |
| *** Sincler has joined #openstack-infra | 19:48 | |
| jeblair | that _probably_ means adding "color=never" as a function arg there | 19:48 |
| *** zul has quit IRC | 19:49 | |
| mordred | hrm | 19:49 |
| mriedem | dhellmann: was there a patch for the oslotest fix? | 19:50 |
| mriedem | dhellmann: i'll put that in the bug report when i close it | 19:50 |
| clarkb | mordred: dfw and iad look done now. only two to go, though the hpcloud node is slower. is pypi hosted in rackspace? that may explain it | 19:51 |
| mordred | jeblair: color='never' does not break things | 19:52 |
| mordred | jeblair: I do not know if it _fixes_ notmyname's problem | 19:52 |
| *** chuckC has quit IRC | 19:52 | |
| mordred | but simple tests show that it does not cause vomitting | 19:52 |
| *** chuckC has joined #openstack-infra | 19:53 | |
| clarkb | hpcloud is to topyc | 19:53 |
| *** emagana has joined #openstack-infra | 19:53 | |
| notmyname | mordred: give me a few minutes and I'll try to use it locally | 19:53 |
| mordred | notmyname: one more sec ... | 19:53 |
| openstackgerrit | Monty Taylor proposed a change to stackforge/gertty: Have git not colorize output for diffs https://review.openstack.org/119455 | 19:53 |
| mordred | notmyname: try taht version ^^ | 19:53 |
| * notmyname is using the gerrit web UI to patch the gerrit tty UI | 19:54 | |
| *** imcsk8 has quit IRC | 19:54 | |
| *** imcsk8 has joined #openstack-infra | 19:54 | |
| fungi | multipath code reviewing ;) | 19:55 |
| dhellmann | mriedem: https://review.openstack.org/118551 | 19:56 |
| *** _nadya_ has quit IRC | 19:56 | |
| mriedem | dhellmann: thanks | 19:56 |
| notmyname | mordred: jeblair: that works. thanks! I'll leave a comment in gerrit | 19:57 |
| mordred | woot! | 19:57 |
| *** Ryan_Lane has joined #openstack-infra | 19:57 | |
| * mordred did something marginally useful today - goes back to binge drinking | 19:57 | |
| *** emagana has quit IRC | 19:57 | |
| *** cody-somerville has quit IRC | 19:58 | |
| clarkb | mordred: starting your weekend right? | 19:59 |
| mordred | clarkb: wait, did the weekend stop at some point? | 20:00 |
| clarkb | mordred: no that must've been an illusion all this work and stuff | 20:00 |
| *** loki184 has quit IRC | 20:02 | |
| *** robemcallister has quit IRC | 20:02 | |
| openstackgerrit | Kevin Benton proposed a change to openstack-infra/elastic-recheck: Add recheck query for bug 1365829 https://review.openstack.org/119299 | 20:02 |
| uvirtbot | Launchpad bug 1365829 in neutron "ipv6 tests leave ipv6 flag disabled" [High,Fix committed] https://launchpad.net/bugs/1365829 | 20:02 |
| *** datsun180b has quit IRC | 20:02 | |
| fungi | i'm now marginally disappointed that mutt hides the ansi color escapes in my ml message, but then again perhaps i'm *glad* that it does so as well ;) | 20:03 |
| mordred | fungi: haha | 20:03 |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/infra-manual: Add Working with Bugs section to Dev guide https://review.openstack.org/107741 | 20:04 |
| *** e0ne has joined #openstack-infra | 20:04 | |
| mtreinish | fungi: heh, I didn't even notice that mutt did that, I saved it locally and catted it | 20:05 |
| mtreinish | that is kinda disappointing | 20:05 |
| fungi | clarkb: (at least some of) pypi is hosted in rackspace i think. also dstufft works there | 20:05 |
| sdake | off for a vacation see folks next week \o/ | 20:05 |
| pleia2 | sdake: nice, enjoy :) | 20:06 |
| dstufft | all of pypi is hosted in rackspace, but you never hit rackspace servers directly | 20:06 |
| dstufft | only Fastly | 20:06 |
| dstufft | Fastly has some IAD servers though I think... dunno hwat DC | 20:06 |
| mordred | ooh. the hpcloud server is almost done ... | 20:06 |
| mordred | it's on ztfy.skin | 20:06 |
| fungi | mtreinish: or |more in mutt while the message is selected | 20:07 |
| clarkb | dstufft: ok, so probably either that or hpcloud network having different throttles than rax networiking | 20:07 |
| mtreinish | fungi: ooh even better, I really need to learn how to use my tools | 20:07 |
| fungi | mtreinish: everything is a hammer. that's my motto | 20:07 |
| mordred | clarkb: hp is done | 20:08 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-dev/hacking: Update H501 to check for self.__dict__ https://review.openstack.org/119460 | 20:08 |
| *** e0ne has quit IRC | 20:08 | |
| clarkb | cool I think that means we can take the plunge now? | 20:08 |
| jeblair | mordred, notmyname: woot! | 20:08 |
| clarkb | I am going to do some quick local testing before that though | 20:08 |
| mordred | clarkb: yah. all 4 should be good now | 20:08 |
| *** lttrl has quit IRC | 20:09 | |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Fixed gate-trove-functional-dsvm jobs https://review.openstack.org/105692 | 20:09 |
| clarkb | mordred: jeblair: oh and does the select mirror change need to update pydistutils.conf too? | 20:09 |
| clarkb | or hwatever that file is | 20:09 |
| clarkb | iirc it only does pip.conf | 20:09 |
| openstackgerrit | A change was merged to stackforge/gertty: Handle unicode emails in git commits https://review.openstack.org/119410 | 20:09 |
| openstackgerrit | A change was merged to stackforge/gertty: Have git not colorize output for diffs https://review.openstack.org/119455 | 20:10 |
| jeblair | clarkb: oh, i think you are right. updating. | 20:10 |
| *** e0ne has joined #openstack-infra | 20:11 | |
| mordred | clarkb: I'm gong to go ahead and land the launch-node patches | 20:11 |
| clarkb | mordred: kk | 20:11 |
| *** Ryan_Lane has quit IRC | 20:12 | |
| clarkb | I was just able to install novaclient from all 4 new mirrors | 20:12 |
| clarkb | which isn't the most thorough test but seems to indicate things are there and pip is happy | 20:13 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Configure pip with a per-region mirror https://review.openstack.org/119158 | 20:13 |
| jeblair | clarkb, mordred, fungi: now with more config file mangling | 20:13 |
| jeblair | clarkb, mordred, fungi: those are the only 2 files related to this sort of thing that devstack-gate copies to different users, so i think that should be it. | 20:13 |
| mordred | jeblair: I agree with you | 20:14 |
| clarkb | yup that should be it | 20:14 |
| *** primemin1sterp has quit IRC | 20:14 | |
| *** cody-somerville has joined #openstack-infra | 20:14 | |
| clarkb | jeblair: and nodepool uses the jenkins user for that ssh connection right? we won't need to sudo in that script | 20:14 |
| jeblair | clarkb: to the best of my knowledge, yes :) | 20:14 |
| * mordred is going to be excited about this | 20:14 | |
| openstackgerrit | A change was merged to openstack-infra/config: Update sideways Ironic job https://review.openstack.org/119166 | 20:14 |
| *** primeministerp has joined #openstack-infra | 20:15 | |
| jeblair | clarkb: at least, i thought about it and looked at things and read code and came to that conclusion. | 20:15 |
| clarkb | +2 | 20:15 |
| clarkb | jeblair: good enough for me :) | 20:15 |
| clarkb | I am pretty sure that is the case as well | 20:15 |
| jeblair | i'm really hesitant to say something like "oh definitely yes" when it comes to nodepool :) | 20:15 |
| *** koolhead17 has quit IRC | 20:15 | |
| mordred | clarkb: once this is live, should we just do round-robin dns on the four of them? or set up a 'proper' load balancer to be pypi.openstack.org ? | 20:15 |
| fungi | yeah, those are the only two i know of | 20:15 |
| clarkb | mordred: round robin is probably fine | 20:16 |
| fungi | and the recentish work to excise select-mirror has stripped all the varied and random places we used to mangle those two files | 20:16 |
| clarkb | mordred: since the jobs themselves are already smart balanced | 20:16 |
| mordred | yah. mainly just thinking there is no reason to have a pypi mirror on static.o.o anymore | 20:16 |
| clarkb | mordred: if the entry point is the pypi.o.o name I would say use a real load balancer | 20:16 |
| jeblair | we could get rid of pypi.openstack.org to drive home the idea that people shouldn't use it :) | 20:16 |
| mordred | jeblair: except _I_ use it | 20:17 |
| mordred | :) | 20:17 |
| clarkb | does rax support geo based dns load balancing | 20:17 |
| jeblair | mordred: not that you should do this, but you could benchmark the 4 and pick one :) | 20:17 |
| mordred | jeblair: mmm | 20:17 |
| mordred | 4 | 20:17 |
| jeblair | i hesitate to put a real lb in front of them, because it won't actually have the right info | 20:18 |
| jeblair | (since most of their load would bypass the lb) | 20:18 |
| mordred | that's an excellent point | 20:18 |
| fungi | jeblair: mordred: at least deleting the old "openstack" tree out from under pypi.o.o would help ferret out the remaining people who haven't already wandered in here complaining that they're having trouble installing new enough requirements | 20:18 |
| openstackgerrit | A change was merged to openstack-infra/config: Add support to launch node for attaching volumes https://review.openstack.org/119143 | 20:18 |
| mordred | fungi: I'm game to go ahead and do that | 20:18 |
| clarkb | or rename it openstack-stop-using-this | 20:19 |
| jeblair | okay, so are we set to aprv 119158? | 20:19 |
| fungi | i think so | 20:19 |
| clarkb | jeblair: I am | 20:19 |
| jeblair | done; all we need to do is wait for nodepool to get around to spinning up some precise nodes | 20:20 |
| *** tsg has quit IRC | 20:20 | |
| *** cody-somerville has quit IRC | 20:20 | |
| jeblair | if i were to guess, i'd give it an eta of an hour | 20:20 |
| clarkb | so lunchtime | 20:20 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove partial mirror from static.o.o https://review.openstack.org/119465 | 20:21 |
| clarkb | dhellmann: fwiw there is or was a zmq devstack job applied experimentally? to nova maybe | 20:21 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Remove pypi mirror from static.openstack.org https://review.openstack.org/119466 | 20:21 |
| clarkb | dhellmann: but I think it is well bitrotted | 20:21 |
| mordred | there are changes to do the old mirror removal | 20:21 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove references to old volume from fstab https://review.openstack.org/119230 | 20:22 |
| openstackgerrit | A change was merged to openstack-infra/config: Add a floating IP if needed https://review.openstack.org/119231 | 20:23 |
| openstackgerrit | A change was merged to openstack-infra/config: Add option to keep failed server https://review.openstack.org/119233 | 20:25 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Fix crash on prev/next change https://review.openstack.org/119468 | 20:26 |
| *** Ryan_Lane has joined #openstack-infra | 20:26 | |
| jeblair | mordred: ^ | 20:26 |
| *** arborism has joined #openstack-infra | 20:26 | |
| mordred | jeblair: woot | 20:27 |
| *** tsg has joined #openstack-infra | 20:28 | |
| *** koolhead17 has joined #openstack-infra | 20:29 | |
| *** zul has joined #openstack-infra | 20:29 | |
| openstackgerrit | A change was merged to openstack-infra/config: Split pypi_mirror into its own class https://review.openstack.org/119124 | 20:31 |
| *** emagana has joined #openstack-infra | 20:31 | |
| jeblair | clarkb, flashgordon: http://logs.openstack.org/58/113658/1/check/check-grenade-dsvm/11e7111/logs/javelin.txt.gz | 20:32 |
| *** Ajaeger has quit IRC | 20:33 | |
| clarkb | any idea why it doesn't have the header thing to select log level? | 20:33 |
| mordred | jeblair, clarkb, fungi: I'm considering re-naming the old pypi.openstack.org/openstack directory before we remove it, just in case there is something catastrophic we didn't catch - any objections? | 20:33 |
| jeblair | this variant of the grenade test failed: http://logs.openstack.org/58/113658/1/check/check-grenade-dsvm-partial-ncpu/c34bc31/console.html | 20:33 |
| jeblair | mordred: none | 20:33 |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack-infra/devstack-gate: Add option for building Ironic deploy ramdisks https://review.openstack.org/119473 | 20:33 |
| clarkb | mordred: nope go for it | 20:33 |
| mordred | done | 20:33 |
| *** nelsnelson has quit IRC | 20:33 | |
| clarkb | jeblair: I think that variant has been having trouble lately | 20:34 |
| clarkb | flashgordon: ^ any news on that front? | 20:34 |
| jeblair | 2014-09-05 19:35:58.120 | Expected running services not running: ,n-cond | 20:34 |
| jeblair | 2014-09-05 19:35:58.121 | + die 348 'Failure in check-sanity' | 20:34 |
| jeblair | http://logs.openstack.org/58/113658/1/check/check-grenade-dsvm-partial-ncpu/c34bc31/logs/grenade.sh.txt.gz | 20:34 |
| clarkb | random gertty feature thought. some way to turn off comments | 20:35 |
| clarkb | sometimes reading code is just easier that way | 20:35 |
| fungi | mordred: i can't imagine anything we would have missed which couldn't be fixed by a symlink to the bandersnatch mirron on static.o.o, but sure, that's fine too | 20:35 |
| jeblair | why is there no n-cpu log? | 20:35 |
| jeblair | clarkb: good idea | 20:35 |
| clarkb | jeblair: because in that job n-cpu is not restarted | 20:36 |
| clarkb | jeblair: so the only log is the log from the old/ dir | 20:36 |
| dhellmann | clarkb: yeah, the bitrot is getting smelly :-) | 20:36 |
| flashgordon | jeblair: oh the javelin jogs are in | 20:36 |
| clarkb | jeblair: that test checks that if you upgrade the rest of nova the old compute nodes will keep running so that you don't have to forklist them all at once | 20:36 |
| flashgordon | looking into the failure | 20:36 |
| fungi | clarkb: yeah, i could see a key to toggle comment visibility being super helpful actually | 20:37 |
| mordred | jeblair: do we need to put per-region mirrors in the tripleo regions too? | 20:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Add entry for new PyPI mirrors https://review.openstack.org/119129 | 20:37 |
| jeblair | mordred: i only set the ready script to run on non-tripleo nodes | 20:37 |
| mordred | kk | 20:37 |
| fungi | mordred: the only jobs running in tripleo regions are (at least currently) tripleo jobs, so we should ask them | 20:37 |
| flashgordon | clarkb fungi: that was bug 1331274 | 20:37 |
| uvirtbot | Launchpad bug 1331274 in grenade "Services not starting on new side of Grenade" [Undecided,New] https://launchpad.net/bugs/1331274 | 20:38 |
| jeblair | mordred: i guess that means we should keep pypi.o.o around for a bit because we are using it :) | 20:38 |
| flashgordon | jeblair: ^ | 20:38 |
| jeblair | flashgordon: that applies to the partial-ncpu job? | 20:38 |
| jeblair | yep apparently :) | 20:38 |
| clarkb | jeblair: ya or round robin the others as pypi.o.o | 20:39 |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack-infra/devstack-gate: Add option for building Ironic deploy ramdisks https://review.openstack.org/119473 | 20:39 |
| flashgordon | jeblair: yup any grenade thing | 20:39 |
| clarkb | ok I am +2 on javelin change. any opposition to approving? | 20:39 |
| fungi | i got the impression that tripleo jobs might be using dib-cached packages and devpi or whatever and not actually hitting our pypi mirror at all, but don't know for sure | 20:39 |
| jeblair | clarkb: go for it | 20:39 |
| clarkb | fungi: I think they do that on their laptops | 20:39 |
| clarkb | fungi: but not in their cloud | 20:39 |
| fungi | ohhh, got it | 20:39 |
| flashgordon | jeblair: you already have per region pypi mirrors set up? | 20:39 |
| jeblair | flashgordon: ah, right and the actual error this time was from n-cond, not n-cpu. | 20:40 |
| jeblair | flashgordon: so all makes sense to me now. | 20:40 |
| jeblair | flashgordon: in about an hour or so, jobs will start using them. | 20:40 |
| fungi | flashgordon: we're about to find out whether per-region mirrors are working, at any rate | 20:40 |
| clarkb | flashgordon: yes several of us hacked on it over the last day and a half and moment of truth is shortly | 20:40 |
| flashgordon | jeblair fungi: wow that was fast | 20:40 |
| mordred | jeblair: k. I just sent a mailing list message about mirrors | 20:40 |
| flashgordon | clarkb: thank you to everyone who hacked it up | 20:41 |
| flashgordon | very awesome | 20:41 |
| jeblair | flashgordon: fungi, mordred, clarkb, and i went all in on it :) | 20:41 |
| flashgordon | woot! | 20:41 |
| jeblair | flashgordon: https://review.openstack.org/#/c/119158/ is the change that flips the switch, is in check now. | 20:41 |
| jeblair | flashgordon: heh, so after 2 days of work, we'll find out if this fixes it, or if it's actually internal hpcloud network issues. :) | 20:42 |
| flashgordon | jeblair: yah I was wondering about that possibility | 20:42 |
| fungi | i'd put even money on still getting network timeouts locally to mirrors in the same region | 20:43 |
| jeblair | i think we can yell louder in that case | 20:43 |
| fungi | absolutely | 20:43 |
| clarkb | jeblair: while this doesn't prove anything I had an mtr going a while back between hpcloud and rax and their was an intermediate isp that was dropping packets on the floor | 20:43 |
| *** robemcallister has joined #openstack-infra | 20:43 | |
| clarkb | s/their/there/ | 20:43 |
| fungi | peering relationships are the pits | 20:43 |
| clarkb | so it may be a combination of all of the above :) | 20:44 |
| jeblair | clarkb: was it named 'level3'? | 20:44 |
| clarkb | no it was someone else I forget who now | 20:44 |
| mordred | oh - one fo the folks at HP has done some additional work and suggests that we should expand to 9 routers, btw | 20:44 |
| clarkb | a smaller isp | 20:44 |
| clarkb | not cogent level3 sprint etc | 20:44 |
| mordred | and additionally, he suggests that once we have 9 routers, we should coordinate with them and they'll get them spread to be one router per network node | 20:44 |
| clarkb | mordred: ~50 servers per router? | 20:45 |
| clarkb | mordred: is it that ratio that is important or locality or? | 20:45 |
| mordred | I think it's more locality | 20:45 |
| *** mfink_ has quit IRC | 20:45 | |
| fungi | sounds like isolating them per controller is | 20:45 |
| mordred | yah | 20:45 |
| jeblair | yeah, i think the ratio was more us running out of ips in a network earlier | 20:45 |
| mordred | so, I think the 'right' way to do that would be to create 9 new routers and tell them what they are | 20:46 |
| jeblair | (which of course once we knew that, we could identify the actual number of servers to put on it) | 20:46 |
| mordred | (there is downtime associated with moving a router) | 20:46 |
| mordred | then let them move them one per server | 20:46 |
| mordred | then we can change our nodepool config to use the 9 new routers | 20:46 |
| mordred | does that sound sane? | 20:46 |
| clarkb | we also need to remember that we are unbalanced across AZs. will that affect network controller location? | 20:47 |
| *** mbacchi has quit IRC | 20:47 | |
| jeblair | mordred: yeah, while you were gone we were asked to change our az balance for capacity reasons | 20:47 |
| *** miqui has quit IRC | 20:47 | |
| fungi | yeah, i think the nodepool config changes will really just be swapping out network uuids at that point | 20:48 |
| mordred | fungi: yah | 20:48 |
| mordred | jeblair: excellent | 20:48 |
| jeblair | clarkb: what was the behavior when we did not specify an az? | 20:49 |
| clarkb | also is this something we should be filing bugs against upstream? or is this particular to hp's implementation? | 20:49 |
| clarkb | jeblair: it used az2 iirc | 20:49 |
| fungi | jeblair: all nodes were launched in one az | 20:49 |
| clarkb | jeblair: it is supposed to schedule across all of them but hpcloud has hard set it instead | 20:50 |
| jeblair | oops | 20:50 |
| clarkb | at least upstream nova intends to schedule across all of them in that case but it can be overridden | 20:50 |
| clarkb | apparently this choice was made to give the user a consistent experience and az | 20:50 |
| mordred | why would the user want that? | 20:50 |
| mordred | if a user wanted a consistent az, wouldn't the user request a consistent az? | 20:50 |
| clarkb | well it sounded like fud to me but I don't know nova | 20:51 |
| *** tsg has quit IRC | 20:51 | |
| clarkb | mordred: that is what I said :) | 20:51 |
| mordred | I believe that basically nobody has any clue what anyone using a cloud actually wants | 20:51 |
| mordred | and when we tell them, they consistently tell us we're wrong | 20:51 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add recheck query for bug 1365829 https://review.openstack.org/119299 | 20:51 |
| uvirtbot | Launchpad bug 1365829 in neutron "ipv6 tests leave ipv6 flag disabled" [High,Fix committed] https://launchpad.net/bugs/1365829 | 20:51 |
| * mordred throws rotten vegetables at clouds | 20:51 | |
| * mordred is covered in rotten vegetables | 20:51 | |
| * mordred forgot that clouds are further away than his arm strength | 20:52 | |
| clarkb | speaking of rotten vegetables. I have successfully made it deep into winter playing Don't Starve | 20:52 |
| notmyname | mordred: https://c2.staticflickr.com/4/3073/2767638386_d8efc6fb1f.jpg | 20:52 |
| *** mfer has joined #openstack-infra | 20:52 | |
| clarkb | fungi: you may enjoy it if you enjoy roguelikes | 20:52 |
| mordred | notmyname: I should get that tatooed | 20:52 |
| mordred | clarkb: how rogue like? | 20:53 |
| mordred | clarkb: does it have kestrals? | 20:53 |
| openstackgerrit | Jim Rollenhagen proposed a change to openstack-infra/config: Add check job for Ironic-python-agent + Tempest https://review.openstack.org/112693 | 20:53 |
| mordred | clarkb: and/or does it use ascii art? | 20:53 |
| fungi | clarkb: i've heard that. nethack was particularly good at killing you for hanging around too long and not diving deeper to find more food. fun game mechanic | 20:53 |
| nibalizer | hey dstufft i have a pip question? why hasn't 1.6 released yet? it has a fix we desperately need | 20:53 |
| fungi | angband too, to a great extent | 20:53 |
| mordred | nibalizer: because dstufft is mad at you | 20:53 |
| fungi | angband/moria | 20:54 |
| dstufft | I personally dislike you and wish you pain! | 20:54 |
| nibalizer | mordred: :cry: | 20:54 |
| clarkb | mordred: no and no, but you are plopped into a forest and have to survive the night, then you have to survive winter | 20:54 |
| dstufft | The real answer is because I was really hoping to get PEP 440 final into it | 20:54 |
| clarkb | mordred: eventually if you manage to build yourself a kingdom you can go spelunking | 20:54 |
| clarkb | fungi: have you played mangband? | 20:54 |
| dstufft | and maybe the final version of killing off find links, though I don't know if that's going to happen | 20:54 |
| greghaynes | clarkb: did you rope mordred into dont starve? | 20:54 |
| clarkb | fungi: we may still have a server running somewhere | 20:54 |
| mordred | speaking of pep440 ... lifeless, what do you need from me on the pep440/semver stuff in pbr? | 20:54 |
| clarkb | greghaynes: knowing mordred I don't think so. but fungi may give it a go | 20:54 |
| greghaynes | haha | 20:55 |
| greghaynes | oh | 20:55 |
| greghaynes | wouldnt that effectively mean infra is in feature freeze? | 20:55 |
| clarkb | mordred: I think its mostly in there. biggest thing is how this affects how we do release for preversioned servers | 20:55 |
| clarkb | mordred: I remember reviewing and approving things that check versions and stuff | 20:55 |
| mordred | cool | 20:55 |
| openstackgerrit | James E. Blair proposed a change to stackforge/gertty: Fix another crash on prev/next change https://review.openstack.org/119476 | 20:55 |
| nibalizer | dstufft: okay, what is your guestimate on when it will be released? | 20:55 |
| fungi | clarkb: i have played zangband, but am unfamiliar with mangband | 20:55 |
| mordred | clarkb: does it really have anything to do with how we release? | 20:55 |
| clarkb | mordred: yes because it throws and exception if your preversion is wrong | 20:56 |
| mordred | but preversion is calculated | 20:56 |
| clarkb | mordred: so you have to commit and merge setup.cfg version bump first then tag previous commit | 20:56 |
| dstufft | nibalizer: I'll probably startt the process soonish, PEP 440 is finalized and we have PRs against setuptools and pip implementing it, we're just hashing out the details of doing that | 20:56 |
| mordred | clarkb: that makes no sense | 20:56 |
| *** packet has quit IRC | 20:56 | |
| clarkb | mordred: basically any commits after the preversion will say "Hey this is an error because I am newer than the specified version" | 20:56 |
| dstufft | might try to squeeze in the bandersnatch change if christian pulls in my PR and releases a new bandersnatch before 1.6 | 20:56 |
| fungi | clarkb: oh, mangband is multiplayer angband (more multiplayer than just tombstones and bones files anyway)? | 20:57 |
| mordred | clarkb: I will write a patch - I believe that is bad behavior | 20:57 |
| clarkb | mordred: so your first commit must be bump setup.cfg version | 20:57 |
| dstufft | but I have a PR for that already too, just waiting on bandersnatch to pull + release | 20:57 |
| mordred | clarkb: pbr is intended to do what you mean | 20:57 |
| mordred | clarkb: not to yell at you | 20:57 |
| clarkb | fungi: yeah you all run around in dungeons together its amazing | 20:57 |
| clarkb | mordred: thats fair too | 20:57 |
| mordred | exceptions from setup.py are always the wrong choice | 20:57 |
| mordred | but I can also wait to argue with lifeless when it's not his saturday | 20:57 |
| clarkb | mordred: that said it caught a ton of issues in our versioning so at least a warning would be good | 20:57 |
| clarkb | and yes lifeless is the person to argue with :) | 20:58 |
| mordred | well, but - so, work with me here for a second | 20:58 |
| mordred | if I have 2014.1 in setup.cfg, and commits that arent' tagged will generate pre-release versions of that | 20:58 |
| mordred | right? | 20:58 |
| nibalizer | dstufft: okay sweet, if we can help by testing the -dev branch or whatever please let me know | 20:58 |
| clarkb | right, except for any commits after the 2014.1 tag. what do they do? | 20:58 |
| clarkb | in lifelesses changes they say nope | 20:58 |
| mordred | then, if I tag a release 2014.1, that commit will still have 2014.1 in setup.cfg, so that's fine | 20:58 |
| fungi | clarkb: sounds like the network multiplayer variant of gauntlet, i forget what it was called | 20:58 |
| mordred | clarkb: so the _next_ commit to land should bump the version, yeah? | 20:59 |
| clarkb | mordred: yes | 20:59 |
| mordred | (in setup.cfg) | 20:59 |
| mordred | k. so - what's the problem? | 20:59 |
| clarkb | the problem is that we just need to do that | 21:00 |
| clarkb | and haven't been | 21:00 |
| mordred | once ttx cuts a tag, no more commits can land until the commit that bumps the version | 21:00 |
| mordred | ah | 21:00 |
| mordred | ok | 21:00 |
| mordred | sorry, I thought it was a crazy dance where ttx needed to land the version bump commit _then_ retroactively land the tag | 21:00 |
| mordred | which is crazy | 21:00 |
| clarkb | well that is the safest way to do it | 21:00 |
| mordred | it's not | 21:00 |
| clarkb | because then anyone proposing code doesn't break | 21:00 |
| clarkb | and there isn't a window of lolwtf | 21:00 |
| mordred | hrm | 21:00 |
| fungi | mordred: i think the baked-in assumption was that project with pre-versioning should never have post-versioned development snapshots or something like that, though in practice our pre-versioned projects update setup.cfg in the commit after the tag anyway so no problemo | 21:01 |
| *** robemcallister has quit IRC | 21:01 | |
| mordred | fungi: yeah - I think this is a place where we need to relax perfection | 21:01 |
| mordred | because perfection does not have value and makes things hard for people, not easier | 21:01 |
| clarkb | fungi: in practice they do not | 21:01 |
| mordred | but it doesn't matter if they do or do not | 21:01 |
| clarkb | fungi: and that is the only reason I am not arguing for removing the check | 21:01 |
| fungi | mordred: i similarly didn't see any problem with having changes after the release tag but before the setup.cfg edit or before the next pre-release tag being post-versioned development snapshot version numbers | 21:02 |
| mordred | fungi: ++ | 21:02 |
| clarkb | mordred: I think it does in that one commit can have an ambiguous version | 21:02 |
| mordred | fungi: it's incorrect, but basically, Who Cares | 21:02 |
| mordred | yup | 21:02 |
| clarkb | but I may be wrong about that. | 21:02 |
| *** craigbr has quit IRC | 21:02 | |
| mordred | that one commit will have an abiguous version | 21:02 |
| clarkb | basically if you tag 2014.1 but don't update setup.cfg what should the next version be | 21:03 |
| mordred | but that's what we get from having async patch landing combined with pre-versioning | 21:03 |
| mordred | clarkb: exactly | 21:03 |
| mordred | I'm just saying that the cost to solving that is higher than the value for solving it | 21:03 |
| clarkb | sure | 21:03 |
| jeblair | clarkb: http://logs.openstack.org/44/117044/2/check/gate-ci-docs/1d9c121/console.html | 21:03 |
| jeblair | clarkb: there's a solution to this, right? | 21:03 |
| clarkb | jeblair: thats the mirror thing right? | 21:04 |
| clarkb | oh wait sorry | 21:04 |
| jeblair | clarkb: that's the mirror derp + capitalization thing | 21:04 |
| clarkb | jeblair: ya, you get more reliable results out of pip if you name your requireemtns case sensitively | 21:04 |
| jeblair | clarkb: so we should say Sphinx in config/test-requirements? | 21:05 |
| clarkb | yup | 21:05 |
| jeblair | incoming | 21:05 |
| *** andreaf has quit IRC | 21:05 | |
| fungi | using the wrong casing of a package name causes a fallback search which means more round-trips which means greater chance to break | 21:05 |
| clarkb | jeblair: dstufft is working on fixing this but it requires a dance between pip and bandersnatch | 21:05 |
| clarkb | jeblair: but hopefully soon it will be less of an issue | 21:05 |
| fungi | and pypi | 21:05 |
| *** gokrokve has quit IRC | 21:05 | |
| fungi | where he just posted earlier this week it's done now | 21:06 |
| * mordred hands dstufft a very warm and recently blown-dry cat | 21:06 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: capitalize sphinx in test-requirements https://review.openstack.org/119477 | 21:06 |
| *** mfer has quit IRC | 21:06 | |
| dstufft | I'm waiting on Christian to review my bandersnatch PR and hopefully merge it and release it | 21:06 |
| dstufft | once he does that i'll merge my PR against pip | 21:06 |
| dstufft | pip 1.5.* won't work very well with bandersnatch prior to that change, but oh well | 21:07 |
| *** pcm_ has quit IRC | 21:08 | |
| *** dims has quit IRC | 21:08 | |
| jeblair | 119158 is in gate, waiting on precise nodes | 21:09 |
| mordred | dstufft: when you say "won't work very well" - is there something we should be bracing for? | 21:09 |
| clarkb | jeblair: I ninja approved Sphinx change | 21:10 |
| dstufft | er I had those versions wrong | 21:10 |
| dstufft | pip 1.6+ won't work very well with bandersnatch prior to the version that has my PR in it | 21:10 |
| dstufft | if you upgrade your bandersnatch before pip 1.6 releases you'll be fine | 21:10 |
| mordred | neat! | 21:10 |
| * mordred continues to hand dstufft fluffy cats in appreciation at a rate of 5 cats per second | 21:11 | |
| dstufft | (because pip 1.6 will normalize the name always, so "pip install Django" will effectively become "pip install django", which without my change on the bandersnatch side will cause the failure that you're all seeing now when you have a mistake in the caps in the name | 21:11 |
| mordred | this is gonna be a really good change when it's landed, you know | 21:12 |
| *** craigbr has joined #openstack-infra | 21:12 | |
| dstufft | but the new version of bandersnatch will work fine with old versions of pip (it has both /simple/django/ and /simple/Django/), well as fine as it does now | 21:12 |
| *** r-daneel has quit IRC | 21:12 | |
| jeblair | "the trouble with fluffy cats" | 21:12 |
| salv-orlando | clarkb, fungi, jeblair: sorry to interrupt your work. mtreinish has a patch for the top gate offender - bug 1349617. The patch is https://review.openstack.org/#/c/119268/ - as it seems we’re about to hit a gate reset, is there a possibility of considering a promotion? | 21:12 |
| uvirtbot | Launchpad bug 1349617 in tempest "SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer" [Undecided,In progress] https://launchpad.net/bugs/1349617 | 21:12 |
| *** dims has joined #openstack-infra | 21:12 | |
| mordred | jeblair: takea moment and imagine room containing 10k fluffy cats milling about | 21:13 |
| dstufft | mordred: the pypi change already fixed the "someone renamed their thing on PyPI causing a redirect loop because lol caching" problem | 21:13 |
| jeblair | mordred: this is burning man? | 21:13 |
| *** craigbr_ has joined #openstack-infra | 21:13 | |
| flashgordon | https://jenkins03.openstack.org/job/gate-tempest-dsvm-neutron-full/1317/console is about to fail | 21:13 |
| mordred | jeblair: if that's what you want burningman to be, then ye | 21:13 |
| mordred | yes | 21:13 |
| clarkb | huh its in there twice. once for check and once for gate | 21:13 |
| flashgordon | and that is for the top job in the gate | 21:13 |
| clarkb | did it already get bumped manually once? | 21:13 |
| openstackgerrit | A change was merged to openstack-infra/config: Configure pip with a per-region mirror https://review.openstack.org/119158 | 21:14 |
| clarkb | salv-orlando: I am on board with it | 21:14 |
| *** leakypipes is now known as jaypipes-afk | 21:14 | |
| jeblair | clarkb: lgtm; you going to execute? | 21:14 |
| *** gokrokve has joined #openstack-infra | 21:14 | |
| salv-orlando | clarkb: thanks. I need to go afk now - mtreinish and flashgordon are available if you have any questions. | 21:14 |
| flashgordon | clarkb: https://review.openstack.org/#/c/119268/ has not gotten bumped manually before this | 21:14 |
| clarkb | jeblair: I can yes | 21:14 |
| jeblair | salv-orlando: i'm assuming neutron-icehouse is currently always broken, and it's not that patch's fault? | 21:15 |
| jeblair | (for grenade) | 21:15 |
| clarkb | jeblair: go for it now? | 21:15 |
| mordred | jeblair: there is a non-broken version? | 21:15 |
| jeblair | clarkb: wfm | 21:15 |
| jeblair | yeah, grenade-neutron-icehous seems to be failing wherever i see it | 21:16 |
| openstackgerrit | A change was merged to openstack-infra/config: Add the tempest-lib project to openstack https://review.openstack.org/117640 | 21:16 |
| clarkb | promotion in progress | 21:16 |
| jeblair | oh! | 21:16 |
| jeblair | 21:13 < openstackgerrit> A change was merged to openstack-infra/config: Configure pip with a per-region mirror https://review.openstack.org/119158 | 21:17 |
| clarkb | ooh | 21:17 |
| jeblair | didn't see that! :) | 21:17 |
| clarkb | jeblair: so that should be in place real soon now | 21:17 |
| *** craigbr has quit IRC | 21:17 | |
| *** craigbr_ is now known as craigbr | 21:17 | |
| salv-orlando | jeblair: it seems like it. I can’t look at the failures right now, I will refer the problem to #openstack-neutron | 21:17 |
| clarkb | ~10 minutes | 21:17 |
| flashgordon | thanks! | 21:17 |
| *** craigbr has quit IRC | 21:17 | |
| clarkb | and done | 21:17 |
| clarkb | thank you for flying air zuul where throughput is our number one priority | 21:17 |
| jeblair | clarkb: i thought it was 'we eventually get you there'? | 21:18 |
| jeblair | eventual consistency is a trait i look for in an airline | 21:18 |
| *** emagana has quit IRC | 21:18 | |
| flashgordon | clarkb: ^_^ | 21:19 |
| *** mmedvede has quit IRC | 21:19 | |
| *** emagana has joined #openstack-infra | 21:19 | |
| fungi | as airline sloagans go, "you'll probably get there alive" is my favorite | 21:19 |
| *** dustins has quit IRC | 21:21 | |
| *** mwagner_lap has joined #openstack-infra | 21:21 | |
| *** bswartz has quit IRC | 21:22 | |
| dstufft | mordred: How easy would it be for you to run a bunch of tests with a forked copy of setuptools? | 21:22 |
| clarkb | dstufft: not to bad I don't think. you would just propose a change to requirements that explicitly installs that version from sdist or git or wherever | 21:23 |
| clarkb | dstufft: and since in our tox stuff we install deps first it should work | 21:23 |
| *** emagana has quit IRC | 21:23 | |
| *** jasondotstar has quit IRC | 21:24 | |
| mordred | clarkb: I thought we stopped supporting sdist lines in g-r ? | 21:24 |
| clarkb | mordred: we did if you want to merge the change | 21:24 |
| dstufft | I have a PR against setuptools that implements PEP 440, it passes all of setuptools tests, and almost all of pip's (the 3 failing tests are legit changes where the tests need updated), and it works in my manual exercises of it.... but if it be possible to run it through some more paces that'd be great | 21:25 |
| clarkb | but dstufft only wants to test it, iirc its job enforcement that faisl on those not actually workingness | 21:25 |
| *** eharney has quit IRC | 21:25 | |
| clarkb | I may be wrong about that though, mordred is sort of the expert on these things around here | 21:25 |
| mordred | dstufft: I'm sure we can figure out something ... got a link to the pr? | 21:25 |
| fungi | right, basically just propose it and mark it as experimental/do-not-review set to work in progress, so that drive-by reviewers don't waste their time trying to tell you what a horrible person you are for proposing a requirements change which would pull in external unpackaged stuff | 21:25 |
| dstufft | https://github.com/jaraco/setuptools/pull/13 | 21:26 |
| *** tomoe_ has joined #openstack-infra | 21:27 | |
| *** aysyd has quit IRC | 21:27 | |
| mordred | clarkb: actually, I think what we want to do is propose a noop change to devstack | 21:28 |
| mordred | clarkb: since we do not install setuptools from requirements | 21:28 |
| fungi | that would be more than a one-liner (a several-liner) but should work and would still be fairly trivial | 21:28 |
| clarkb | that should work too, we do install it from requirements though if we list it right? | 21:28 |
| * mordred on it | 21:29 | |
| jeblair | http://puppetboard.openstack.org/report/nodepool.openstack.org/e740c1f0d42a4580625626db377eb58d465f75b6 | 21:29 |
| mordred | clarkb: no | 21:29 |
| jeblair | it's applied | 21:29 |
| clarkb | mordred: since tox targets have dep lists before we setup.py or whaetver | 21:29 |
| * fungi braces for impact | 21:29 | |
| mordred | right. but setuptools is not in a dep list | 21:29 |
| clarkb | jeblair: https://jenkins07.openstack.org/job/gate-nova-python27/6218/console that change may or may not use it depending on how old the node is | 21:29 |
| mordred | so the dep syncing will sync nothing | 21:29 |
| clarkb | mordred: you can add it | 21:29 |
| mordred | to a project? | 21:29 |
| clarkb | so the test would be put it in requirements.txt | 21:29 |
| clarkb | ya just as a WIP change | 21:30 |
| mordred | let me try devstack first ... | 21:30 |
| jeblair | clarkb: negative on that node | 21:30 |
| *** tomoe_ has quit IRC | 21:31 | |
| *** openstackgerrit has quit IRC | 21:31 | |
| mordred | clarkb, dstufft, fungi: https://review.openstack.org/119486 | 21:32 |
| dstufft | mordred: thanks! | 21:32 |
| jeblair | Exception: Unable to run ready script | 21:32 |
| jeblair | command: cd /opt/nodepool-scripts && ./configure_mirror.sh bare-trusty-rax-dfw-1960662.slave.openstack.org | 21:32 |
| jeblair | stdout: bash: ./configure_mirror.sh: No such file or directory | 21:32 |
| *** openstackgerrit has joined #openstack-infra | 21:32 | |
| clarkb | hrm | 21:33 |
| jeblair | that _might_ be temporary | 21:33 |
| clarkb | oh right | 21:33 |
| clarkb | because it could be a race between puppet on the nodes building and puppet applying to nodepool | 21:33 |
| fungi | yeah, i suppose the first indicator of failure is going to be if our node graph starts shrinking in used and growing substantially in building and delete | 21:34 |
| jeblair | clarkb: i don't think puppet's involved | 21:34 |
| clarkb | jeblair: fungi: do we need new images? | 21:34 |
| jeblair | clarkb: no | 21:34 |
| jeblair | oh | 21:34 |
| jeblair | maybe | 21:34 |
| mordred | dstufft: I re-uploaded it slightly differently | 21:34 |
| fungi | clarkb: we *did* need new images for the metadatam but those built (at least some of them did) | 21:34 |
| mordred | dstufft: so that it's just doing a pip install of git+git://github.com/dstufft/setuptools@a2b61035e9 | 21:34 |
| jeblair | clarkb: yes | 21:35 |
| fungi | checking to see if we maybe had image update failures earlier | 21:35 |
| clarkb | should we manually unpatch nodepool.yaml while we build them? | 21:35 |
| clarkb | I can start some image builds right now | 21:35 |
| jeblair | clarkb: yes. i'll do the manual patch. | 21:35 |
| dstufft | mordred: ok cool | 21:35 |
| mordred | dstufft: but that should run all of our integration tests including all of the package installation with that setuptools installed | 21:35 |
| clarkb | ok starting builds | 21:35 |
| mordred | which mean ... _something_ | 21:35 |
| fungi | what else did we patch which needed image updates besides the provider metadata file? | 21:36 |
| fungi | or is it just that the image update didn't succeed everywhere? | 21:36 |
| jeblair | fungi: the metada file isn't in the image; it's just that the ready script is expected to be in the image | 21:37 |
| fungi | ahh, yep, some didn't build today | 21:37 |
| fungi | right, the ready script would get run to create the metadata file | 21:38 |
| *** ociuhandu has quit IRC | 21:38 | |
| clarkb | ok I have hpcloud-b1 running really quickly to smoke out any potential bad image names | 21:39 |
| jeblair | fungi: image build copies ready script into image; when node is spun up, nodepool writes metadata file (some data are not available until the node actually exists); nodepool runs ready script (which it expects to already exist) | 21:39 |
| *** lttrl has joined #openstack-infra | 21:39 | |
| jeblair | (ready script expects provider file to exist) | 21:39 |
| *** otherwiseguy has quit IRC | 21:40 | |
| *** dimtruck is now known as zz_dimtruck | 21:40 | |
| jeblair | i've manually changed the config file; we probably have about 10 mins of failed builds to go from the old config | 21:40 |
| fungi | and the change to add the ready script merged yesterday, but some images didn't successfully update today | 21:40 |
| jeblair | fungi: no, the change to add the ready script was erroneously part of the script that enabled it. we should have done it that way. (though it would have been wrong since we didn't remember pydistutils until today) | 21:41 |
| jeblair | part of the *change* that enabled it | 21:41 |
| *** alexpilotti has quit IRC | 21:42 | |
| fungi | oh, right, the ready script was the one creating the mirror files, not the one creating the provider metadata files | 21:42 |
| fungi | so really they all need rebuilding then | 21:42 |
| clarkb | and rax-dfw are in progress now | 21:43 |
| clarkb | that is a lot of screen windows | 21:43 |
| *** andreykurilin_ has quit IRC | 21:43 | |
| fungi | did someone already disable puppet agent on nodepool.o.o? | 21:43 |
| jeblair | fungi: yes | 21:43 |
| fungi | okay, cool | 21:44 |
| jeblair | fungi: and yes | 21:44 |
| *** zns has joined #openstack-infra | 21:44 | |
| *** dims_ has joined #openstack-infra | 21:44 | |
| *** zz_dimtruck is now known as dimtruck | 21:44 | |
| clarkb | ya I have hpcloud-b1 and rax-dfw going. not sure how many I can manage without losing sanity | 21:44 |
| jeblair | i think we're sucessfully adding nodes now | 21:44 |
| *** dims_ has quit IRC | 21:46 | |
| *** emagana has joined #openstack-infra | 21:46 | |
| *** dims has quit IRC | 21:47 | |
| *** dims_ has joined #openstack-infra | 21:47 | |
| fungi | i would offer to babysit some slice of the image updates, but i have somewhere else i'm supposed to be for the next several hours, so i'll need to check back in later | 21:48 |
| jeblair | fungi: np, enjoy! | 21:48 |
| jeblair | clarkb: want me to take some? | 21:48 |
| clarkb | jeblair: sure maybe start on the opposite end of the hpcloud digits? | 21:49 |
| clarkb | they all seem to be building there so no bad image names | 21:49 |
| *** hdd has quit IRC | 21:49 | |
| clarkb | I am going to start ord now | 21:49 |
| fungi | thanks, if there's more to do when i get back, i'll take up the reins | 21:50 |
| *** e0ne has quit IRC | 21:51 | |
| clarkb | fungi: have fun | 21:51 |
| kevinbenton | I heard neutron stable/icehouse jobs failing in the gate. Does someone have a link to a failure? | 21:51 |
| clarkb | kevinbenton: http://logs.openstack.org/68/119268/1/check/check-tempest-dsvm-neutron-icehouse/bda14cd that one I think | 21:53 |
| *** otherwiseguy has joined #openstack-infra | 21:53 | |
| openstackgerrit | A change was merged to openstack-infra/config: capitalize sphinx in test-requirements https://review.openstack.org/119477 | 21:54 |
| *** cipcosma has quit IRC | 21:55 | |
| *** e0ne has joined #openstack-infra | 21:55 | |
| kevinbenton | clarkb: that’s a master branch patch | 21:55 |
| kevinbenton | clarkb: oh i see | 21:55 |
| kevinbenton | clarkb: tempest affects all | 21:56 |
| jeblair | clarkb: i've started b5; how low should i go? including b2? | 21:56 |
| clarkb | jeblair: sure | 21:56 |
| clarkb | jeblair: I am just about to start iad which will cover all of them | 21:56 |
| clarkb | except for tripleo | 21:57 |
| clarkb | butwe didn't ready them right? | 21:57 |
| jeblair | clarkb: correct | 21:57 |
| *** dims_ has quit IRC | 21:57 | |
| jeblair | clarkb: out of keypairs | 21:58 |
| *** robemcallister has joined #openstack-infra | 21:58 | |
| *** dims has joined #openstack-infra | 21:58 | |
| jeblair | clarkb: i'll try to correct | 21:58 |
| clarkb | ok iad is in progress now | 21:59 |
| *** bhuvan_ has joined #openstack-infra | 21:59 | |
| jesusaurus | jeblair: i commented on 115116 with a find command to run whisper-resize.py on existing data. note that the policy is space separated in the command but comma separated in the file | 22:00 |
| *** doug-fish has left #openstack-infra | 22:01 | |
| *** marcoemorais has quit IRC | 22:01 | |
| *** flaviof is now known as flaviof_zzz | 22:01 | |
| jeblair | jesusaurus: cool, thanks! | 22:01 |
| *** bhuvan has quit IRC | 22:02 | |
| *** marcoemorais has joined #openstack-infra | 22:02 | |
| *** marcoemorais has quit IRC | 22:02 | |
| *** marcoemorais has joined #openstack-infra | 22:02 | |
| *** marcoemorais has quit IRC | 22:03 | |
| *** marcoemorais has joined #openstack-infra | 22:03 | |
| *** robemcallister has quit IRC | 22:03 | |
| *** marcoemorais has quit IRC | 22:03 | |
| *** marcoemorais has joined #openstack-infra | 22:03 | |
| *** bhuvan_ has quit IRC | 22:04 | |
| *** robemcallister has joined #openstack-infra | 22:04 | |
| clarkb | the first handful of nodes have done their disconnect and are snapshotting | 22:04 |
| jeblair | clarkb: are your hpcloud-b1 nodes done? | 22:05 |
| jeblair | (i'm trying to figure out which keys are safe to delete | 22:05 |
| *** bhuvan has joined #openstack-infra | 22:05 | |
| clarkb | jeblair: no they are not done yet | 22:05 |
| *** dimtruck is now known as zz_dimtruck | 22:05 | |
| clarkb | they are all cloning or snapshotting so nearly tere | 22:05 |
| clarkb | I think we must ddos cirros and vmware image hosts :) | 22:06 |
| *** dims has quit IRC | 22:06 | |
| clarkb | I suppose this is far better than the slaves doing it | 22:06 |
| jeblair | okay, i think i've got the right ones to delete; deleting keypairs now | 22:08 |
| *** e0ne has quit IRC | 22:09 | |
| jeblair | and builds are running for b2-b5 | 22:10 |
| clarkb | great | 22:10 |
| jeblair | i think maybe we should wait on the 9 region split until after dib :) | 22:10 |
| clarkb | ++ | 22:11 |
| *** kgiusti has left #openstack-infra | 22:11 | |
| *** mfink has joined #openstack-infra | 22:12 | |
| clarkb | jeblair: I think -b1 is done now | 22:14 |
| *** gondoi is now known as zz_gondoi | 22:15 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 22:15 |
| *** koolhead17 has quit IRC | 22:15 | |
| jeblair | ERROR: Could not find a valid gem 'puppetlabs_spec_helper' (>= 0) in any repository | 22:16 |
| jeblair | that happened in centos6 on b5 | 22:16 |
| clarkb | I haven't seen it. maybe the gem server pulled a pypi? | 22:16 |
| jeblair | prolly so, will restart | 22:16 |
| *** robemcallister has quit IRC | 22:16 | |
| jeblair | clarkb: we could probably enable the ready script for one of the labels... | 22:17 |
| *** robemcallister has joined #openstack-infra | 22:17 | |
| jeblair | clarkb: it applies to all the providers, and will fail for the ones that haven't finished yet | 22:17 |
| clarkb | let me see which ones have finished | 22:18 |
| jeblair | clarkb: but we might get a head start on any subsequent problems for the ones wich have finished | 22:18 |
| *** otherwiseguy has quit IRC | 22:18 | |
| clarkb | bare-centos6 and bare-trusty are both done in -b1 and dfw | 22:18 |
| *** otherwiseguy has joined #openstack-infra | 22:19 | |
| jeblair | 2014-09-05 22:12:05,278 INFO nodepool.image.build.hpcloud-b3.devstack-f20: Error | 22:19 |
| jeblair | : Cannot retrieve metalink for repository: fedora/20/x86_64. Please verify its path and try again | 22:19 |
| jeblair | nice | 22:19 |
| clarkb | and centos6 is snapshotting in ord | 22:19 |
| clarkb | so maybe centos6? | 22:19 |
| jeblair | clarkb: ok | 22:19 |
| *** koolhead17 has joined #openstack-infra | 22:19 | |
| *** koolhead17 has joined #openstack-infra | 22:19 | |
| jeblair | clarkb: shall i edit the file? | 22:19 |
| clarkb | go for it | 22:19 |
| *** koolhead17 has quit IRC | 22:19 | |
| clarkb | I now realize I should've gone by image and not provider | 22:20 |
| clarkb | whoops | 22:20 |
| *** bdpayne has quit IRC | 22:20 | |
| jeblair | centos6 is configured to run the ready script | 22:21 |
| jeblair | wow, it's really amazing we ever get images | 22:21 |
| clarkb | :/ | 22:22 |
| asselin__ | how do you setup a job to override a particular project with a gerrit revision? i.e. checkout the latest patch of https://review.openstack.org/#/c/118492/ | 22:23 |
| jeblair | 2014-09-05 22:23:22,740 INFO nodepool.NodeLauncher: Creating server with hostname bare-centos6-rax-dfw-1961515.slave.openstack.org in rax-dfw from image bare-centos6 for node id: 1961515 | 22:23 |
| jeblair | 2014-09-05 22:23:22,754 INFO nodepool.NodeLauncher: Creating server with hostname bare-centos6-rax-dfw-1961516.slave.openstack.org in rax-dfw from image bare-centos6 for node id: 1961516 | 22:23 |
| *** emagana has quit IRC | 22:24 | |
| jeblair | clarkb: ^ those 2 should end up running the script | 22:24 |
| *** emagana has joined #openstack-infra | 22:24 | |
| openstackgerrit | Edward Raigosa (wenlock) proposed a change to openstack-infra/config: give install_modules options for loading requirements externally https://review.openstack.org/117892 | 22:24 |
| *** robemcallister has quit IRC | 22:25 | |
| clarkb | awesome | 22:26 |
| *** stevemar has quit IRC | 22:28 | |
| *** emagana has quit IRC | 22:28 | |
| clarkb | dfw is almost done with all of its images | 22:28 |
| *** reed has quit IRC | 22:30 | |
| *** bknudson has quit IRC | 22:30 | |
| *** SumitNaiksatam has quit IRC | 22:31 | |
| *** leifmadsen has quit IRC | 22:35 | |
| *** robemcallister has joined #openstack-infra | 22:36 | |
| *** MaxV has joined #openstack-infra | 22:39 | |
| jeblair | jobs starting on https://jenkins03.openstack.org/job/gate-sahara-python26/617/ | 22:39 |
| jeblair | index-url = http://pypi.DFW.openstack.org/simple | 22:40 |
| jeblair | that's from the node | 22:40 |
| jeblair | and the job succeeded | 22:40 |
| *** leifmadsen has joined #openstack-infra | 22:40 | |
| clarkb | https://jenkins03.openstack.org/job/gate-python-saharaclient-python26/68/console is another | 22:41 |
| clarkb | its in progress though | 22:41 |
| lifeless | mordred: there are other ways forward | 22:41 |
| lifeless | mordred: such as making the version marker in setup.cfg a desire not an absolute | 22:41 |
| lifeless | mordred: generating historical version numbers is not good - we've had bugs reported on that in pbr | 22:42 |
| clarkb | jeblair: did you get the index url off the machine? | 22:42 |
| lifeless | mordred: and see the thread I started about this with ttc | 22:42 |
| lifeless | erm | 22:42 |
| lifeless | ttx | 22:42 |
| jeblair | clarkb: yep | 22:42 |
| *** signed8bit is now known as signed8bit_ZZZzz | 22:42 | |
| lifeless | mordred: ttx and I believe its not a pragmatic issue for OpenStack today, because we do release tags on branches not trunk, so we don't run into the huge race with the gate | 22:42 |
| clarkb | cool I checked the one for te job I linked too and same thing | 22:43 |
| clarkb | so I think that is working | 22:43 |
| jeblair | yay | 22:43 |
| jeblair | clarkb: i'd guess my builds are about 1/3 done | 22:43 |
| jeblair | i mean, about 1/3 of them are done :) | 22:44 |
| clarkb | ord is done, iad has one image snapshotting left, and dfw has 2 | 22:44 |
| clarkb | so rax is almost there | 22:44 |
| lifeless | clarkb: do you know what brought the ^ subject up with mordred? I can't see it in scrollback | 22:44 |
| clarkb | lifeless: he said he wanted to catch up on pep440 and pbr related things | 22:45 |
| lifeless | ok | 22:45 |
| clarkb | lifeless: and I mentioned that you were talking to ttx about how ti affects release time | 22:45 |
| lifeless | so thats good, but he's going off half cocked at the moment :) | 22:45 |
| *** signed8bit_ZZZzz has quit IRC | 22:46 | |
| *** arborism is now known as amcrn | 22:46 | |
| *** lcheng_ has quit IRC | 22:47 | |
| *** homeless has quit IRC | 22:47 | |
| *** dmsimard is now known as dmsimard_away | 22:47 | |
| lifeless | shiny https://code.facebook.com/posts/681382905244727/introducing-wedge-and-fboss-the-next-steps-toward-a-disaggregated-network/ | 22:48 |
| jeblair | b5 is done | 22:50 |
| mordred | lifeless: I often do that ... if there's a plan, OK. but I certainly don't want to make release more baroque | 22:51 |
| jeblair | 2014-09-05 22:48:10,160 INFO nodepool.image.build.hpcloud-b4.py3k-precise: ConnectionError: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /packages/source/g/git-review/git-review-1.17.tar.gz (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer) | 22:51 |
| jeblair | oh the irony ^ | 22:51 |
| notmyname | lifeless: I want to call your attention to https://review.openstack.org/#/c/118200/ since you were (indirectly) referenced in the comments. | 22:51 |
| clarkb | jeblair: just 3 more images on my end | 22:51 |
| jeblair | b2 is done | 22:53 |
| clarkb | all in dfw oddly | 22:54 |
| clarkb | dfw glance is probably wondering why we hate it | 22:54 |
| *** emagana has joined #openstack-infra | 22:55 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Set required devstack variables for Ironic+grenade https://review.openstack.org/116761 | 22:55 |
| *** emagana has quit IRC | 22:55 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Adds support for Ironic parallel Tempest testing https://review.openstack.org/118700 | 22:56 |
| *** emagana has joined #openstack-infra | 22:56 | |
| *** _nadya_ has joined #openstack-infra | 22:56 | |
| *** Sincler has quit IRC | 22:56 | |
| clarkb | adam_g: is 116761 close to being ready? | 22:57 |
| adam_g | clarkb, yeah, was just going to ping you about it | 22:57 |
| *** otherwiseguy has quit IRC | 22:57 | |
| jeblair | clarkb: b3 is done; 3 more in b4 | 22:57 |
| *** craigbr has joined #openstack-infra | 22:58 | |
| *** oanufriev has joined #openstack-infra | 22:58 | |
| clarkb | jeblair: still waiting on these 3 in dfw | 22:59 |
| adam_g | clarkb, those two should be good to go, and get us close to having the sideways stuff functional. we're still waiting on one tempest fix. we also need to disable cinder in features.yaml, which will require https://review.openstack.org/#/c/112134/ otherwise we'll break the current ironic gate | 22:59 |
| *** tonytan4ever has quit IRC | 22:59 | |
| clarkb | they are all sitting there at the resetting dropped connection output | 22:59 |
| clarkb | so they must be snapshotting | 22:59 |
| *** harlowja is now known as harlowja_away | 23:00 | |
| clarkb | adam_g: so is there a preferred order? probably d-g first since it takes the longest then the config changes? | 23:00 |
| clarkb | jeblair: now down to 2 | 23:01 |
| *** _nadya_ has quit IRC | 23:01 | |
| jeblair | clarkb: i'm down to 2 as well | 23:01 |
| jeblair | one is snapshotting, one is in puppet (it was a rerun of a late failure) | 23:02 |
| adam_g | clarkb, i dont order matters for anything grenade related but having https://review.openstack.org/#/c/112134/ needs to land before https://review.openstack.org/#/c/115803/ | 23:02 |
| clarkb | ok | 23:02 |
| *** craigbr has quit IRC | 23:02 | |
| clarkb | I probably won't get that stack reviewed tonight. I am starving and need to find food as soon as these nodes are happy | 23:03 |
| clarkb | jeblair: should we undo the patched nodepool.yaml now? | 23:03 |
| adam_g | clarkb, that move the current virtual-ironic job to the main ironic gating job, and will use the ugly regex to skip all volume tests. we can then disable cinder in features.yaml, and (assuming its stable) migrate the new experimental parallel job to the main ironic gate job (which doesn't need to use a regex and can gracefully handle no cinder | 23:03 |
| lifeless | mordred: release isn't changing | 23:04 |
| lifeless | mordred: I think we can make it simpler by changing pbr further, but see the thread. | 23:04 |
| *** otherwiseguy has joined #openstack-infra | 23:04 | |
| clarkb | adam_g: sounds good | 23:04 |
| lifeless | mordred: issuing pre-release versions of the last release is bad though - and like I said we had bug reports about it - it affects people | 23:04 |
| clarkb | jeblair: just one left now | 23:05 |
| adam_g | clarkb, no problem, monday is probably a better time to babysit all that reconfiguration | 23:05 |
| lifeless | notmyname: hi | 23:05 |
| notmyname | lifeless: hi | 23:06 |
| notmyname | lifeless: I wanted to tell you that as an FYI | 23:06 |
| *** emagana has quit IRC | 23:06 | |
| lifeless | ok | 23:06 |
| *** emagana has joined #openstack-infra | 23:07 | |
| notmyname | lifeless: basically, it's something that I really want to land in Swift for Juno so that we can take the next step for kilo. but I also don't want triple-o + swift to be broken in juno | 23:07 |
| notmyname | lifeless: and since juno is getting closer... | 23:07 |
| *** lttrl has quit IRC | 23:08 | |
| *** bhuvan has quit IRC | 23:09 | |
| *** reed has joined #openstack-infra | 23:09 | |
| notmyname | lifeless: if this will cause major issues in juno for triple-o, I'd like to know and try to work it out asap | 23:09 |
| *** craigbr has joined #openstack-infra | 23:10 | |
| lifeless | notmyname: commented on the review, though I don't really have context for it | 23:10 |
| *** robemcallister has quit IRC | 23:10 | |
| lifeless | notmyname: and there's no blueprint referenced | 23:11 |
| *** dangers is now known as dangers_away | 23:11 | |
| *** emagana has quit IRC | 23:11 | |
| lifeless | notmyname: I say this in my comment - if you're changing the default port, requiring the option seems counterproductive. | 23:11 |
| lifeless | notmyname: wouldn't it be simpler to just bind to two ports by default, and deprecate the current one for a cycle? | 23:12 |
| notmyname | lifeless: replying in gerrit | 23:12 |
| *** bhuvan has joined #openstack-infra | 23:17 | |
| *** robemcallister has joined #openstack-infra | 23:17 | |
| *** craigbr has quit IRC | 23:18 | |
| *** robemcallister has quit IRC | 23:21 | |
| clarkb | jeblair: done | 23:22 |
| *** praneshp has quit IRC | 23:23 | |
| *** emagana has joined #openstack-infra | 23:23 | |
| *** otherwiseguy has quit IRC | 23:23 | |
| *** melwitt has quit IRC | 23:24 | |
| jeblair | clarkb: my last one is snapshotting now; i say we go ahead and re-enable puppet | 23:24 |
| clarkb | jeblair: ++ | 23:24 |
| jeblair | clarkb: done, and ran puppet too, so it's enabled everywhere | 23:25 |
| clarkb | woot | 23:25 |
| *** marcoemorais has quit IRC | 23:26 | |
| *** praneshp has joined #openstack-infra | 23:26 | |
| *** marcoemorais has joined #openstack-infra | 23:26 | |
| *** marcoemorais has quit IRC | 23:26 | |
| *** marcoemorais has joined #openstack-infra | 23:26 | |
| *** openstack_ has quit IRC | 23:26 | |
| *** marcoemorais has quit IRC | 23:27 | |
| *** marcoemorais has joined #openstack-infra | 23:27 | |
| *** tomoe_ has joined #openstack-infra | 23:28 | |
| *** harlowja_away is now known as harlowja | 23:31 | |
| *** robemcallister has joined #openstack-infra | 23:31 | |
| jeblair | clarkb: i don't see lots of exceptions | 23:32 |
| jeblair | clarkb: also, final snapshot is complete | 23:32 |
| *** tomoe_ has quit IRC | 23:32 | |
| *** robemcallister has quit IRC | 23:33 | |
| *** emagana has quit IRC | 23:33 | |
| jeblair | flashgordon: per-region mirrors should be in use in jobs starting after nowish. | 23:34 |
| *** emagana has joined #openstack-infra | 23:34 | |
| clarkb | ya I was tailin the log too | 23:34 |
| jeblair | mordred: fungi: ^ | 23:34 |
| *** robemcallister has joined #openstack-infra | 23:35 | |
| jeblair | so, you know, just in time for the weekend drop in traffic :) | 23:35 |
| *** leifmadsen has quit IRC | 23:37 | |
| *** emagana has quit IRC | 23:38 | |
| *** arnaud__ has joined #openstack-infra | 23:40 | |
| *** robemcallister has quit IRC | 23:40 | |
| *** jaypipes-afk has quit IRC | 23:40 | |
| HenryThe8th | Hi, in the #neutron channel salv-orlando mentioned he received reports that "the icehouse jobs for neutron are failing at 100%"? | 23:41 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: turn off clean log checking https://review.openstack.org/119512 | 23:41 |
| sdague | jeblair: so, time to call it a weekend, but there is a final patch for the week. | 23:41 |
| clarkb | HenryThe8th: http://logs.openstack.org/68/119268/1/check/check-tempest-dsvm-neutron-icehouse/bda14cd that test. kevinbenton was looking at it too | 23:41 |
| clarkb | HenryThe8th: and he wasn't sure if it was 100% or not | 23:42 |
| clarkb | the failure needed debugging | 23:42 |
| jeblair | sdague: that makes me very sad :( | 23:42 |
| sdague | jeblair: yeh, well such it is | 23:42 |
| clarkb | sdague: fwiw the cleaner output around it has helped me at least | 23:43 |
| kevinbenton | clarkb: i got caught up with other stuff. I didn’t see anything obvious though | 23:43 |
| sdague | clarkb: which cleaner output? | 23:43 |
| clarkb | sdague: mtreinish has it spitting out the files that failed rather than every line that failed | 23:43 |
| clarkb | sdague: it was really hard to read logs before with the old behavior. It would also report non witelisted errors but we were whitelisting at the file level | 23:44 |
| clarkb | so it was completely confusing | 23:44 |
| sdague | clarkb: sure, but the overall process right now looks mostly like it's failing a bunch of jobs on n-cond deadlocks, which are really oslo.db issues | 23:44 |
| clarkb | yeah | 23:44 |
| sdague | so, honestly, I'm not sure it's adding much value | 23:44 |
| sdague | but, feel free to disagree :) | 23:45 |
| *** robemcallister has joined #openstack-infra | 23:45 | |
| clarkb | ok thats fine | 23:45 |
| clarkb | I defer to you :) | 23:45 |
| sdague | I honestly wonder if this is a better thing to do closer to the projects. | 23:46 |
| sdague | ok, weekend, enjoy folks | 23:47 |
| jeblair | sdague: you too | 23:48 |
| jeblair | i'm going to go watch some people move a printing press | 23:49 |
| *** bdpayne has joined #openstack-infra | 23:49 | |
| clarkb | jeblair: does it involve even more heavy machinery? | 23:49 |
| *** david-lyle has quit IRC | 23:49 | |
| *** MaxV has quit IRC | 23:49 | |
| *** mmaglana has quit IRC | 23:49 | |
| jeblair | clarkb: amazingly probably not; i've only ever seen them use like a crowbar and some dollies. though probably a forklift to get it through the window. | 23:50 |
| jeblair | simple machines are the best i guess? :) | 23:50 |
| *** gokrokve has quit IRC | 23:51 | |
| clarkb | but forklifts sound like fun :) | 23:53 |
| clarkb | well things don't appear to be on fire | 23:54 |
| clarkb | I am going to go find food now | 23:54 |
| *** marcoemorais has quit IRC | 23:55 | |
| *** marcoemorais has joined #openstack-infra | 23:56 | |
| *** bdpayne has quit IRC | 23:58 | |
| *** gyee has quit IRC | 23:58 | |
| *** dmsimard_away is now known as dmsimard | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
