| *** eharney has quit IRC | 00:12 | |
| *** rloo has quit IRC | 00:38 | |
| rm_work | zzzeek: hey, any tips on how I might be able to check the *writability* of a DB? without having any specific data I want to write to a specific table... we check general "upness" with `engine.scalar(select([1]))` but that doesn't help us detect if the DB is read-only | 00:45 |
|---|---|---|
| rm_work | hoping you might have a recommendation -- in MySQL I think I can just read the global `read_only`, but that's not portable | 00:46 |
| rm_work | also i don't know if that's totally reliable anyway | 00:47 |
| *** bauzas has quit IRC | 01:01 | |
| fungi | and that probably won't detect if the underlying filesystem has been remounted read-only due to errors or the like | 01:11 |
| zzzeek | rm_work: "read only" usually means you dont have grants on the table you're targeting, so, "show grants"? or are you trying to detect a read-only slave DB ? | 01:20 |
| zzzeek | nothing in that area would be portable across different DB types | 01:20 |
| *** raildo has quit IRC | 01:24 | |
| rm_work | yeah, the latter | 01:24 |
| rm_work | so it's looking like my best option for at least mysql is: | 01:25 |
| rm_work | `SELECT * FROM someTable WHERE id=0 FOR UPDATE;` | 01:25 |
| rm_work | but yeah, not portable | 01:25 |
| rm_work | but openstack only officially supports mysql now? so ... i mean ... ehhhhhhhhh i hate it so much but not sure what else I can do | 01:25 |
| *** bauzas has joined #openstack-dev | 01:29 | |
| *** corvus has quit IRC | 01:48 | |
| *** corvus has joined #openstack-dev | 01:49 | |
| *** mrunge_ has joined #openstack-dev | 02:02 | |
| *** mrunge has quit IRC | 02:03 | |
| *** smarcet has joined #openstack-dev | 02:06 | |
| rm_work | zzzeek: if i can do a select with_for_update() I think it'll work in a fairly general case? | 02:09 |
| rm_work | just need to figure out how to get a table object to do a select on | 02:09 |
| zzzeek | rm_work: that might tell you something about one particular table on some database backends, but probably not most of them, not sqlite at least, and probably wont tell you the answer for all varities of "read only" | 02:15 |
| zzzeek | "read only" is not really a general thing with databases | 02:15 |
| zzzeek | also this is one of those "you shouldnt have to be doing what you are trying to do" kinds of issues | 02:16 |
| rm_work | hmm | 02:16 |
| rm_work | well, maybe it'd be close enough since really a single table is all i care about in this instance | 02:16 |
| zzzeek | most database systems have grants | 02:16 |
| zzzeek | but that is not the only reason a table owujldnt be writable | 02:16 |
| zzzeek | in SQLite if you dont have write permissions on the file, you wont be able to write | 02:17 |
| zzzeek | some kinds of "read only" you dont find out about until you commit a transaction | 02:17 |
| rm_work | if an UPDATE fails and kicks me into a `DBConnectionError` exception block, i have to wait until a specific table is writeable again, then start a delay countdown before allowing writes again | 02:17 |
| zzzeek | a DBConnectionError, meaning, the database got disconnected? | 02:18 |
| rm_work | err i guess it'd be ... | 02:18 |
| zzzeek | that's not a thing that is specific to an UPDATE | 02:18 |
| zzzeek | that's a network issue or the DB connection got corrupted or it timed out | 02:18 |
| rm_work | what error would that be actually | 02:18 |
| rm_work | I don't see like a ReadOnly one | 02:18 |
| rm_work | `ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement` is what MySQL says, for example | 02:19 |
| zzzeek | there is no "readonly" error | 02:19 |
| rm_work | how does that translate | 02:19 |
| zzzeek | ah | 02:19 |
| zzzeek | that's a totally mysql specific thing | 02:19 |
| rm_work | what exception does that raise | 02:19 |
| rm_work | and wouldn't the equivalent in like, postgres raise the same exception? | 02:20 |
| zzzeek | oslo.db if that's what we're talking about probably raises DBError for that | 02:20 |
| zzzeek | e.g. totally generic error | 02:20 |
| zzzeek | that's not the kind of thing that "goes away" someone has to restart the DB without that flag | 02:20 |
| rm_work | ah yes this does come from oslo_db, not SQLA directly bleh | 02:20 |
| zzzeek | the driver probably does OperationalError for that | 02:21 |
| rm_work | I think you can just set it? `SET GLOBAL read_only = 0;` | 02:21 |
| zzzeek | OK | 02:21 |
| rm_work | ok so maybe I need your advice on WTF to do at a higher level | 02:21 |
| zzzeek | doesnt soudn like "the server is running with the --read-only option" then | 02:21 |
| rm_work | rather than how to do this specific thing | 02:21 |
| zzzeek | i would...not set that flag | 02:22 |
| rm_work | well right | 02:22 |
| rm_work | but if a master fails and we're forced onto a R/O slave; or something breaks internally and the DB goes R/O | 02:22 |
| rm_work | this happens | 02:22 |
| zzzeek | but that flag might be set for various forms of replication or warm standby etc. | 02:22 |
| rm_work | right, there are cases | 02:22 |
| rm_work | it happened last week during a DB upgrade here, apparently the runbook for that involves setting the galera cluster to R/O somehow first | 02:23 |
| rm_work | which I think is via that global | 02:23 |
| zzzeek | so if you want to fail gracefully for system was moved onto a read slave, you'd have to catch that exception and handle it specifically. this is where oslo.db is useful because oslo.db can be enhanced to detect that error and give you the ReadOnly error you are looking for | 02:23 |
| zzzeek | oh its Galera? galera "read only" is not a normal thing. galera is multi-master and when a node goes into read only mode there shuld be haproxy that will bounce the connection over to another working node | 02:24 |
| zzzeek | so if you are getting that error in a brief window in failover again you'd want oslo.db to tell you that's a disconnecvt sitaution | 02:24 |
| rm_work | the issue is, if the DB goes R/O for some time, we can't write healthchecks to our health table, so failovers try to initiate -- which don't work because they require a writeable DB too -- but then the second the DB becomes writable, before new healthchecks have a chance to be written for otherwise healthy nodes, the failovers suddenly WORK and everything is failed over basically instantly | 02:25 |
| zzzeek | you can also use sqlalchemy event handlers to set that up as a disconnect too | 02:25 |
| rm_work | (this is for Octavia, but the idea is pretty generic I think) | 02:25 |
| zzzeek | but this is galera, there should always be a writeable master unless you are dealing with the cluster losing quorum ? | 02:26 |
| rm_work | i think there's only two nodes in it, so yes | 02:26 |
| rm_work | i don't know the specifics, I could query folks | 02:26 |
| zzzeek | you....cant have a galera cluster with only two nodes, really | 02:26 |
| zzzeek | i mean you *can* but it would be not smart | 02:26 |
| zzzeek | anyway this kind of thing is definiltey not very generalizable to other DBs galera is very unique | 02:27 |
| rm_work | hmm k | 02:27 |
| zzzeek | if you want the "we're in read only mode" to result in oslo.db / sqlalchemy throwing away that connection and reconnecting, which we would assume goes to your haproxy that will give you a good connection on the next go, that can be done | 02:28 |
| rm_work | actually i'm assuming there's only one DB up, and it's in R/O, as the other is down for maintenance? but i'm trying to figure out WHY it's done that way from our DBAs | 02:28 |
| zzzeek | if galera can't do quorum then it goes into read only | 02:30 |
| zzzeek | which is why...you have an odd number of nodes | 02:30 |
| rm_work | ah ok i guess this cluster isn't actually galera | 02:30 |
| rm_work | `Dual Master/Single Writer MySQL` | 02:30 |
| zzzeek | depends on how they did it | 02:30 |
| rm_work | `There is no automatic failover so if 1 DB goes down the other is only RO until it is made master.` | 02:30 |
| zzzeek | we use gaelra in this way, though three nodse | 02:30 |
| zzzeek | we have haproxy point all traffic to one node at a time | 02:31 |
| zzzeek | so it is multi master / single writer | 02:31 |
| rm_work | hmm | 02:31 |
| rm_work | trying to get more info still | 02:33 |
| rm_work | yeah ok so it's to a LB and the LB has a healthcheck for whether the DB node is set to "master" (if it hits the healthcheck endpoint and the result is MASTER then it's "up", otherwise it's down) | 02:46 |
| rm_work | so the DB has only one active member at a time | 02:46 |
| *** smarcet has quit IRC | 02:46 | |
| rm_work | I guess the way they went to RO was because BOTH were manually set to RO for some special maintenance operation | 02:46 |
| zzzeek | well I wouldnt code for that :) | 02:47 |
| rm_work | yeah that's what i'm coming around to as well, lol | 02:47 |
| zzzeek | gotta run it's 11 pm here lates | 02:47 |
| rm_work | thanks for humoring me | 02:47 |
| rm_work | later! | 02:47 |
| *** smarcet has joined #openstack-dev | 03:03 | |
| *** smarcet has left #openstack-dev | 03:13 | |
| *** smarcet has joined #openstack-dev | 04:08 | |
| *** smarcet has quit IRC | 04:25 | |
| *** smarcet has joined #openstack-dev | 04:25 | |
| *** evrardjp has quit IRC | 04:37 | |
| *** evrardjp has joined #openstack-dev | 04:37 | |
| *** udesale has joined #openstack-dev | 04:40 | |
| *** logan- has quit IRC | 04:45 | |
| *** logan_ has joined #openstack-dev | 04:47 | |
| *** logan_ is now known as logan- | 04:48 | |
| *** _mlavalle_1 has joined #openstack-dev | 04:50 | |
| *** mlavalle has quit IRC | 04:53 | |
| *** dpawlik has joined #openstack-dev | 06:07 | |
| *** miloa has joined #openstack-dev | 06:11 | |
| *** ttsiouts has joined #openstack-dev | 06:30 | |
| *** slaweq_ has quit IRC | 06:36 | |
| *** dpawlik has quit IRC | 06:37 | |
| *** slaweq_ has joined #openstack-dev | 06:37 | |
| *** dpawlik has joined #openstack-dev | 06:39 | |
| *** slaweq_ has quit IRC | 06:41 | |
| *** _mlavalle_1 has quit IRC | 06:41 | |
| *** nightmare_unreal has joined #openstack-dev | 06:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 06:55 | |
| *** jcapitao has joined #openstack-dev | 06:57 | |
| *** jtomasek has joined #openstack-dev | 06:57 | |
| *** jtomasek has quit IRC | 06:59 | |
| *** jtomasek has joined #openstack-dev | 06:59 | |
| *** ricolin has quit IRC | 06:59 | |
| *** slaweq_ has joined #openstack-dev | 07:00 | |
| *** dklyle has quit IRC | 07:00 | |
| *** tovin07 has joined #openstack-dev | 07:01 | |
| *** dancn has joined #openstack-dev | 07:02 | |
| *** slaweq_ is now known as slaweq | 07:03 | |
| *** luigi has joined #openstack-dev | 07:03 | |
| *** tovin07 has quit IRC | 07:06 | |
| *** dtantsur|afk is now known as dtantsur | 07:06 | |
| *** iurygregory has joined #openstack-dev | 07:11 | |
| *** tesseract has joined #openstack-dev | 07:15 | |
| *** rpittau|afk is now known as rpittau | 07:18 | |
| *** avolkov has joined #openstack-dev | 07:26 | |
| *** psachin has joined #openstack-dev | 07:37 | |
| *** tosky has joined #openstack-dev | 07:42 | |
| *** ricolin has joined #openstack-dev | 07:44 | |
| *** ralonsoh has joined #openstack-dev | 07:45 | |
| *** abdysn has joined #openstack-dev | 07:54 | |
| *** jpich has joined #openstack-dev | 07:55 | |
| *** threestrands has quit IRC | 08:00 | |
| *** __ministry has joined #openstack-dev | 08:49 | |
| *** surpatil has joined #openstack-dev | 09:02 | |
| *** tkajinam has quit IRC | 09:03 | |
| *** dpawlik has quit IRC | 09:10 | |
| *** surpatil has quit IRC | 09:30 | |
| *** udesale_ has joined #openstack-dev | 09:34 | |
| *** udesale has quit IRC | 09:38 | |
| *** dpawlik has joined #openstack-dev | 09:49 | |
| *** d34dh0r53 has quit IRC | 10:02 | |
| *** d34dh0r53 has joined #openstack-dev | 10:03 | |
| *** jpich has quit IRC | 10:06 | |
| *** jpich has joined #openstack-dev | 10:07 | |
| *** Abdallahyas has joined #openstack-dev | 10:07 | |
| *** abdysn has quit IRC | 10:10 | |
| *** gfidente|afk is now known as gfidente | 10:11 | |
| *** rpittau is now known as rpittau|bbl | 10:13 | |
| *** tonythomas has joined #openstack-dev | 10:19 | |
| *** zadock has joined #openstack-dev | 10:24 | |
| *** zadock has quit IRC | 10:25 | |
| openstackgerrit | Thierry Carrez proposed openstack/governance master: Appoint Ben Nemec as Oslo PTL https://review.opendev.org/718655 | 10:36 |
| *** ttsiouts has quit IRC | 10:46 | |
| *** ttsiouts has joined #openstack-dev | 10:51 | |
| *** jcapitao is now known as jcapitao_lunch | 10:59 | |
| *** ttsiouts has quit IRC | 11:07 | |
| *** ttsiouts has joined #openstack-dev | 11:26 | |
| *** ttsiouts has quit IRC | 11:42 | |
| *** ttsiouts has joined #openstack-dev | 11:43 | |
| *** dpawlik has quit IRC | 11:56 | |
| *** rpittau|bbl is now known as rpittau | 11:57 | |
| *** nweinber has joined #openstack-dev | 12:02 | |
| *** eharney has joined #openstack-dev | 12:03 | |
| *** Abdallahyas has quit IRC | 12:07 | |
| *** dpawlik has joined #openstack-dev | 12:08 | |
| *** raildo has joined #openstack-dev | 12:10 | |
| *** ierdem has joined #openstack-dev | 12:12 | |
| ierdem | Hello, i am trying to install murano on devstack but it throws an error "murano-dashboard requires Python '>=3.6' but the running Python is 2.7.17". I have python 3.6 on my environmet, can you help me please? | 12:12 |
| *** jcapitao_lunch is now known as jcapitao | 12:13 | |
| *** igordc has quit IRC | 12:18 | |
| *** kgiusti has joined #openstack-dev | 12:20 | |
| *** rloo has joined #openstack-dev | 12:23 | |
| *** psachin has quit IRC | 12:26 | |
| *** iurygregory has quit IRC | 12:43 | |
| *** iurygregory has joined #openstack-dev | 12:44 | |
| *** ttsiouts has quit IRC | 12:48 | |
| *** ttsiouts has joined #openstack-dev | 12:50 | |
| *** lbragstad_ has joined #openstack-dev | 12:52 | |
| *** lbragstad has quit IRC | 12:54 | |
| *** ttsiouts has quit IRC | 12:59 | |
| *** ttsiouts has joined #openstack-dev | 13:03 | |
| *** dancn has quit IRC | 13:03 | |
| *** dancn has joined #openstack-dev | 13:09 | |
| *** mrunge_ is now known as mrunge | 13:14 | |
| *** ttsiouts has quit IRC | 13:19 | |
| *** ttsiouts has joined #openstack-dev | 13:29 | |
| *** priteau has joined #openstack-dev | 13:48 | |
| *** ttsiouts has quit IRC | 13:49 | |
| *** ttsiouts has joined #openstack-dev | 13:49 | |
| *** rcernin has quit IRC | 13:54 | |
| *** nightmare_unreal has quit IRC | 14:00 | |
| *** abdysn has joined #openstack-dev | 14:04 | |
| *** ttsiouts has quit IRC | 14:05 | |
| *** d34dh0r53 has quit IRC | 14:08 | |
| *** d34dh0r53 has joined #openstack-dev | 14:10 | |
| *** tonythomas has quit IRC | 14:10 | |
| *** d34dh0r53 has quit IRC | 14:11 | |
| *** tkajinam has joined #openstack-dev | 14:13 | |
| *** dave-mccowan has joined #openstack-dev | 14:14 | |
| *** lpetrut has joined #openstack-dev | 14:19 | |
| *** lbragstad_ is now known as lbragstad | 14:20 | |
| *** nightmare_unreal has joined #openstack-dev | 14:26 | |
| *** dave-mccowan has quit IRC | 14:30 | |
| *** Hien has quit IRC | 14:35 | |
| *** Hien has joined #openstack-dev | 14:37 | |
| *** ttsiouts has joined #openstack-dev | 14:45 | |
| *** dklyle has joined #openstack-dev | 14:47 | |
| *** ttsiouts has quit IRC | 14:50 | |
| *** dancn has quit IRC | 14:51 | |
| *** abdysn has quit IRC | 14:54 | |
| *** abdysn has joined #openstack-dev | 14:54 | |
| *** armstrong has joined #openstack-dev | 14:58 | |
| *** dancn has joined #openstack-dev | 14:59 | |
| *** abdysn has quit IRC | 15:00 | |
| *** gyee has joined #openstack-dev | 15:05 | |
| *** ttsiouts has joined #openstack-dev | 15:05 | |
| *** slaweq has quit IRC | 15:06 | |
| *** dklyle has quit IRC | 15:12 | |
| *** dklyle has joined #openstack-dev | 15:18 | |
| *** slaweq has joined #openstack-dev | 15:21 | |
| *** ierdem has quit IRC | 15:24 | |
| *** avolkov has quit IRC | 15:25 | |
| *** luigi has quit IRC | 15:26 | |
| *** ttsiouts has quit IRC | 15:45 | |
| *** tkajinam has quit IRC | 15:49 | |
| *** diablo_rojo has joined #openstack-dev | 15:57 | |
| *** rpittau is now known as rpittau|afk | 15:59 | |
| *** mikefix has quit IRC | 16:02 | |
| *** iurygregory has quit IRC | 16:03 | |
| *** dpawlik has quit IRC | 16:06 | |
| *** smarcet has quit IRC | 16:23 | |
| *** lpetrut has quit IRC | 16:26 | |
| *** jpich has quit IRC | 16:28 | |
| *** udesale_ has quit IRC | 16:35 | |
| *** evrardjp has quit IRC | 16:37 | |
| *** evrardjp has joined #openstack-dev | 16:37 | |
| *** jcapitao has quit IRC | 16:37 | |
| *** dtantsur is now known as dtantsur|afk | 16:46 | |
| *** nightmare_unreal has quit IRC | 16:50 | |
| *** ccamposr__ has quit IRC | 17:02 | |
| *** ricolin has quit IRC | 17:04 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:12 | |
| *** ccamposr has joined #openstack-dev | 17:16 | |
| *** priteau has quit IRC | 17:22 | |
| *** smarcet has joined #openstack-dev | 17:22 | |
| *** priteau has joined #openstack-dev | 17:29 | |
| *** smarcet has quit IRC | 17:34 | |
| *** priteau has quit IRC | 17:34 | |
| *** mlavalle has joined #openstack-dev | 17:52 | |
| *** priteau has joined #openstack-dev | 17:55 | |
| *** priteau has quit IRC | 17:59 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 18:02 | |
| *** ccamposr__ has joined #openstack-dev | 18:17 | |
| *** dancn has quit IRC | 18:20 | |
| *** ralonsoh has quit IRC | 18:20 | |
| *** ccamposr__ has quit IRC | 18:21 | |
| *** ccamposr has quit IRC | 18:23 | |
| *** dancn has joined #openstack-dev | 18:25 | |
| *** markmcclain has quit IRC | 18:27 | |
| *** markmcclain has joined #openstack-dev | 18:29 | |
| *** priteau has joined #openstack-dev | 18:33 | |
| *** priteau has quit IRC | 18:38 | |
| *** spsurya_ has quit IRC | 19:08 | |
| *** priteau has joined #openstack-dev | 19:09 | |
| *** shaner has joined #openstack-dev | 19:09 | |
| *** priteau has quit IRC | 19:14 | |
| *** diablo_rojo has quit IRC | 19:16 | |
| *** diablo_rojo has joined #openstack-dev | 19:17 | |
| *** Lucas_Gray has joined #openstack-dev | 19:20 | |
| *** markmcclain has quit IRC | 19:22 | |
| *** markmcclain has joined #openstack-dev | 19:24 | |
| *** Lucas_Gray has quit IRC | 19:26 | |
| *** __ministry has quit IRC | 19:26 | |
| *** Lucas_Gray has joined #openstack-dev | 19:27 | |
| *** Lucas_Gray has quit IRC | 19:53 | |
| *** gfidente is now known as gfidente|afk | 19:53 | |
| *** priteau has joined #openstack-dev | 19:54 | |
| *** tonythomas has joined #openstack-dev | 19:55 | |
| *** armstrong has quit IRC | 19:57 | |
| *** tesseract has quit IRC | 20:00 | |
| *** priteau has quit IRC | 20:05 | |
| *** tesseract has joined #openstack-dev | 20:06 | |
| *** dancn has quit IRC | 20:08 | |
| *** tesseract has quit IRC | 20:17 | |
| *** Lucas_Gray has joined #openstack-dev | 20:23 | |
| *** raildo_ has joined #openstack-dev | 20:24 | |
| *** raildo has quit IRC | 20:26 | |
| *** Lucas_Gray has quit IRC | 20:28 | |
| *** dancn has joined #openstack-dev | 20:32 | |
| *** dancn has quit IRC | 20:40 | |
| *** jtomasek has quit IRC | 20:41 | |
| *** markmcclain has quit IRC | 20:42 | |
| *** markmcclain has joined #openstack-dev | 20:42 | |
| *** owalsh is now known as owalsh_afk | 20:43 | |
| *** dancn has joined #openstack-dev | 20:45 | |
| *** dancn has quit IRC | 20:52 | |
| *** dancn has joined #openstack-dev | 21:00 | |
| *** Lucas_Gray has joined #openstack-dev | 21:09 | |
| *** dlbewley has joined #openstack-dev | 21:11 | |
| *** Lucas_Gray has quit IRC | 21:17 | |
| *** Lucas_Gray has joined #openstack-dev | 21:18 | |
| *** Lucas_Gray has quit IRC | 21:38 | |
| *** Lucas_Gray has joined #openstack-dev | 21:39 | |
| *** tonythomas has quit IRC | 22:05 | |
| *** dancn has quit IRC | 22:13 | |
| *** slaweq has quit IRC | 22:14 | |
| *** nweinber has quit IRC | 22:15 | |
| *** slaweq has joined #openstack-dev | 22:15 | |
| *** slaweq has quit IRC | 22:20 | |
| *** raildo_ has quit IRC | 22:21 | |
| *** tkajinam has joined #openstack-dev | 22:53 | |
| *** Lucas_Gray has quit IRC | 23:01 | |
| *** Lucas_Gray has joined #openstack-dev | 23:02 | |
| *** tosky has quit IRC | 23:08 | |
| *** diablo_rojo has quit IRC | 23:11 | |
| *** Lucas_Gray has quit IRC | 23:18 | |
| *** rloo has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!