2024 06 21 adapt filtering export selection

[sjib]

updating configuration from qgep QGEP/qgepqwat2ili#144

[sjib]

E sqlalchemy.exc.InvalidRequestError: Don't know how to join to <Mapper at 0x7f0285318af0; throttle_shut_off_unit>. Please use the .select_from() method to establish an explicit left side, as well as providing an explicit ON clause if not present already to help resolve the ambiguity

[sjib]

Ein Fehler trat bei der Ausführung von Python-Code auf: 

sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) FEHLER:  Einf��gen oder Aktualisieren in Tabelle ��anschlussobjekt�� verletzt Fremdschl��ssel-Constraint ��anschlussobjekt_abwassernetzelementref_fkey�� DETAIL:  Schl��ssel (abwassernetzelementref)=(7377) ist nicht in Tabelle ��abwassernetzelement�� vorhanden. (Background on this error at: http://sqlalche.me/e/13/gkpj) 
Traceback (most recent call last):
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 771, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\default.py", line 561, in do_commit
    dbapi_connection.commit()
psycopg2.errors.ForeignKeyViolation: FEHLER:  Einf��gen oder Aktualisieren in Tabelle ��anschlussobjekt�� verletzt Fremdschl��ssel-Constraint ��anschlussobjekt_abwassernetzelementref_fkey��
DETAIL:  Schl��ssel (abwassernetzelementref)=(7377) ist nicht in Tabelle ��abwassernetzelement�� vorhanden.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\teksi_wastewater_plugin.py", line 470, in actionExportClicked
    self.interlisImporterExporter.action_export()
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\gui\interlis_importer_exporter_gui.py", line 141, in action_export
    raise exception
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\gui\interlis_importer_exporter_gui.py", line 111, in action_export
    self.interlis_importer_exporter.interlis_export(
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\interlis_importer_exporter.py", line 192, in interlis_export
    self._export_to_intermediate_schema(
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\interlis_importer_exporter.py", line 383, in _export_to_intermediate_schema
    twwInterlisExporter.tww_export()
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_exporter_to_intermediate_schema.py", line 87, in tww_export
    raise exception
  File "C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\interlis_model_mapping\interlis_exporter_to_intermediate_schema.py", line 83, in tww_export
    self.abwasser_session.commit()
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\orm\session.py", line 1046, in commit
    self.transaction.commit()
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\orm\session.py", line 508, in commit
    t[1].commit()
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 1762, in commit
    self._do_commit()
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 1793, in _do_commit
    self.connection._commit_impl()
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 773, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\util\compat.py", line 182, in raise_
    raise exception
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\base.py", line 771, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages\sqlalchemy\engine\default.py", line 561, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) FEHLER:  Einf��gen oder Aktualisieren in Tabelle ��anschlussobjekt�� verletzt Fremdschl��ssel-Constraint ��anschlussobjekt_abwassernetzelementref_fkey��
DETAIL:  Schl��ssel (abwassernetzelementref)=(7377) ist nicht in Tabelle ��abwassernetzelement�� vorhanden.

(Background on this error at: http://sqlalche.me/e/13/gkpj)


Python-Version: 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] 
QGIS-Version: 3.28.4-Firenze Firenze, fd0fb72ab3 

Python-Pfad:
C:/PROGRA~1/QGIS 3.28.4/apps/qgis-ltr/./python
C:/Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python
C:/Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins
C:/PROGRA~1/QGIS 3.28.4/apps/qgis-ltr/./python/plugins
C:\Program Files\QGIS 3.28.4\bin\python39.zip
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\DLLs
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\lib
C:\Program Files\QGIS 3.28.4\bin
C:\Users\Stefan\AppData\Roaming\Python\Python39\site-packages
C:\PROGRA~1\QGIS 3.28.4\apps\Python39
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\lib\site-packages
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\lib\site-packages\win32
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\lib\site-packages\win32\lib
C:\PROGRA~1\QGIS 3.28.4\apps\Python39\lib\site-packages\Pythonwin
C:/Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python
C:/Daten/QGEP/testing_2024.0.rc2/project-translations
C:\Users/Stefan/AppData/Roaming/QGIS/QGIS3\profiles\tww_2024.0.rc2/python/plugins\teksi_wastewater\interlis\gui\..\..
.

But the wastewater_network exists in the data:

Could the missing reference fk_wws be a problem? If yes why?

[sjib]

The problem is with the data:

'ch080qwzRV001266' exists only in tww_od.wastewater_networkelement, but not in one of the subclasses wwn or reach

This could have been caused by the import when we still had the class definition error of connection_object and the reservoir dataset was added to wastewater_network and to reservoir (because this was thought to be the subclass).

```
	<DSS_2020_1_LV95.Siedlungsentwaesserung.Reservoir TID="ch080qwzRV001266">
			<Letzte_Aenderung>20191014</Letzte_Aenderung>
			<DatenherrRef REF="ch080qwzPR000018"/>
			<DatenlieferantRef REF="ch080qwzPR000017"/>

			<Bezeichnung>Reservoir</Bezeichnung>
			<Fremdwasseranfall>5.000</Fremdwasseranfall>
			<Lage>
				<COORD>
					<C1>2645300.000</C1>
					<C2>1153200.000</C2>
				</COORD>
			</Lage>
			<Standortname>Berg</Standortname>
		</DSS_2020_1_LV95.Siedlungsentwaesserung.Reservoir>

[sjib]

We should add the subclass checks also for connection_object, this would have helped a lot to find that error!

see https://github.com/QGEP/qgepqwat2ili/blob/9416466019d4e35f1342dad8f014ccf9e5609286/qgepqwat2ili/utils/ili2db.py#L50

[sjib]

@urskaufmann The following data has to be deleted in wastewater_networkelement (all wrongly imported reservoir, fountain, individual_surface and building, see explanation above):

Then the export works - can you test and confirm?

[urskaufmann]

Yes it works (after setting all missing fk_dataowner... ), when Export is for all records. (not for a selection).

[cymed]

This PR also solves an issue I just encountered: If the dataset has an connection_object.identifier and there exists a networkelement_identifier with the same identifier, we get an error. Thanks to the alteration in model_tww_od.py this bug is fixed

[ponceta]

Let me know if we can go further with this.

[sjib]

@ponceta I cleaned the code from all unnecessary comments. Once the tests are run thru and ok you can merge and we have a working version.

Further improvements to this are described here #280 and can be added with 2024.1