Author Topic: Native XML import hangs  (Read 5688 times)

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 11315
  • Karma: +422/-33
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Native XML import hangs
« on: March 24, 2021, 07:40:13 pm »
I'm using EA v15.2.1558 and wanted to copy a data model from one repository to another.

The data model has about 400 Classes, Enumerations and Datatypes, and about 4000 attributes.
So it's not a small model, but definitely not an extremely large model.
An earlier version of this model already exists in the target repository.

Since I've heard all kinds of good things about the Native xml format I thought I'd give that a try.
Exporting to xml was fairly quick and resulted in a 53 MB file.

Importing this file into the other repository however took ages and seemed to be stuck on importing connectors.
After about 15 minutes I killed the process.

Then I went back to using XMI 1.1 for the export/import.

Anyone else seeing this? Am I doing something wrong? Are there cases in which the new native xml format should be avoided?

Geert

Roman Liebsch

  • EA Novice
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Native XML import hangs
« Reply #1 on: March 24, 2021, 11:54:57 pm »
Hi Geert,

I faced those issues very often using the cloud connection either with XMI 1.1, Asset Service (=XMI) or Native. I have already reported it to Sparx end of 2019.
Shortly, for those big imports where a lot of references are missing in the new repos, using cloud connection will fail on our side with a timeout.
However in the intranet (not via VPN) it is working most of the times with a direct ODBC mySQL connection. Native is working only with ODBC mySQL and EA1558 - I used it last week that way quite successfully.
For me it is a bug introduced in EA14 handling unresolvable relationships together with the cloud connection backend, so I use ODBC only for that.

Here my bug description, maybe it helps you investigating:

Summary:
EA crashes with data loss on bigger transaction (XMI Import, Baseline...)
Details:
If I want to exchange bigger data among different repositories than I use first an XMI export with Version 1.1(in EA15 I also tried "Native") that is working so far.
But the import is crashing or hanging when using larger files and often when there are a lot of references not solvable as I update a sub package of a larger context.
So I will have an inconsistent project afterwards.
I often get an WAIT_TIMEOUT from the Cloud Service, based on a OBDC Warning of a found Deadlock.
"Pro Cloud Server: Query in batch execute failed with message: FAIL: Microsoft OLE DB Provider for ODBC Drivers [-2147467259] [MySQL][ODBC 8.0(w) Driver][mysqld-8.0.19]Deadlock found when trying to get lock; try restarting transaction"
Either EA will hang (and GUI will soften in the background) at the import stage "Resolving Link" or "Fix Up external References" for example, the adding stage is often not critical.
Or I will get the message OLED Persistence Provider : ....not enough Memory for the transaction that ends in an "Out of Memory" error of EA. Then EA will not react anymore.
I guess it is even worse if I keep on working with other programs and the focus to the EA GUI is lost.
I is a little bit more stable using a direct OBDC connection from the client instead of the Cloud connection but it crashes likewise.

However if I use the EA1354 the deadlock warning still appears from time to time but EA13 will keep on importing and fixing all references and so right without and Out of Memory error.

Sparx Support answer:
When you were importing the XML file(s) and receiving the deadlock error, did the folder containing the information being imported already exist in the repository?  The only time we were able to duplicate that error is when EA had to delete the previous information before doing the import, this occurs for all XMI 1.1 imports.  We will need to investigate the cause of these deadlocks.

One of your comments mentioned Native imports.  Can you round trip Native XML export/imports without errors?  We ask because the Native XML import does not delete previous information before importing, it simple updates any data already present.  The Native XML export/import has been designed from the ground up to be as fast as possible by reducing the amount of data being transferred over the network which greatly improves performance.

Can you described your environment in particular the location of the Pro Cloud and Database servers?  The communications between EA and the Pro Cloud Server have been optimized however the Pro Cloud Server requires/assumes it has near immediate access to the database server, and will open 100's (if not 1000's) of database queries, therefore if the PCS is located in one network and the database server in another general performance will be impacted and in the worst cases, timeouts will occur.


Regards
Roman

qwerty

  • EA Guru
  • *****
  • Posts: 12446
  • Karma: +348/-291
  • I'm no guru at all
    • View Profile
Re: Native XML import hangs
« Reply #2 on: April 17, 2021, 07:07:14 am »
I had a serious issue with that native import. It just fucked up my repo so I had to go to the backup. I will touch it again earliest in 2 years from now.

q.
« Last Edit: September 07, 2021, 01:25:41 am by qwerty »

MatthiasG

  • EA Novice
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Native XML import hangs
« Reply #3 on: September 06, 2021, 06:56:52 pm »
Hi Gert and Q,

just a measurement for the native XMI import/export on my side. I am using EA1522 with an eap-file. The XML-file is about 8MB in size.
XMI1.1: export: 1:03min import: 1:15min
Native: export: 0:04min import: 4:50min (yes much slower!)

With this figures, the native XMI is ruled out because of the import time.
And with about 80 XMIs in our model we need at least 100min for XMI1.1 or >400min (estimated) with native XMI.

regards,
Matthias
« Last Edit: September 09, 2021, 09:51:19 pm by MatthiasG »

qwerty

  • EA Guru
  • *****
  • Posts: 12446
  • Karma: +348/-291
  • I'm no guru at all
    • View Profile
Re: Native XML import hangs
« Reply #4 on: September 07, 2021, 01:27:15 am »
So the native import is even slower? (As said, I only used it for a short test which mad me put it aside immediately)

q.

Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8001
  • Karma: +211/-127
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: Native XML import hangs
« Reply #5 on: September 07, 2021, 09:59:44 am »
Hi Mathias,

Are these initial loads or re-loads?  We found that Native Import was OK and fast for initial loads, but had problems with reloads.  So we stopped using it for those.  We were waiting for the issues to be fixed in a future release.

Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

Guillaume

  • EA Practitioner
  • ***
  • Posts: 1088
  • Karma: +34/-2
    • View Profile
    • www.umlchannel.com
Re: Native XML import hangs
« Reply #6 on: September 08, 2021, 10:48:02 pm »
I find the following explanation interesting as I didn't expect it to work this way.

Sparx Support answer:
... the Native XML import does not delete previous information before importing, it simple updates any data already present.  The Native XML export/import has been designed from the ground up to be as fast as possible by reducing the amount of data being transferred over the network which greatly improves performance.

Since the purpose of the Native XML is to replace/load all the data, shouldn't it make sense to provide an option to clear the data from all tables (same as the Project Transfer) if it improves the time to run a Native XML import ?
Guillaume

Blog: www.umlchannel.com | Free utilities addin: www.eautils.com


Paolo F Cantoni

  • EA Guru
  • *****
  • Posts: 8001
  • Karma: +211/-127
  • Inconsistently correct systems DON'T EXIST!
    • View Profile
Re: Native XML import hangs
« Reply #7 on: September 08, 2021, 11:29:01 pm »
I find the following explanation interesting as I didn't expect it to work this way.

Sparx Support answer:
... the Native XML import does not delete previous information before importing, it simply updates any data already present.  The Native XML export/import has been designed from the ground up to be as fast as possible by reducing the amount of data being transferred over the network which greatly improves performance.
Since the purpose of the Native XML is to replace/load all the data, shouldn't it make sense to provide an option to clear the data from all tables (same as the Project Transfer) if it improves the time to run a Native XML import ?
I didn't expect it to work otherwise.  It's only transferring PART of a repository.  Project Transfer transfers the ENTIRE repository.  It has to clear the old data before it starts.


Paolo
Inconsistently correct systems DON'T EXIST!
... Therefore, aim for consistency; in the expectation of achieving correctness....
-Semantica-
Helsinki Principle Rules!

qwerty

  • EA Guru
  • *****
  • Posts: 12446
  • Karma: +348/-291
  • I'm no guru at all
    • View Profile
Re: Native XML import hangs
« Reply #8 on: September 09, 2021, 04:52:22 am »
If one needs a project transfer then why not use the project transfer which is already there. More native isn't probably possible (except you copy whole EAP files).

q.

MatthiasG

  • EA Novice
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Native XML import hangs
« Reply #9 on: September 09, 2021, 09:43:54 pm »
Hi Mathias,

Are these initial loads or re-loads?  We found that Native Import was OK and fast for initial loads, but had problems with reloads.  So we stopped using it for those.  We were waiting for the issues to be fixed in a future release.

Paolo

Hi Paolo,

these where re-loads. With all the comments given here we will not switch to native-XML and wait for further updates.

Matthias