Author Topic: Import of large model from an external source  (Read 545 times)

michielper

  • EA User
  • **
  • Posts: 105
  • Karma: +1/-0
    • View Profile
Import of large model from an external source
« on: April 11, 2019, 09:12:26 pm »
I want to import a very large class model, consisting of 100.000+ elements and relationships from a non-EA SQL database into Sparx EA. What is the best way to do this?
I know there is a VBA importer to import from an Excel spreadsheet (written by Geert Bellekens) but that is probably not scalable enough. My idea is that it should be possible to create an XML file that can be imported as XMI. Is this possible? And what exactly is the format I must use?
And what about the relationships? All elements have of course already a unique ID, how can I use that to create the right relationships in EA? EA will want to assign its own GUID of course and the relationships must contain the EA GIUD, not the original ID.

Any help or suggestions are highly appreciated!

qwerty

  • EA Guru
  • *****
  • Posts: 10270
  • Karma: +220/-177
  • I'm no guru at all
    • View Profile
Re: Import of large model from an external source
« Reply #1 on: April 11, 2019, 10:04:23 pm »
I suggest to hire Geert to do that for you.

q.

Arshad

  • EA User
  • **
  • Posts: 176
  • Karma: +10/-1
    • View Profile
Re: Import of large model from an external source
« Reply #2 on: April 11, 2019, 10:22:44 pm »
Give a Try with

Sparx Excel Importer in MDG Integration For Microsoft Office


  • Create all the objects in a single click
  • Create Diagrams in each package level
  • Create connectors based on the profile definition
  • Profile definition based import
  • Live progress bar showing the status of the import
  • Synchronization


Geert Bellekens

  • EA Guru
  • *****
  • Posts: 9148
  • Karma: +240/-27
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Import of large model from an external source
« Reply #3 on: April 11, 2019, 10:31:25 pm »
I wouldn't try to go through xmi. That really isn't the most easy format to use.

There are a few options:

- Use the excel importer to import classes and attributes. (Excel scales pretty good and goes up to 1M of rows)  You might want to write additional code to import datatypes (and link them to the attributes) and relations.

- Write a custom import program that reads your source database and creates the corresponding model in EA.

I can't image it being more than a few days of work to get it all done.

If you want to hire me as qwerty suggested (thanks qwerty ;)) you can email me at geert@bellekens.com. But please take into account that I have a pretty full calendar already. Next available days at the moment are in July.

Geert

michielper

  • EA User
  • **
  • Posts: 105
  • Karma: +1/-0
    • View Profile
Re: Import of large model from an external source
« Reply #4 on: April 11, 2019, 10:56:47 pm »
I wouldn't try to go through xmi. That really isn't the most easy format to use.

There are a few options:

- Use the excel importer to import classes and attributes. (Excel scales pretty good and goes up to 1M of rows)  You might want to write additional code to import datatypes (and link them to the attributes) and relations.

- Write a custom import program that reads your source database and creates the corresponding model in EA.

I can't image it being more than a few days of work to get it all done.

If you want to hire me as qwerty suggested (thanks qwerty ;)) you can email me at geert@bellekens.com. But please take into account that I have a pretty full calendar already. Next available days at the moment are in July.

Geert

Thanks for these suggestions. This does not sound as a very easy job.

We are in fact investigating the use of EA for the maintenance of a complex model instead of the current text-based interface. The advantage of EA is its graphical representation of the data but getting the data in EA and creating useful diagrams is then a new challenge.

But is EA in itself capable of handling a model with say 200.000 elements and relations?

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 9148
  • Karma: +240/-27
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Import of large model from an external source
« Reply #5 on: April 12, 2019, 12:39:02 am »
I wouldn't try to go through xmi. That really isn't the most easy format to use.

There are a few options:

- Use the excel importer to import classes and attributes. (Excel scales pretty good and goes up to 1M of rows)  You might want to write additional code to import datatypes (and link them to the attributes) and relations.

- Write a custom import program that reads your source database and creates the corresponding model in EA.

I can't image it being more than a few days of work to get it all done.

If you want to hire me as qwerty suggested (thanks qwerty ;)) you can email me at geert@bellekens.com. But please take into account that I have a pretty full calendar already. Next available days at the moment are in July.

Geert

Thanks for these suggestions. This does not sound as a very easy job.

We are in fact investigating the use of EA for the maintenance of a complex model instead of the current text-based interface. The advantage of EA is its graphical representation of the data but getting the data in EA and creating useful diagrams is then a new challenge.

But is EA in itself capable of handling a model with say 200.000 elements and relations?
Not "very easy" but not that difficult either.
Yes, EA scales pretty well up to hundreds of thousands of elements. (thanks to the database backend)

Geert

michielper

  • EA User
  • **
  • Posts: 105
  • Karma: +1/-0
    • View Profile
Re: Import of large model from an external source
« Reply #6 on: April 12, 2019, 01:03:49 am »
I wouldn't try to go through xmi. That really isn't the most easy format to use.

There are a few options:

- Use the excel importer to import classes and attributes. (Excel scales pretty good and goes up to 1M of rows)  You might want to write additional code to import datatypes (and link them to the attributes) and relations.

- Write a custom import program that reads your source database and creates the corresponding model in EA.

I can't image it being more than a few days of work to get it all done.

If you want to hire me as qwerty suggested (thanks qwerty ;)) you can email me at geert@bellekens.com. But please take into account that I have a pretty full calendar already. Next available days at the moment are in July.

Geert

Thanks for these suggestions. This does not sound as a very easy job.

We are in fact investigating the use of EA for the maintenance of a complex model instead of the current text-based interface. The advantage of EA is its graphical representation of the data but getting the data in EA and creating useful diagrams is then a new challenge.

But is EA in itself capable of handling a model with say 200.000 elements and relations?
Not "very easy" but not that difficult either.
Yes, EA scales pretty well up to hundreds of thousands of elements. (thanks to the database backend)

Geert

Ok, so this might be doable with the XLS-importer? I obtained it from your website but I did not see how it can also import relationships.

Geert Bellekens

  • EA Guru
  • *****
  • Posts: 9148
  • Karma: +240/-27
  • Make EA work for YOU!
    • View Profile
    • Enterprise Architect Consultant and Value Added Reseller
Re: Import of large model from an external source
« Reply #7 on: April 12, 2019, 01:41:05 am »
Ok, so this might be doable with the XLS-importer? I obtained it from your website but I did not see how it can also import relationships.
It can't. As I mentioned it only does elements and attributes. You'll need to write some additional code to link attributes to datatypes, and add relations.

Geert

Giuseppe Platania

  • EA User
  • **
  • Posts: 53
  • Karma: +2/-0
  • As above so below
    • View Profile
    • Linkedin
Re: Import of large model from an external source
« Reply #8 on: May 23, 2019, 05:05:50 am »
At Sparx Services North America we have created ModelFlow to solve this kind of complex issues. For example we have recently imported the complete TM Forum Frameworx industry model from a proprietary DB into EA.

 https://sparxsystems.us/home/software/modelflow/
Giuseppe Platania
Enterprise Architect
+++ +++ +++ +++ +++ +++ ++++
"As above so below, to accomplish the miracles of the One Thing" - Hermes Trismegistus, first Enterprise Architect

Sunshine

  • EA User
  • **
  • Posts: 819
  • Karma: +62/-4
  • Its the results that count
    • View Profile
Re: Import of large model from an external source
« Reply #9 on: May 23, 2019, 09:03:52 am »
At Sparx Services North America we have created ModelFlow to solve this kind of complex issues. For example we have recently imported the complete TM Forum Frameworx industry model from a proprietary DB into EA.

 https://sparxsystems.us/home/software/modelflow/
Excellent.
Are you planning on supporting exchanges with Power Designer in the future?
Happy to help
:)

Giuseppe Platania

  • EA User
  • **
  • Posts: 53
  • Karma: +2/-0
  • As above so below
    • View Profile
    • Linkedin
Re: Import of large model from an external source
« Reply #10 on: May 24, 2019, 02:04:31 am »
Quote
Excellent.
Are you planning on supporting exchanges with Power Designer in the future?

It depends from the format. Following Wikipedia:
Quote
PowerDesigner stores models using a variety of file extensions, such as .bpm, .cdm and .pdm. The internal file structure can be either XML or a compressed binary file format. PowerDesigner can also store models in a database repository.
PowerDesigner as a XML Eclipse file would be different that the DB version.
Do you have a specific  case in mind?
Giuseppe Platania
Enterprise Architect
+++ +++ +++ +++ +++ +++ ++++
"As above so below, to accomplish the miracles of the One Thing" - Hermes Trismegistus, first Enterprise Architect

Guillaume

  • EA User
  • **
  • Posts: 723
  • Karma: +21/-0
    • View Profile
    • www.umlchannel.com
Re: Import of large model from an external source
« Reply #11 on: May 27, 2019, 05:46:46 pm »
Ok, so this might be doable with the XLS-importer? I obtained it from your website but I did not see how it can also import relationships.
It can't. As I mentioned it only does elements and attributes. You'll need to write some additional code to link attributes to datatypes, and add relations.

Geert

To confirm Geert suggestion, I came across a similar issue where we had to import relationships and link attributes data types for classes & attributes to import from Excel in EA:
- For the relationships, details were imported via custom tagged values that were later processed via a script to create/update the appropriate relationships
- Another script was written to search for Data Types in a designed package and replace the text value with a link on the matching data type (class)
It took a reasonable amount of time to write and test.
Guillaume

Blog: www.umlchannel.com | Free utilities addin: www.eautils.com


Giuseppe Platania

  • EA User
  • **
  • Posts: 53
  • Karma: +2/-0
  • As above so below
    • View Profile
    • Linkedin
Re: Import of large model from an external source
« Reply #12 on: May 27, 2019, 10:27:19 pm »
Ok, so this might be doable with the XLS-importer? I obtained it from your website but I did not see how it can also import relationships.
It can't. As I mentioned it only does elements and attributes. You'll need to write some additional code to link attributes to datatypes, and add relations.

Geert

To confirm Geert suggestion, I came across a similar issue where we had to import relationships and link attributes data types for classes & attributes to import from Excel in EA:
I try in the past to use this approach and requires lot of manual work.
This is where ModelFlow shines: 
  • Can import from XML, ODBC, OLEDB, CSV
  • Can import any MDG technology
  • import of Diagrams, Diagrams elements,  relationships and so on
  • Can create complex relationships
  • Supports the concept of owned element
You can achieve the same with lot of scripting and or SQL but with ModelFlow import of complex models requires hours and not minutes.
Giuseppe Platania
Enterprise Architect
+++ +++ +++ +++ +++ +++ ++++
"As above so below, to accomplish the miracles of the One Thing" - Hermes Trismegistus, first Enterprise Architect

Sunshine

  • EA User
  • **
  • Posts: 819
  • Karma: +62/-4
  • Its the results that count
    • View Profile
Re: Import of large model from an external source
« Reply #13 on: May 29, 2019, 07:42:38 am »
Quote
Excellent.
Are you planning on supporting exchanges with Power Designer in the future?

It depends from the format. Following Wikipedia:
Quote
PowerDesigner stores models using a variety of file extensions, such as .bpm, .cdm and .pdm. The internal file structure can be either XML or a compressed binary file format. PowerDesigner can also store models in a database repository.
PowerDesigner as a XML Eclipse file would be different that the DB version.
Do you have a specific  case in mind?
Yes we have some data architects using power designer to create conceptual, logical and physical models and I'd like to import into Sparx EA to map the various behaviour elements.
I understand they are .cdm, ldm, pdm formats.
Happy to help
:)

Glassboy

  • EA Practitioner
  • ***
  • Posts: 1253
  • Karma: +97/-75
    • View Profile
Re: Import of large model from an external source
« Reply #14 on: May 29, 2019, 08:40:43 am »
Yes we have some data architects using power designer to create conceptual, logical and physical models and I'd like to import into Sparx EA to map the various behaviour elements.
I understand they are .cdm, ldm, pdm formats.

I wouldn't consider the PowerDesigner conceptual and logical models to anyway similar to those types of models you'd create in EA.