Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Welcome to the CollectiveAccess support forum! Here the developers and community answer questions related to use of the software. Please include the following information in every new issue posted here:

  1. Version of the software that is used, along with browser and version

  2. If the issue pertains to Providence, Pawtucket or both

  3. What steps you’ve taken to try to resolve the issue

  4. Screenshots demonstrating the issue

  5. The relevant sections of your installation profile or configuration including the codes and settings defined for your local elements.


If your question pertains to data import or export, please also include:

  1. Data sample

  2. Your mapping


Answers may be delayed for posts that do not include sufficient information.

Howto Import PastPerfect data into Collective Access?

edited November -1 in Troubleshooting
Has anybody successfully imported there data from PastPerfect into Collective Access? Are there any howto guides available?

I have exported the PastPerfect Data into an ASCII comma delimited file for each of the sections PastPerfect exports: Objects, Photos, Archives, Library, Accession, and Contacts.

How do i go about mapping each of those old PastPerfect data sections into the Collective Access DublinCore configuration so MySQL can import the data into the Collective Access database?

Any pointers would be appreciated!
«1

Comments

  • We have written scripts to pull data out of PastPerfect XML files into CA. I can share them with you, but keep in mind that they're tuned to the specifics of the specific users' CA configuration. Since you can configure CA with whatever fields you like there's no standard set mapping between CA and PastPerfect.

    In other words, these scripts will work but you'll have to adjust where it's putting PP data in CA to suit your needs.
  • That would be super! I'm willing to give your scripts a try as our historical society is small county one and we don't have alot in our catalog but i don't want to have to re-enter everything by hand if i don't have to.

    Is there any place i can download the script files or do you need to send them to me via email?
  • Hi, can you please send me the PastPerfect data migration instructions, too? Thanks!
  • I can send you the scripts we use but unless you know how to write PHP code and understand a bit about XML parsing it's not going to be terribly useful. Perhaps we should try and make a project out of this, so we can finally have an established process for migrating PP data to CA.

    To both Rosie and wchs, what parts of PP do you actually use? What is the critical data you are looking to migrate? Are there parts of PP you don't care about? And is there a existing CA configuration (such as DublinCore) that works for you?

    thanks

    seth
  • Seth, i can program in PHP and understand XML parsing so if you send me the scripts i can probably figure it out.

    The sections PastPerfect exported were: Objects, Photos, Archives, Library, Accession, and Contacts. In each of the exported files there was data in a number of columns but most were empty (indicating to me they were not collecting the data under those columns).

    Since PastPerfect is supposedly based on DublinCore i was figuring on trying to use that configuration or the State Historical Society of Pennsylvania contributed configuration to setup our collective access database.
  • Bump. Scripts?
  • I would also love to get a copy of the previously created scripts as well to see if we can modify them for our installation.
  • Hi,

    Here are the scripts. There are six in all, each of which handles a different export file out of PP. Note that these scripts only handle a subset of the exported fields – the ones that mattered for the project at hand (an historical society) – and map them to field in that organization's particular profile. I hope they are useful to you.

    seth
  • Thanks for giving us access to your scripts!

    I tried getting them to work on my localhost server and while my xml file gets read in (there is data in the $vs_xml, $o_xml, $t_lot variables) nothing seems to get into the next section of code:
    $vn_c = 1; //WHAT IS THIS VARIABLE FOR? IT APPEARS NOWHERE ELSE IN THE CODE
    foreach($o_xml->{'export'} as $va_record) //WHAT DOES 'EXPORT' MEAN HERE? NOTHING HAPPENS NO DATA GOES INTO $va_row

    Any help would be appreciated.
  • "export" is the root element of the XML file(s) from PP. The $vn_c variable is a counter we use for debugging purposes. The increment was deleted but that initialization was not. You can discard that line if you like, but it doesn't do anything good or bad at this point, unless you decide you want to keep a count of inserts.
  • OK, i see what the problem is: my xml output is not PP xml output.
    Our PastPerfect would not export the data in their native dublincore xml format; it kept erroring-out. So i got it to export the data in csv format and made xml output from it after i got the csv data into phpMyAdmin.

    Can i get your scripts to work if my xml output begins like what is pasted below? I tried substituting {'photos'} and then {'collection'} as my root element but that did not produce any results.

    If you don't think your scripts will work because the structure of my xml files, is there any other way i can the data into collective access?

    <pma_xml_export version="1.0">
    <!--
    - Database: 'collection'
    -->
    <database name="collection">
    <!-- Table photos -->
    <table name="photos">
    <column name="ACCESSNO">1999.XXXX</column>
    <column name="ACQVALUE">0.000</column>
    <column name="APPNOTES"></column>
    <column name="APPRAISOR"></column>
    <column name="CABINET"></column>
    <column name="CAPTION"></column>
    <column name="CAT">P</column>
    <column name="CATBY"></column>
    <column name="CATDATE">04/20/1999</column>
    <column name="CATTYPE"></column>
    <column name="CLASSES"></column>
    <column name="COLLECTION"></column>
    <column name="CONDDATE">/ /</column>
    <column name="CONDEXAM"></column>
    <column name="CONDITION"></column>
    <column name="CONDNOTES"></column>
    <column name="CONTAINER"></column>
    <column name="COPYRIGHT"></column>
    <column name="CREATOR"></column>
    <column name="CREDIT"></column>
    <column name="CURVALMAX">0.000</column>
    <column name="CURVALUE">0.000</column>
    <column name="DATASET"></column>
    <column name="DATE"></column>
    <column name="DESCRIP">PICTURE OF WHATEVER</column>
    .
    .
    . etc....
  • Does this mean that 1.3 will not have a reasonable to use (non-scripting) import process?
  • There is code towards that goal but it's not ready yet. We're working to finally get 1.3 released with the batch editing tools and new media importer by the end of this month. A v1.4 version will follow that is solely focussed on import/export several weeks after that.
  • Kaial, sounds like the developers have some great plans for the future.

    However, back to my immediate problem: any way to get my data into collective access using the above xml generated from the csv file? Or any other way to transfer the data?
  • You'd have to rewrite the scripts to work with the XML you've synthesized. It's an arbitrary format, and arbitrary formats need code written for them.

    What error is PP giving you when you try and export XML directly?
  • That's what i was afraid of: rewriting; as xml is not my favorite programming language to work with.

    I don't remember the exact phrasing of the PastPerfect error message but it was basically something like, "data error" followed by a hexadecimal number which was less than helpful to figure out what the problem was/is. Plus the error only occurred while trying to export using the DublinCore XML option, as exporting the data as plain old XML or CSV worked.
  • Ok, well if you can't get XML out of PP then you might as well just use the CSV directly. There's no advantage to re-encoding it as XML.

    What profile are you planning to use for your system? Are you writing/customizing your own? The first step is to figure out where each column in that CSV is going to live in your CA system. Then we can figure out how to get it in with the new import system, that v1.4 thing I mentioned above.
  • One other thing i noticed in your import scripts there is a reference to 'ind'(which i believe is meant to be "individual" type_id) in the various scripts like:

    `
    $t_entity->set('type_id', 'ind');
    $vn_entity_id = DataMigrationUtils::getEntityID(DataMigrationUtils::splitEntityName($va_row['recfrom']), 'ind', $pn_locale_id, $va_vals);'

    This reference to 'ind' throws an error msg:
    "entity 'ind' for type_id is not numeric; is invalid"

    So do you know what the numeric value for 'ind' is supposed to be set to?
  • 'ind' is the identifier ("idno") for the "individual" entity type. The v1.3 code in GitHub lets you use these codes in place of numeric ids wherever you need to specify a type_id. It also works for list items and relationship types. Using the codes rather than having to covert codes to numeric ids all over the place makes import code a lot more readable and quicker to write.

    This script is run against a v1.3 installation. I guess you're running this on 1.2. On 1.2 you'd have to look up the type in the database first, converting the idno to a numeric id. The code to do that is something like this:

    $t_list = new ca_lists();
    $vn_entity_type_id = $t_list->getItemIDFromList('entity_types', 'ind');
  • Thanks seth your code helped get the correct type_id for 'ind'. Which is 79 for anybody else that might have to edit the scripts to get them to work.

    Yes, i'm running CA v.1.2 as that is what appears to be available when a person clicks your "Download" button in the main menu of the collective access website. Is version 1.3 the latest stable version?
Sign In or Register to comment.