Browsing "Older Posts"

Browsing Category "XML"

Importing XML file with invalid character 22 (U+0016)

Von Tobias Arnhold → 1.15.2016
I have to import a set of XML files from time to time. Most of those XML files can be imported with out any problems. But at least one file includes a special character U+0016 which occurs randomly some where inside the file.

When I try to import that file I get this ORA- error message:
ORA-31011: XML-Parsing nicht erfolgreich
ORA-19202: Fehler bei XML-Verarbeitung
LPX-00217: Ungültiges Zeichen 22 (U+0016)
Error at line 39409 aufgetreten

Die besten HowTo's rund um fortgeschrittene SQL-Techniken

Von Tobias Arnhold → 9.25.2015
Im Zuge des aktuellen Community Tips, möchte ich mal eine Liste an HowTo's, Tips und Tricks rund um die verschiedenen aktuellen
SQL-Techniken veröffentlichen. Dabei habe ich mich möglichst auf deutsche Lektüre fokussiert. Am häufigsten habe ich Artikel von Carsten Czarski gefunden, was wohl wenig verwunderlich ist. :)
Die folgenden Links richten sich an jene Entwickler, die mehr als SQL-92 aus Ihrer Oracle Datenbank rausholen möchten.


Analytische SQL-Funktionen
 - Daten zusammenfassen mit Aggregatsfunktionen
 - Mächtige Berichte mit "Analytischen Funktionen"
 - Analytische Funktionen - eine Einführung
 - Analytische Funktionen - Einführung
 - Einsatz analytischer Funktionen
 - Probably the Coolest SQL Feature: Window Functions (Englisch)

Volltextsuche
 - Ähnlichkeitssuche in SQL-Abfragen: Oracle TEXT für alle!
 - Volltextrecherche und mehr: Wozu man Oracle Text nutzen kann
 - Oracle Text - Einführung
 - Ähnlichkeitssuche mit Oracle Text
 - "Unscharfe Suche" in Datenbeständen

Räumliche Abfragen (Geodaten)
 - oracle-spatial.blogspot.de
 - spatialdbadvisor.com (Englisch)
 - SDO_GEOMETRY für Newbies
 - Was kann der Oracle Locator (Englisch)

XML-Funktionen
 - Working with XML files and APEX - Part 1: Upload (Englisch)
 - Working with XML files and APEX - Part 2: Selecting data from XMLType column with XMLTable (Englisch)
 - XML-Dokumente relational aufbereiten: mit SQL!
 - XML-Strukturen analysieren - mit SQL!

JSON-Funktionen
 - APEX_JSON ohne APEX: JSON parsen mit SQL und PL/SQL
 - APEX_JSON ohne APEX: JSON erzeugen mit SQL und PL/SQL

 - Native JSON-Unterstützung in der Oracle-Datenbank 12c (12.1.0.2)
 - Oracle 12.1.0.2 ist da: Ein Einblick in die JSON-Unterstützung
 - Speed Up Your APEX apps with JSON and Mustache.js

SQL Pattern Matching (ab 12.1)
 - Datenmuster finden: SQL Pattern Matching in Oracle12c
 - Oracle12c: Was für SQL- und PL/SQL-Entwickler einfacher wird
 - Pattern Matching (MATCH_RECOGNIZE) in Oracle Database 12c Release 1 (Englisch)

SQL MODEL-Klausel
 - Kalkulationen, Simulationen und mehr - in APEX: SQL MODEL Klausel
 - SQL Model Clause Tutorial (Englisch)
 - Rechnen wie in Excel - in einer SQL-Abfrage
 - SQL Model Clause Tutorial
 - Model Klausel - Der Excel-Killer von Oracle?
 - Forum Lösung - Gantt Chart überschneidende Einträge (Englisch)

Stringverkettungen
 - LISTAGG function in 11g release 2 (Englisch)
 - SQL LISTAGG mit CLOB-Ausgabe - kein VARCHAR2-Limit mehr
 - XML CLOB Trick - Custom CSV Export in APEX


String to Table
 - Split comma delimited strings in a table in Oracle
 - emulating string-to-table functionality using sql
 - When APEX_UTIL.STRING_TO_TABLE is not enough

Ähnlichkeitssuche (Beispiel: Jaro Winkler)
 - UTL_MATCH : String Matching by Testing Levels of Similarity/Difference (Englisch)
 - Oracle UTL_MATCH (Englisch)

Hierarchische Abfragen (CONNECT BY)
 - Treewalking mit SQL (Hierarchische Strukturen) 
 - Oracle: connect by (Hierarchische Queries)
 - Hierarchische Abfragen in Oracle11g Rel2: Recursive WITH clause 

Regular Expressions
 - Reguläre Ausdrücke in der Oracle Datenbank in SQL verwenden
 - Reguläre Ausdrücke ... in der Datenbank!
 - Reguläre Ausdrücke in Application Express 
 - Regular Expressions in Oracle

Kreuztabellen (Pivot / Unpivot)
 - Kreuztabellen in Oracle11g: SQL PIVOT-Klausel
 - Pivoting in 11g - Zeilen zu Spalten
 - pivot and unpivot queries in 11g (Englisch)
 - Dynamic SQL Pivoting – Stealing Anton’s Thunder (Englisch)
 - APEX Beispielanwendung

Zufallszahlen
 - Arbeiten mit DBMS_CRYPTO
 - Generating random numbers and strings in Oracle (Englisch)
 - Oracle DBMS_CRYPTO (Englisch) und Oracle DBMS_RANDOM (Englisch)

Zeilengenerierung
 - Oracle Row Generator Techniques (Englisch)

Working with XML files and APEX - Part 3: Detail elements in a row with OUTER JOIN

Von Tobias Arnhold → 1.08.2015
After I described how to successfully import XML files into an APEX application and gave some basic information about the analyzing of xml data.
It's time for another example. This time we want to get all detail elements of a master element in one row.
Example XML data:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>  
<data>
 <stadium> 
   <team name="SG Dynamo Dresden" league="3. Liga"/>
   <event name="Konzert"/>
   <event name="Stadionführung"/>
 </stadium> 
 <stadium> 
   <team name="RB Leipzig" league="2. Liga"/>
   <event name="Konzert 1"/>
   <event name="Konzert 2"/>
   <event name="Konzert 3"/>
 </stadium> 
</data>
As you see we have two detail entries for element <stadium>. What I want are two rows including all the data of element <stadium>. Of course this means taking off some flexibility from my selection. Because I need to know what data I will have to deal with.
In my case I assume maximum one team and three events per stadium element.

The magic sql statement:
-- table: IMP_FM
-- XMLType column: XML_FILE
select
    t1_team,
    t1_league,
    e1_event,
    e2_event,
    e3_event
from    IMP_FM nx,
        XMLTable('/data/stadium' passing nx.xml_file
                 columns   
                    "P_T1" XMLTYPE PATH 'team[1]',  
                    "P_E1" XMLTYPE PATH 'event[1]',   
                    "P_E2" XMLTYPE PATH 'event[2]',  
                    "P_E3" XMLTYPE PATH 'event[3]' 
                ) STAD,
        XMLTable('team' PASSING STAD.P_T1
                 columns   t1_team varchar2(200) path '@name',
                           t1_league varchar2(200) path '@league'
                ) (+) T1,
        XMLTable('event' PASSING STAD.P_E1
                 columns   e1_event varchar2(200) path '@name'
                ) (+) E1,
        XMLTable('event' PASSING STAD.P_E2
                 columns   e2_event varchar2(200) path '@name'
                ) (+) E2,
        XMLTable('event' PASSING STAD.P_E3
                 columns   e3_event varchar2(200) path '@name'
                ) (+) E3
where nx.id = 1
Inside the sql I first create a set of four detail xml columns per row:
columns
  "P_T1" XMLTYPE PATH 'team[1]',
  "P_E1" XMLTYPE PATH 'event[1]',
  "P_E2" XMLTYPE PATH 'event[2]',
  "P_E3" XMLTYPE PATH 'event[3]'

Next I select the data of each xml element via an OUTER JOIN.
XMLTable('team' PASSING STAD.P_T1
    columns t1_team varchar2(200) path '@name',
                     t1_league varchar2(200) path '@league'
) (+) T1,

Finally I only have to select the columns I want to see.
If I wouldn't use the OUTER JOIN syntax then no result would appear. Because Oracle handles the XMLTable integration as INNER JOIN.
As I told this example only works if you know the exact definition of your XML detail data elements.
APEX-AT-WORK no image

Working with XML files and APEX - Part 2: Selecting data from XMLType column with XMLTable

Von Tobias Arnhold → 12.05.2013
After I described how to successfully import XML files into an APEX application. It's time to start to analyze them.

I still assume that this is the example XML file:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<leagues>
  <league id="1" name="2. Bundeliga">
    <teams>
      <team>
        <id>1</id>
        <name location="Dresden" stadium="Glücksgas-Stadion">SG Dynamo Dresden</name>
      </team>
      <team>
        <id>2</id>
        <name location="Cologne" stadium="RheinEnergieStadion">1. FC Köln</name>
      </team>
      <team>
        <id>3</id>
        <name location="Berlin" stadium="Alte Försterei">1. FC Union Berlin</name>
      </team>
      <team>
        <id>4</id>
        <name location="Düsseldorf" stadium="Esprit Arena">Fortuna Düsseldorf</name>
      </team>
    </teams>
  </league>
</leagues>
The first select will find all team names and locations:
-- table: IMP_FM
-- XMLType column: XML_FILE

SELECT T.TEAM, 
       T.LOCATION
FROM
       IMP_FM FM,
       XMLTable('/leagues/league/teams/team/name' PASSING FM.XML_FILE
       COLUMNS  "TEAM"         VARCHAR2(255) PATH 'text()',
                "LOCATION"     VARCHAR2(255) PATH '@location'
                ) T
What am I doing here?
In the FROM-clause I define the XML table "IMP_FM" and right after that I generate a new XMLTable object. Inside this object I define the start entry "/leagues/league/teams/team/name" from where I want to select the data. In the PASSING-clause I define the XMLType column from where the data is coming.
Next step is to set the SQL columns based on the XML data.
To get the team names I need to select the text from the xml element "name". To do that I use the function "text()".
To select the attribute "location" I need to use an "@" in front of the attribute name.
 This newly created table will have the name "T" and can be selected in the SELECT-clause.

In the next example I will select league and team names. Actually they would be two tables with an 1-n relationship.
Because of that I have to implement two XMLTable objects in the select statement.
SELECT L.LEAGUE,
       T.TEAM, 
       T.LOCATION
FROM
       IMP_FM FM,
       XMLTable('/leagues/league' PASSING FM.XML_FILE
       COLUMNS  "LEAGUE"     VARCHAR2(255) PATH '@name',
                "T_XML"      XMLTYPE       PATH 'teams/team'
                ) L,
       XMLTable('team/name' PASSING L.T_XML
       COLUMNS  "TEAM"         VARCHAR2(255) PATH 'text()',
                "LOCATION"     VARCHAR2(255) PATH '@location'
                ) T
What am I doing here?
In the first XMLTable object I generate a XMLType column with the data from path "teams/team". This column is named as "T_XML". The generated table is named as "L".
Now I create a new XMLTable object based on the data of the XMLType column "L.T_XML". I follow the same logic as in first select and just use a shorter PATH "team/name".
More information about XML sub-selects can be found in this blog post: New 12c XMLTABLE’s “RETURNING SEQUENCE BY REF” clause

In the last example I want to show how to work with XML namespaces. For that our XML changes a bit.
<?xml version="1.0" encoding="UTF-8"?>
<leagues xmlns:tt="http://www.footballleagues_xml.org/schemas/team" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.footballleagues_xml.org/schemas http://www.footballleagues_xml.org/schemas/xml/XML.xsd" version="2.1" xmlns="http://www.footballleagues_xml.org/schemas">
  <league id="1" name="2. Bundeliga">
    <teams>
      <team>
        <tt:id>1</tt:id>
        <tt:name location="Dresden" stadium="Glücksgas-Stadion">SG Dynamo Dresden</tt:name>
      </team>
      <team>
        <tt:id>2</tt:id>
        <tt:name location="Cologne" stadium="RheinEnergieStadion">1. FC Köln</tt:name>
      </team>
      <team>
        <tt:id>3</tt:id>
        <tt:name location="Berlin" stadium="Alte Försterei">1. FC Union Berlin</tt:name>
      </team>
      <team>
        <tt:id>4</tt:id>
        <tt:name location="Düsseldorf" stadium="Esprit Arena">Fortuna Düsseldorf</tt:name>
      </team>
    </teams>
  </league>
</leagues>
To select the data now is a bit more complicated:
SELECT T.TEAM, 
       T.LOCATION
FROM
       IMP_FM FM,
       XMLTable(XMLNameSpaces('http://www.footballleagues_xml.org/schemas/team' as "tt", 
                              default 'http://www.footballleagues_xml.org/schemas'),
               '/leagues/league/teams/team' PASSING FM.XML_FILE
       COLUMNS  "TEAM"         VARCHAR2(255) PATH 'tt:name/text()',
                "LOCATION"     VARCHAR2(255) PATH 'tt:name/@location'
                ) T
What am I doing here?
Inside the XMLTable object I define a "XMLNameSpace" which is marked as "tt".
Inside the column definition I use "tt:" to select the data.
More information about multiple Namespaces can be found here: https://forums.oracle.com/thread/2381998

That's it for today. Cheers Tobias
APEX-AT-WORK no image

Working with XML files and APEX - Part 1: Upload

Von Tobias Arnhold → 11.28.2013
Working with Oracle and XML can be a pain in the ass especially at the beginning when you don't know the hidden secrets. :) That's why I want to give some major hints how to integrate XML files in APEX applications.

This time I will provide an easy way how to upload a file into a table with a XMLType column.

Let's assume that this is our example XML file:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<leagues>
  <league id="1" name="2. Bundeliga">
    <teams>
      <team>
        <id>1</id>
        <name location="Dresden" stadium="Glücksgas-Stadion">SG Dynamo Dresden</name>
      </team>
      <team>
        <id>2</id>
        <name location="Cologne" stadium="RheinEnergieStadion">1. FC Köln</name>
      </team>
      <team>
        <id>3</id>
        <name location="Berlin" stadium="Alte Försterei">1. FC Union Berlin</name>
      </team>
      <team>
        <id>4</id>
        <name location="Düsseldorf" stadium="Esprit Arena">Fortuna Düsseldorf</name>
      </team>
    </teams>
  </league>
</leagues>

Now we need the necessary DDL:

--------------------------------------------------------
--  DDL for XML Table IMP_FM
--------------------------------------------------------
CREATE TABLE IMP_FM 
(
  ID NUMBER NOT NULL 
, USER_NAME VARCHAR2(20 BYTE) 
, XML_FILE XMLTYPE 
, CONSTRAINT IMP_FM_PK PRIMARY KEY 
  (
    ID 
  )
  ENABLE 
) 
XMLTYPE XML_FILE STORE AS BINARY XML ALLOW NONSCHEMA;

/

CREATE SEQUENCE IMP_FM_SEQ INCREMENT BY 1 MAXVALUE 9999999999999999999999999999 MINVALUE 1 NOCACHE;

/

CREATE OR REPLACE TRIGGER "IMP_FM_BI_TR" BEFORE
INSERT ON "IMP_FM" FOR EACH row 
BEGIN
 IF :NEW.USER_NAME IS NULL THEN 
    :NEW.USER_NAME := UPPER(NVL(v('APP_USER'),USER));
 END IF;
 IF :NEW.ID IS NULL THEN 
    SELECT IMP_FM_SEQ.NEXTVAL INTO :NEW.ID FROM DUAL;
 END IF;
END;
/

--------------------------------------------------------
--  DDL for Table ERR_LOG
--------------------------------------------------------

  CREATE TABLE "ERR_LOG" 
   ( "AKTION" VARCHAR2(4000), 
 "APP_ID" NUMBER, 
 "APP_PAGE_ID" NUMBER, 
 "APP_USER" VARCHAR2(10), 
 "ORA_ERROR" VARCHAR2(4000), 
 "CUSTOM_ERROR" VARCHAR2(4000), 
 "PARAMETER" VARCHAR2(4000), 
 "TIME_STAMP" DATE, 
 "PROC_NAME" VARCHAR2(500), 
 "CLOB_FIELD" CLOB
   ) ;
/

  ALTER TABLE "ERR_LOG" MODIFY ("PROC_NAME" NOT NULL ENABLE);
  ALTER TABLE "ERR_LOG" MODIFY ("TIME_STAMP" NOT NULL ENABLE);
  ALTER TABLE "ERR_LOG" MODIFY ("APP_USER" NOT NULL ENABLE);
  ALTER TABLE "ERR_LOG" MODIFY ("APP_PAGE_ID" NOT NULL ENABLE);
  ALTER TABLE "ERR_LOG" MODIFY ("APP_ID" NOT NULL ENABLE);
  ALTER TABLE "ERR_LOG" MODIFY ("AKTION" NOT NULL ENABLE);
/

Our XML table has the column XML_FILE from type XMLTYPE. Thats the place where our uploaded files will be saved in.
With XMLTYPE you need to define the way the XML should be saved. As I found out (thanks to Carsten Czarski) the "BINARY XML" option is the most effective one.

More details about the saving options can be found here:
http://www.oracle.com/technetwork/database-features/xmldb/xmlchoosestorage-v1-132078.pdf
http://www.liberidu.com/blog/2007/06/24/oracle-11g-xmltype-storage-options/
http://grow-n-shine.blogspot.de/2011/11/one-of-biggest-change-that-oracle-has.html

During my tests (30MB XML file) I started with the option "XMLTYPE XML_FILE STORE AS CLOB" which leaded to a real bad result time.
One of the selects had a executing time of 314 seconds
With the "XMLTYPE XML_FILE STORE AS BINARY XML" it went down to 1 second.

Ok my system did not had a lot of CPU or RAM but to show the difference out of a performance point of view it is a great example.

Last but not least we need the PL/SQL Code to upload our XML file:
For quality aspects I always use packages with some debug features when I need PL/SQL code. Thats why this example becomes maybe a bit bigger then it normally would be.

CREATE OR REPLACE 
PACKAGE PKG_IMP AS 

  procedure fm_imp (p_filename varchar2);

END PKG_IMP;

/

CREATE OR REPLACE 
PACKAGE BODY PKG_IMP AS

/* ********************* */
/* Package Variables     */
/* ********************* */
  gv_proc_name    VARCHAR2(100);
  gv_action       VARCHAR2(4000);
  gv_ora_error    VARCHAR2(4000);
  gv_custom_error VARCHAR2(4000);
  gv_parameter    VARCHAR2(4000);
  gv_apex_err_txt VARCHAR2(500);
  

  GV_USERNAME     VARCHAR2(100)     := UPPER(NVL(v('APP_USER'),USER));

/* ********************* */
/* Save errors           */
/* ********************* */
PROCEDURE ADD_ERR  IS
   PRAGMA AUTONOMOUS_TRANSACTION;
BEGIN
     INSERT
     INTO ERR_LOG
      ( PROC_NAME,AKTION,APP_ID,APP_PAGE_ID,APP_USER,ORA_ERROR,CUSTOM_ERROR,PARAMETER,TIME_STAMP )
      VALUES
      ( gv_proc_name,gv_action,nvl(v('APP_ID'),0),nvl(v('APP_PAGE_ID'),0),nvl(GV_USERNAME,'Unknown'),
        gv_ora_error,gv_custom_error,gv_parameter,sysdate );
     COMMIT;
END; 

/* ********************* */
/* Import Procedure      */ 
/* ********************* */

procedure fm_imp (p_filename varchar2) AS
 
  v_blob BLOB;
  v_xml  XMLTYPE;

BEGIN
  gv_proc_name := 'pkg_imp.fm_imp';
  gv_parameter := '';
 
  gv_parameter := 'p_filename: ' || p_filename;
  
  gv_action := 'Delete old data';
  DELETE FROM IMP_FM
   WHERE user_name = GV_USERNAME;
   
 
  gv_action := 'Read file';
  SELECT blob_content 
    INTO v_blob
    FROM wwv_flow_files
   WHERE name = p_filename;
  
  gv_action := 'XML Conversion';
  v_xml := XMLTYPE (v_blob,NLS_CHARSET_ID('AL32UTF8'));
  /* UTF-8 clause because we use it in our XML file */

  gv_action := 'Insert in IMP_BESTELLLISTE_XML';
  INSERT
  INTO IMP_FM
    ( USER_NAME, XML_DATEI )
  VALUES
    ( GV_USERNAME, v_xml );

  gv_action := 'Delete file';
  DELETE FROM wwv_flow_files
    WHERE name = p_filename;

  COMMIT;
  
EXCEPTION
WHEN OTHERS THEN   
      gv_ora_error := SQLERRM;
      gv_custom_error := 'Internal Error. Action canceled.';
      ROLLBACK;
      ADD_ERR; raise_application_error(-20001, gv_custom_error);
          
END fm_imp; 
END PKG_IMP;

Inside APEX you call our procedure as SUBMIT PL/SQL Process:

  PKG_IMP.FM_IMP (:P1_FILE);
  -- P1_FILE is the file browse item.
That's it.

BTW: My SQL Developer stopped working / got really slow when I tried to update a BINARY XML column.

Next time I will write about my troubles in selecting XML data as readable SQL result.
APEX-AT-WORK no image

Two ways using string to table in APEX selects

Von Tobias Arnhold → 5.03.2011
First way
This example uses "REGULAR expressions" and the "connect by" clause. Perfect when you select on columns including lists.
Forum entry: http://forums.oracle.com/forums/thread.jspa?messageID=9494074

WITH TABLE1
AS (SELECT 1 my_id, '8092:7054:9237:4232:3333:4023:6781' my_list FROM DUAL
UNION
SELECT 2, '8765:2231:2242:3412:3453' FROM DUAL
UNION
SELECT 3, '2121' FROM DUAL
UNION
SELECT 4, '6565:9121' FROM DUAL)
SELECT my_id,REGEXP_SUBSTR ( my_list, '([^:]+)', 1, lvl)
FROM TABLE1,
(SELECT LEVEL lvl
FROM (SELECT MAX (LENGTH (REGEXP_REPLACE ( my_list || ':', '[^:]'))) mx
FROM TABLE1)
CONNECT BY LEVEL <= mx + 1)
WHERE lvl - 1 <= LENGTH (REGEXP_REPLACE ( my_list || ':', '[^:]'))
AND REGEXP_SUBSTR ( my_list, '([^:]+)', 1, lvl) IS NOT NULL
ORDER BY my_id,lvl;


Second way
This example is based on a nice XML solution. Only needs two lines of code. Easy to use in combination with variables.
Forum entry; http://forums.oracle.com/forums/thread.jspa?threadID=2184251&tstart=0&messageID=9406487

:F_STRING := '8092:7054:9237:4232:3333:4023:6781'

select upper(extractvalue(column_value,'e'))
from table(xmlsequence(xmltype('' || replace(:F_STRING,':','') || '').extract('e/*')));