Use case: You need to be able to copy and optionally transform any PL/SQL REF CURSOR in a uniform way across all editions of Oracle Database from 10g Release 1 and newer. You need to fetch the REF CURSOR into transient (memory) or persistent storage (normal or global temporary table). Performance is not crucial.

Challenges: REF CURSORs are very limited and in some Oracle releases quite fragile to work with.

Solution: I’ve developed PL/SQL code that implements the use case (except for LONG and LONG RAW columns) . This code leverages the REF CURSOR description technique described in my blog post Describing a REF CURSOR in Oracle 10g+ Using PL/SQL, Java and C. This blog post describes the code, which has taken a significant amount of time to develop.

Algorithm

This is the basic algorithm of the solution, implemented in a PL/SQL package REF_CURSOR_COPY:

• Describe given REF CURSOR.
• Create/reuse object types and table compatible with given REF CURSOR.
• Generate dynamic PL/SQL code that fetches all rows from given REF CURSOR into either memory or table, returning a new REF CURSOR selecting from this copy, optionally calling:
  • Given PL/SQL function for each row that was fetched.
  • Given PL/SQL block before the new copy is opened.
  • Given PL/SQL block after the new copy is opened.
• Call dynamic PL/SQL code, returning a new REF CURSOR opened for selecting from the copy.

The code generated would have the following form:

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  fetched_row ref_cur_copy_<seq>_t := ref_cur_copy_<seq>_t(null, ..., null);
  fetched_rows ref_cur_copy_<seq>_c := ref_cur_copy_<seq>_c();
  n pls_integer := 0;
  ...
begin
  rc1 := :rc1;
  loop
    exit when not rc1%isopen;
    fetch rc1 into fetched_row.<col1>, ..., fetched_row.<coln>;
    exit when rc1%notfound;
    n := n + 1;

    -- Optionally manipulate each row.
    <plsql_block_for_each>

    fetched_rows.extend(1);
    fetched_rows(n) := fetched_row;
  end loop;
  begin
    if rc1%isopen then
      close rc1;
    end if;
  exception
    when others then
      null;
  end;
  :row_count := n;

  -- Optionally call PL/SQL block before copy is opened.
  <plsql_block_before>

  open rc2 for
  select *
  from   table(cast(fetched_rows as ref_cur_copy_<seq>_c));

  -- Optionally call PL/SQL block after copy is opened.
  <plsql_block_after>

  :rc2 := rc2;
end;

Where we’re supplying the following bind variables:

• :RC1: Input REF CURSOR.
• :ROW_COUNT: Output number of rows fetched from :RC1.
• :RC2: Output REF CURSOR.

It would have been nice if you could write code like the following, but unfortunately Oracle doesn’t allow that:

fetch rc into fetched_row;

If copying to a table instead the code generated is slightly different:

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  fetched_row ref_cur_copy_<seq>_t := ref_cur_copy_<seq>_t(null, ..., null);
  fetched_rows ref_cur_copy_<seq>_c := ref_cur_copy_<seq>_c();
  n pls_integer := 0;
  ...
begin
  rc1 := :rc1;
  loop
    exit when not rc1%isopen;
    fetch rc1 into fetched_row.<col1>, ..., fetched_row.<coln>;
    exit when rc1%notfound;
    n := n + 1;

    -- Optionally manipulate each row.
    <plsql_block_for_each>

    insert into ref_cur_copy_<seq> (
      -- Session ID if not a global temporary table.
      "sessionid",
      "rownum",
      <col1>,
      ...
      <coln>
    )
    values (
      userenv('sessionid'),
      n,
      fetched_row.<col1>,
      ...
      fetched_row.<coln>
    );
  end loop;
  begin
    if rc1%isopen then
      close rc1;
    end if;
  exception
    when others then
      null;
  end;
  :row_count := n;
  commit;

  -- Optionally call PL/SQL block before copy is opened.
  <plsql_block_before>

  open rc2 for
  select t.<col1>, ..., t.<coln>
  from   ref_cur_copy_<seq> t
  where  "sessionid" = userenv('sessionid')
  order  by "rownum";

  -- Optionally call PL/SQL block after copy is opened.
  <plsql_block_after>

  :rc2 := rc2;
end;

If any of the columns in the REF CURSOR is a REF CURSOR itself (weak, strong or CURSOR expression, called a sub or an embedded REF CURSOR in the following), it gets a little more complicated. Unfortunately, we cannot use the SYS_REFCURSOR datatype for an object type attribute and we cannot use it in an associative array either, which is the reason why I didn't bother using BULK COLLECT when fetching to memory. Instead, we must declare a local variable of datatype SYS_REFCURSOR for each sub REF CURSOR, "sub fetch" this into an appropriate "sub" object type and convert this to an ANYDATA instance, such that we can hold the value in the "top" object type. Since we in Oracle cannot describe a REF CURSOR before it has been executed, this is the only solution possible as we cannot describe the sub REF CURSOR until it has been fetched for each row in the top REF CURSOR.

Here is the code generated when we have embedded REF CURSORs: (here we assume that the sub REF CURSOR columns are the last columns of the top REF CURSOR. This is just in order to simplify the code shown below, REF_CURSOR_COPY doesn’t assume that):

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  fetched_row ref_cur_copy_<seq>_t := ref_cur_copy_<seq>_t(null, ..., null);
  fetched_rows ref_cur_copy_<seq>_c := ref_cur_copy_<seq>_c();
  n pls_integer := 0;
  rce1 sys_refcursor;
  type_name_e1 user_types.type_name%type;
  ...
  rce<m> sys_refcursor;
  type_name_e<m> user_types.type_name%type;
  ...
begin
  rc1 := :rc1;
  loop
    exit when not rc1%isopen;
    fetch rc1 into fetched_row.<col1>, ..., fetched_row.<coln-m>, rce1, ..., rce<m>;
    exit when rc1%notfound;
    fetched_row.<coln-m+1> := ref_cursor_copy.fetch_data(rce1, type_name_e1);
    type_name_e1 := ref_cursor_copy.get_collection_type_name(type_name_e1);
    ...
    fetched_row.<colm> := ref_cursor_copy.fetch_data(rce<m>, type_name_e<m>);
    type_name_e<m> := ref_cursor_copy.get_collection_type_name(type_name_e<m>);
    n := n + 1;

    -- Optionally manipulate each row.
    <plsql_block_for_each>

    fetched_rows.extend(1);
    fetched_rows(n) := fetched_row;
  end loop;
  begin
    if rc1%isopen then
      close rc1;
    end if;
  exception
    when others then
      null;
  end;
  :row_count := n;

  -- Optionally call PL/SQL block before copy is opened.
  <plsql_block_before>

  open rc2 for q'[
  select <col1>,
         ...,
         <coln-m>,
         cursor(select * from table(<coln-m+1>)) <coln-m+1>,
         ...,
         cursor(select * from table(<coln>)) <coln>
  from   (
           select <col1>,
                  ...,
                  <coln-m>,
                  cast(<coln-m+1> as ]' || type_name_e1 || q'[) <coln-m+1>,
                  ...,
                  cast(<coln> as ]' || type_name_e<m> || q'[) <coln>
           from   table(cast(:fetched_rows as ref_cur_copy_<seq>_c))
         )]'
  using in fetched_rows;

  -- Optionally call PL/SQL block after copy is opened.
  <plsql_block_after>

  :rc2 := rc2;
end;

drop type ref_cur_copy_0_c;
drop type ref_cur_copy_0_t;
drop type ref_cur_copy_0e_c;
drop type ref_cur_copy_0e_t;

create type ref_cur_copy_0_t as object (
  deptno     number(2),
  dname      varchar2(14),
  emp_cursor sys.anydata
);
/

show err

create type ref_cur_copy_0_c as
table of ref_cur_copy_0_t;
/

show err

create type ref_cur_copy_0e_t as object (
  empno     number(4),
  ename     varchar2(10)
);
/

show err

create type ref_cur_copy_0e_c as
table of ref_cur_copy_0e_t;
/

show err

variable rc2 refcursor

prompt Expected to succeed

declare
  rc1 sys_refcursor;

  fetched_row ref_cur_copy_0_t := ref_cur_copy_0_t(null, null, null);
  fetched_rows ref_cur_copy_0_c := ref_cur_copy_0_c();

  fetched_row_e ref_cur_copy_0e_t := ref_cur_copy_0e_t(null, null);
  fetched_rows_e ref_cur_copy_0e_c := ref_cur_copy_0e_c();

  rce1 sys_refcursor;
  type_name_e1 user_types.type_name%type := 'REF_CUR_COPY_0E_C';
begin
  open rc1 for
  select deptno,
         dname,
         cursor(
           select empno,
                  ename
           from   emp e
           where  e.deptno = d.deptno and
                  rownum <= 2
         ) emp_cursor
  from   dept d
  where  deptno <= 20;

  loop
    -- Fetch one top REF CURSOR row.
    fetch rc1 into fetched_row.deptno, fetched_row.dname, rce1;
    exit when rc1%notfound;

    -- Fetch all sub REF CURSOR rows for the top row.
    fetched_rows_e := ref_cur_copy_0e_c();
    loop
      fetch rce1 into fetched_row_e.empno, fetched_row_e.ename;
      exit when rce1%notfound;
      fetched_rows_e.extend(1);
      fetched_rows_e(fetched_rows_e.count) := fetched_row_e;
    end loop;
    --if portable.get_major_minor_version != 10.1 then
      close rce1;
    --end if;
    fetched_row.emp_cursor := anydata.convertcollection(fetched_rows_e);
    fetched_rows.extend(1);
    fetched_rows(fetched_rows.count) := fetched_row;
  end loop;
  --if portable.get_major_minor_version != 10.1 then
    close rc1;
  --end if;

  open :rc2 for
  select deptno,
         dname,
         cursor(
           select *
           from   table(emp_cursor)
         ) emp_cursor
  from   (
           select deptno,
                  dname,
                  cast(emp_cursor as ref_cur_copy_0e_c) emp_cursor
           from   table(cast(fetched_rows as ref_cur_copy_0_c))
         );
end;
/

print :rc2

prompt Expected to fail

declare
  rc1 sys_refcursor;

  fetched_row ref_cur_copy_0_t := ref_cur_copy_0_t(null, null, null);
  fetched_rows ref_cur_copy_0_c := ref_cur_copy_0_c();

  fetched_row_e ref_cur_copy_0e_t := ref_cur_copy_0e_t(null, null);
  fetched_rows_e ref_cur_copy_0e_c := ref_cur_copy_0e_c();

  rce1 sys_refcursor;
  type_name_e1 user_types.type_name%type := 'REF_CUR_COPY_0E_C';
begin
  open rc1 for
  select deptno,
         dname,
         cursor(
           select empno,
                  ename
           from   emp e
           where  e.deptno = d.deptno and
                  rownum <= 2
         ) emp_cursor
  from   dept d
  where  deptno <= 20;

  loop
    -- Fetch one top REF CURSOR row.
    fetch rc1 into fetched_row.deptno, fetched_row.dname, rce1;
    exit when rc1%notfound;

    -- Fetch all sub REF CURSOR rows for the top row.
    fetched_rows_e := ref_cur_copy_0e_c();
    loop
      fetch rce1 into fetched_row_e.empno, fetched_row_e.ename;
      exit when rce1%notfound;
      fetched_rows_e.extend(1);
      fetched_rows_e(fetched_rows_e.count) := fetched_row_e;
    end loop;
    --if portable.get_major_minor_version != 10.1 then
      close rce1;
    --end if;
    fetched_row.emp_cursor := anydata.convertcollection(fetched_rows_e);
    fetched_rows.extend(1);
    fetched_rows(fetched_rows.count) := fetched_row;
  end loop;
  --if portable.get_major_minor_version != 10.1 then
    close rc1;
  --end if;

  open :rc2 for
  select deptno,
         dname,
         cursor(
           select empno,
                  ename
           from   table(cast(emp_cursor as ref_cur_copy_0e_c))
         ) emp_cursor
  from   table(cast(fetched_rows as ref_cur_copy_0_c));
end;
/

print :rc2

drop type ref_cur_copy_0_c;
drop type ref_cur_copy_0_t;
drop type ref_cur_copy_0e_c;
drop type ref_cur_copy_0e_t;

Expected to succeed

PL/SQL procedure successfully completed.


    DEPTNO DNAME          EMP_CURSOR
---------- -------------- --------------------
        10 ACCOUNTING     CURSOR STATEMENT : 3

CURSOR STATEMENT : 3

     EMPNO ENAME
---------- ----------
      7782 CLARK
      7839 KING

        20 RESEARCH       CURSOR STATEMENT : 3

CURSOR STATEMENT : 3

     EMPNO ENAME
---------- ----------
      7369 SMITH
      7566 JONES


Expected to fail

PL/SQL procedure successfully completed.

ERROR:
ORA-00600: internal error code, arguments: [kocgpn129], [2], [], [], [], [], [], []

We could have simply selected all columns from the ANYDATA columns, letting Oracle unnest appropriately. However, this would mean that the structure of the copy wouldn’t be the same as the original REF CURSOR (we would suddenly have a collection instead of a REF CURSOR for the embedded REF CURSOR), so this is not a viable solution.

I'm assuming that the REF CURSOR columns are compatible across the rows – ie, my implementation doesn't cater for a REF CURSOR returned in given column for row 1 that has 3 columns COLA, COLB, COLC and for row 2 it has two columns COLB and COLD.

Object Type

Because object types cannot have attributes of datatypes LONG and LONG RAW, because it’s not possible to fetch more than 32KB from these in PL/SQL unless resorting to dynamic PL/SQL and because in general their use has been deprecated for many years, I won’t be supporting them with my code. Please refer to my blog post Converting a LONG Column to a CLOB on the fly on how to convert a LONG column to a CLOB value on the fly if you need this.

For obvious performance reasons it’s important that we reuse the object type instead of creating a new for each use. This is controlled through a table REF_CURSOR_COPY_TYPES that has the following definition:

create table ref_cursor_copy_types (
  description_md5_hash   varchar2(32),
  description_length     number(5),
  ref_cursor_description xmltype,
  type_name              varchar2(30),
  type_name_collection   varchar2(30),
  table_name             varchar2(30),
  table_impl             varchar2(2)
);

Table

The table generated for each REF CURSORhas the following structure:

create table ref_cur_copy_<seq> (
  "sessionid" integer,
  "rownum"    integer,
  <col1>      <col1_datatype>,
  ...
  <coln>      <coln_datatype>
);

• "sessionid" is used to ensure session-specific data.
• "rownum" is used to ensure ordering of the data.

The code can be configured to create a global temporary table instead (with either deletion or preservation of data upon commit) as well, which means that the "sessionid" column is not used, as Oracle guarantees that the table data is session-specific. If you would like to use autonomous transactions in the copy code you need to be aware that this is not compatible with "delete on commit" as this will remove the data from the temporary table such that the copy REF CURSOR will return no rows.

VARRAY vs Nested Table

I would prefer to keep data in the copy in the same order as the original so the default implementation of the collection object types uses VARRAY. However, this creates problems on non-Windows platforms such as Linux, SPARC Solaris and Intel Solaris, as Oracle throws the error ORA-22909: exceeded maximum VARRAY limit even though this obviously isn't true. For this reason, I've made it configurable whether to use VARRAY or a nested table. It must be noted that the ordering is undefined when using the nested table.

REF_CURSOR_COPY

Here’s the implementation of the REF_CURSOR_COPY package specification:

create or replace package ref_cursor_copy
authid current_user as
/**
 * Utility to copy a REF CURSOR with the possibility to inject code before, for
 * each row and after the copy is fetched.
 * Depends on REF_CURSOR_DESCRIBE.
 * Feel free to use at your own risk.
 * @version   $Revision: 3 $
 * @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.
 */

  /**
   * Controls implementation details:
   * 'O': Use ODCI.SAVE-/RESTORECURSOR for top-level.
   * 'o': Use ODCI.SAVE-/RESTORECURSOR for non top-level.
   * 'C': Close original REF CURSOR again, after fetched and close.
   * 'c': Close sub REF CURSORs outside sub fetch code, not inside.
   * 'R': Use DBMS_ODCI.RESTOREREFCURSOR where it shouldn't be necessary.
   * 'B': Don't fetch using BULK COLLECT, fetch one row at a time.
   */
  copy_impl varchar2(5) := '';
  /**
   * Controls close implementation details:
   * '':  Close both top and sub REF CURSORs once in the appropriate place.
   * 'A': Close REF CURSOR again, after fetched and close.
   * 'N': Set REF CURSOR to NULL, after fetched and close.
   * 'S': Close sub REF CURSORs outside sub fetch code, not inside.
   * 'd': Don't close sub REF CURSORs at all.
   * 'D': Don't close top REF CURSORs at all.
   */
  close_impl varchar2(5) := '';

  /**
   * Controls collection type creation details:
   * 'V': Use VARRAY for collections (default).
   * 'T': Use TABLE for collections.
   */
  collection_impl varchar2(2) := 'V';
  /**
   * Controls table creation details:
   * 'T':  Use normal table with "sessionid" column.
   * 'td': Use global temporary table, delete rows upon commit.
   * 'tp': Use global temporary table, preserve rows upon commit (default).
   */
  table_impl varchar2(2) := 'tp';

  function to_ref_cursor(
    rc in out sys_refcursor
  )
  return sys_refcursor;
  function to_ref_cursor(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := '',
    plsql_block_before_copy_open in varchar2 := '',
    plsql_block_after_copy_open in varchar2 := ''
  )
  return sys_refcursor;
  function to_anydata(
    rc in out sys_refcursor,
    plsql_block_after_each_fetch in varchar2 := ''
  )
  return anydata;

  function fetch_data(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := ''
  )
  return anydata;
  function get_fetch_data_plsql(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    name_list in out sys.odcivarchar2list,
    type_code_list in out sys.odcinumberlist,
    ref_cur_count in out pls_integer,
    top in boolean,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := '',
    expose_rows in boolean := true,
    xml out xmltype
  )
  return varchar2;

  function get_row_count
  return pls_integer;
  function get_sub_row_count
  return pls_integer;

  function get_type_name(
    ref_cursor_desc in xmltype
  )
  return user_types.type_name%type;
  function get_collection_type_name(
    type_name in user_types.type_name%type
  )
  return user_types.type_name%type;
  function get_table_name(
    type_name in user_types.type_name%type
  )
  return user_tables.table_name%type;
  function get_table_impl(
    table_name in user_types.type_name%type,
    destination in varchar2
  )
  return ref_cursor_copy_types.table_impl%type;

  function rc_from_anydata(
    ad in anydata,
    type_name_collection in varchar2
  )
  return sys_refcursor;

  function purge_tables
  return integer;
  function drop_types_and_tables
  return integer;

  function use_odci(top in boolean)
  return boolean;
end ref_cursor_copy;
/

create or replace package body ref_cursor_copy as
  --- @version   $Revision: 3 $

  --- Number of rows fetched in latest COPY call.
  row_count pls_integer;
  --- Number of rows fetched in latest FETCH_DATA call.
  sub_row_count pls_integer;
  --- Error info.
  error_info varchar2(32767);

  procedure safely_close(
    rc in out sys_refcursor,
    force in boolean := false
  );

/**
 * Copies a REF CURSOR to memory.
 * @param   rc      REF CURSOR.
 * @return  A new REF CURSOR opened for the copy in memory.
 * @throws  ORA-20000 if the REF CURSOR contains LONG or LONG RAW columns.
 */

  function to_ref_cursor(
    rc in out sys_refcursor
  )
  return sys_refcursor as

    type_name user_types.type_name%type;

  begin
    return to_ref_cursor(rc, type_name);
  end to_ref_cursor;

/**
 * Copies a REF CURSOR to either memory or a table. Provides the ability to
 * execute PL/SQL blocks before the copy is fetched, after each row is fetched,
 * and after the entire copy has been fetched, before opening the new REF
 * CURSOR.
 * @param   rc      REF CURSOR.
 * @param   type_name
 *                  Object type name of object type representing a row on the
 *                  given REF CURSOR. Has the name of the form
 *                  REF_CURSOR_COPY_<sequence>_T.
 * @param   destination
 *                  Whether the copy should be made to memory ('M') or to a
 *                  table ('T').
 * @param   plsql_block_after_each_fetch
 *                  PL/SQL block to run for each row after it has been
 *                  fetched, working on the FETCHED_ROW object type instance
 *                  that holds the column values fetched. This can be used to
 *                  transform the data fetched.
 * @param   plsql_block_before_copy_open
 *                  PL/SQL block to run before the REF CURSOR copy is opened.
 * @param   plsql_block_after_copy_open
 *                  PL/SQL block to run after the REF CURSOR copy is opened.
 * @return  A new REF CURSOR opened for the copy in memory or table.
 * @throws  ORA-20000 if the REF CURSOR contains LONG or LONG RAW columns.
 */

  function to_ref_cursor(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := '',
    plsql_block_before_copy_open in varchar2 := '',
    plsql_block_after_copy_open in varchar2 := ''
  )
  return sys_refcursor as
  /*pragma autonomous_transaction;*/

    plsql          varchar2(32767);
    plsql2         varchar2(32767);
    plsql3         varchar2(32767);
    plsql4         varchar2(32767);
    rcn            number;
    rc2n           number;
    rc2            sys_refcursor;
    type_name_c    user_types.type_name%type;
    table_name     user_tables.table_name%type;
    name_list      sys.odcivarchar2list := sys.odcivarchar2list();
    type_code_list sys.odcinumberlist := sys.odcinumberlist();
    ref_cur_count  pls_integer;
    j              pls_integer := 0;
    xml            xmltype;
    table_impl     ref_cursor_copy_types.table_impl%type;

  begin
    -- This is necessary to make it work on various Oracle versions. If
    -- omitted, Oracle throws various "invalid cursor" errors.
    if use_odci(true) then
      dbms_odci.saverefcursor(rc, rcn);
    end if;

    plsql := get_fetch_data_plsql(
      rc,
      type_name,
      name_list,
      type_code_list,
      ref_cur_count,
      true,
      upper(destination),
      plsql_block_after_each_fetch,
      false,
      xml
    ) || chr(10);
    type_name_c := get_collection_type_name(type_name);
    table_name := get_table_name(type_name);
    table_impl := get_table_impl(table_name, upper(destination));

    if plsql_block_before_copy_open is not null then
      plsql := plsql ||
        '' || chr(10) ||
        '  ' || plsql_block_before_copy_open || chr(10) ||
        '' || chr(10);
    end if;

    plsql := plsql ||
        '' || chr(10);

    if ref_cur_count > 0 then
      -- Embedded REF CURSORs. More complex SELECT statement with CAST and
      -- CURSOR.
      if upper(destination) = 'M' then
        plsql4 :=
            '           from   table(cast(:fetched_rows as ' ||
                type_name_c || '))' || chr(10);
      elsif upper(destination) = 'T' then
        plsql4 :=
            '           from   ' || table_name || ' t' || chr(10) ||
            case when table_impl = 'T' then
              '           where  "sessionid" = userenv(''sessionid'')' || chr(10)
            end ||
            '           order  by "rownum"' || chr(10);
      end if;

      plsql := plsql ||
          '  open rc2 for q''[' || chr(10);
      for i in nvl(name_list.first, 0) .. nvl(name_list.last, -1) loop
        if i > name_list.first then
          plsql2 := plsql2 || '         ';
          plsql3 := plsql3 || '                  ';
        end if;
        if type_code_list(i) = type_codes.tc_ref_cursor then
          j := j + 1;
          plsql2 := plsql2 ||
              'cursor(select * from table(' || name_list(i) || ')) ' ||
              name_list(i);
          plsql3 := plsql3 ||
              'cast(' || name_list(i) || ' as ]'' || type_name_e' || j ||
                  ' || q''[) ' || name_list(i);
        else
          plsql2 := plsql2 ||
              name_list(i);
          plsql3 := plsql3 ||
              name_list(i);
        end if;
        if i < name_list.last then
          plsql2 := plsql2 || ',';
          plsql3 := plsql3 || ',';
        end if;
        plsql2 := plsql2 || chr(10);
        plsql3 := plsql3 || chr(10);
      end loop;
      plsql := plsql ||
          '  select ' ||
          plsql2 ||
          '  from   (' || chr(10) ||
          '           select ' ||
          plsql3 ||
          plsql4 ||
          '         )]''';
      if upper(destination) = 'M' then
        plsql := plsql || chr(10) ||
          '  using in fetched_rows;' || chr(10);
      else
        plsql := plsql || ';' || chr(10);
      end if;
    else
      -- No embedded REF CURSORs. Simple SELECT statement.
      if upper(destination) = 'M' then
        plsql := plsql ||
            '  open rc2 for' || chr(10) ||
            '  select *' || chr(10) ||
            '  from   table(cast(fetched_rows as ' || type_name_c || '));' ||
                chr(10);
      elsif upper(destination) = 'T' then
        plsql := plsql ||
            '  open rc2 for' || chr(10) ||
            '  select ';
        for c in 1 .. name_list.count loop
          plsql := plsql || case when c > 1 then ', ' end || 't.' || name_list(c);
        end loop;
        plsql := plsql || chr(10) ||
            '  from   ' || table_name || ' t' || chr(10) ||
            case when table_impl = 'T' then
              '  where  "sessionid" = userenv(''sessionid'')' || chr(10)
            end ||
            '  order  by "rownum";' || chr(10);
      end if;
    end if;

    if plsql_block_after_copy_open is not null then
      plsql := plsql ||
          '' || chr(10) ||
          '  ' || plsql_block_after_copy_open || chr(10) ||
          '' || chr(10);
    end if;

    if use_odci(true) then
      plsql := plsql ||
          '' || chr(10) ||
          '  dbms_odci.saverefcursor(rc2, :rc2n);' || chr(10);
    else
      plsql := plsql ||
          '' || chr(10) ||
          '  :rc2 := rc2;' || chr(10);
    end if;
    plsql := plsql ||
        'end;';

    if use_odci(true) then
      execute immediate plsql
      using in rcn, out row_count, out rc2n;
      if rcn is not null and copy_impl like '%R%' then
        dbms_odci.restorerefcursor(rc, rcn);
      end if;
      if rc2n is not null then
        dbms_odci.restorerefcursor(rc2, rc2n);
      end if;
    else
      execute immediate plsql
      using in out rc, out row_count, in out rc2;
    end if;
    safely_close(rc);

    return rc2;
  exception
    when others then
      error_info :=
          'ref_cursor_copy.to_ref_cursor: ' || chr(10) ||
          'plsql = ''' || plsql || '''' || chr(10) ||
          rtrim(sqlerrm, chr(10)) || chr(10) ||
          rtrim(dbms_utility.format_error_backtrace, chr(10));
      dbms_output_put_line(error_info);
      raise;
  end to_ref_cursor;

/**
 * Copies a REF CURSOR to memory. Provides the ability to execute a PL/SQL block
 * after each row is fetched.
 * @param   rc      REF CURSOR.
 * @param   plsql_block_after_each_fetch
 *                  PL/SQL block to run for each row after it has been
 *                  fetched, working on the FETCHED_ROW object type instance
 *                  that holds the column values fetched. This can be used to
 *                  transform the data fetched.
 * @return  An ANYDATA instance for the copy in memory.
 */

  function to_anydata(
    rc in out sys_refcursor,
    plsql_block_after_each_fetch in varchar2 := ''
  )
  return anydata as

    type_name user_types.type_name%type;

  begin
    return fetch_data(rc, type_name, 'M', plsql_block_after_each_fetch);
  end to_anydata;

/**
 * Fetches a REF CURSOR to either memory or a table. Provides the ability to
 * execute PL/SQL function for each row fetched.
 * @param   rc      REF CURSOR.
 * @param   type_name
 *                  Object type name of object type representing a row on the
 *                  given REF CURSOR. Has the name of the form
 *                  REF_CURSOR_COPY_<sequence>_T.
 * @param   destination
 *                  Whether the copy should be made to memory ('M') or to a
 *                  table ('T').
 * @param   plsql_block_after_each_fetch
 *                  PL/SQL block to run for each row after it has been
 *                  fetched, working on the FETCHED_ROW object type instance
 *                  that holds the column values fetched. This can be used to
 *                  transform the data fetched.
 * @return  A collection of the data fetched if DESTINATION = 'M'. NULL
 *          otherwise.
 * @throws  ORA-20000 if the REF CURSOR contains LONG or LONG RAW columns.
 */

  function fetch_data(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := ''
  )
  return anydata as
  /*pragma autonomous_transaction;*/

    plsql          varchar2(32767);
    plsql2         varchar2(32767);
    rc1            sys_refcursor;
    rcn            number;
    rc2n           number;
    rc2            sys_refcursor;
    rcd            xmltype;
    type_name_c    user_types.type_name%type;
    table_name     user_tables.table_name%type;
    n              pls_integer;
    name_list      sys.odcivarchar2list := sys.odcivarchar2list();
    type_code_list sys.odcinumberlist := sys.odcinumberlist();
    fetched_rows   anydata;
    j              pls_integer;
    ref_cur_count  pls_integer;
    xml            xmltype;

  begin
    -- This is necessary to make it work on Oracle 10.1. If omitted, Oracle
    -- throws various "invalid cursor" errors.
    if use_odci(false) then
      dbms_odci.saverefcursor(rc, rcn);
    end if;

    plsql := get_fetch_data_plsql(
      rc,
      type_name,
      name_list,
      type_code_list,
      ref_cur_count,
      false,
      destination,
      plsql_block_after_each_fetch,
      true,
      xml
    );

    if plsql is not null then
      plsql := plsql || chr(10) ||
          'end;';
      if ref_cursor_descriptor.describe_impl = 'xC' then
        execute immediate plsql
        using in xml;
      else
        if use_odci(false) then
          execute immediate plsql
          using in rcn, out sub_row_count, out fetched_rows;
          if copy_impl like '%R%' then
            dbms_odci.restorerefcursor(rc, rcn);
          end if;
        else
          execute immediate plsql
          using in out rc, out sub_row_count, out fetched_rows;
        end if;
      end if;
    end if;
    safely_close(rc);

    return fetched_rows;
  exception
    when others then
      error_info :=
          'ref_cursor_copy.fetch_data: ' || chr(10) ||
          'plsql = ''' || plsql || '''' || chr(10) ||
          rtrim(sqlerrm, chr(10)) || chr(10) ||
          rtrim(dbms_utility.format_error_backtrace, chr(10));
      dbms_output_put_line(error_info);
      safely_close(rc, true);
      raise;
  end fetch_data;

/**
 * Builds the PL/SQL that can be used to fetch a REF CURSOR to either memory or
 * a table. Provides the ability to execute PL/SQL function for each row
 * fetched.
 * @param   rc      REF CURSOR.
 * @param   type_name
 *                  Object type name of object type representing a row on the
 *                  given REF CURSOR. Has the name of the form
 *                  REF_CURSOR_COPY_<sequence>_T.
 * @param   name_list
 *                  Output: A collection of the column names of RC.
 * @param   type_code_list
 *                  Output: A collection of the column type codes of RC.
 * @param   ref_cur_count
 *                  Output: Number of REF CURSOR columns in RC.
 * @param   top     Is this a top-level REF CURSOR (TRUE) or an embedded REF
 *                  CURSOR (FALSE)?
 * @param   destination
 *                  Whether the copy should be made to memory ('M') or to a
 *                  table ('T').
 * @param   plsql_block_after_each_fetch
 *                  PL/SQL block to run for each row after it has been
 *                  fetched, working on the FETCHED_ROW object type instance
 *                  that holds the column values fetched. This can be used to
 *                  transform the data fetched.
 * @param   expose_rows
 *                  Should
 * @return  A collection of the data fetched if DESTINATION = 'M'. NULL
 *          otherwise.
 * @throws  ORA-20000 if the REF CURSOR contains LONG or LONG RAW columns.
 */

  function get_fetch_data_plsql(
    rc in out sys_refcursor,
    type_name in out user_types.type_name%type,
    name_list in out sys.odcivarchar2list,
    type_code_list in out sys.odcinumberlist,
    ref_cur_count in out pls_integer,
    top in boolean,
    destination in char := 'M',
    plsql_block_after_each_fetch in varchar2 := '',
    expose_rows in boolean := true,
    xml out xmltype
  )
  return varchar2 as

    plsql         varchar2(32767);
    plsql2        varchar2(32767);
    rcn           integer;
    rc2n          integer;
    rc1           sys_refcursor;
    rc2           sys_refcursor;
    rcd           xmltype;
    type_name_c   user_types.type_name%type;
    table_name    user_tables.table_name%type;
    n             pls_integer;
    fetched_rows  anydata;
    j             pls_integer;
    context       dbms_xmlgen.ctxtype;
    table_impl    ref_cursor_copy_types.table_impl%type;

  begin
    if not top and ref_cursor_descriptor.describe_impl = 'xC' then
      if rc is not null and rc%isopen then
        context := dbms_xmlgen.newcontext(rc);
        xml := dbms_xmlgen.getxmltype(context);
        rcd := ref_cursor_descriptor.describe(rc);
        dbms_xmlgen.closecontext(context);
        safely_close(rc, true);
      end if;
    else
      rcd := ref_cursor_descriptor.describe(rc);
    end if;
    type_name := get_type_name(rcd);
    type_name_c := get_collection_type_name(type_name);
    table_name := get_table_name(type_name);
    table_impl := get_table_impl(table_name, destination);

    -- This is necessary to make it work on Oracle 10.1. If omitted, Oracle
    -- throws various "invalid cursor" errors.
    if use_odci(top) then
      dbms_odci.saverefcursor(rc, rcn);
    end if;

    select count(*)
    into   ref_cur_count
    from   table(
             xmlsequence(
               extract(rcd, '/ROWSET/ROW')
             )
           ) c
    where  extractvalue(value(c), '//TYPE_CODE') = type_codes.tc_ref_cursor;

    if not top and ref_cursor_descriptor.describe_impl = 'xC' then
      plsql :=
          'declare' || chr(10) ||
          '  fetched_rows ' || type_name_c || ' := ' || type_name_c || '();' || chr(10) ||
          '  fetched_row ' || type_name || ';' || chr(10) ||
          '  n pls_integer := 0;' || chr(10) ||
          '  ad_fetched_rows anydata;' || chr(10) ||
          'begin' || chr(10);
    else
      plsql :=
          'declare' || chr(10) ||
          '  rc1 sys_refcursor;' || chr(10) ||
          '  rc2 sys_refcursor;' || chr(10) ||
          '  fetched_rows ' || type_name_c || ' := ' || type_name_c || '();' || chr(10) ||
          '  n pls_integer := 0;' || chr(10) ||
          '  ' || chr(10) ||
          '  type_name user_types.type_name%type;' || chr(10) ||
          '  ad_fetched_rows anydata;' || chr(10);
    end if;
    if not (not top and nvl(ref_cursor_descriptor.describe_impl, '{}') = 'xC') then
      for i in 1 .. ref_cur_count loop
        plsql := plsql ||
            '  rce' || i || ' sys_refcursor;' || chr(10) ||
            '  type_name_e' || i || ' user_types.type_name%type;' || chr(10);
      end loop;
      plsql := plsql ||
          'begin' || chr(10) ||
          case
            when use_odci(top) then
              '  dbms_odci.restorerefcursor(rc1, :rcn);' || chr(10)
            else
              '  rc1 := :rc;' || chr(10)
          end;
    end if;
    if upper(destination) = 'T' then
      -- Clear table.
      plsql := plsql ||
          '  delete' || chr(10) ||
          '  from ' || table_name ||
          case when table_impl = 'T' then
            ' t ' || chr(10) ||
                '  where t."sessionid" = userenv(''sessionid'')'
          end ||
          ';' || chr(10);
    end if;
    if not top and ref_cursor_descriptor.describe_impl = 'xC' then
      plsql := plsql ||
          '  for r in (' || chr(10) ||
          '        select value(x) xml' || chr(10) ||
          '        from   table(xmlsequence(extract(:xml, ''/ROWSET/ROW''))) x' || chr(10) ||
          '      ) loop' || chr(10) ||
          '    r.xml.toobject(fetched_row);' || chr(10) ||
          '    fetched_rows.extend(1);' || chr(10) ||
          '    fetched_rows(fetched_rows.count) := fetched_row;' || chr(10) ||
          '    n := n + 1;' || chr(10) ||
          '  end loop;' || chr(10) ||
          '  :row_count := n;' || chr(10);
    else
      plsql := plsql ||
        '  loop' || chr(10) ||
        '    exit when not rc1%isopen;' || chr(10) ||
        '    fetch rc1 into ';

      n := 0;
      j := 1;
      for c in (
            select extractvalue(value(c), '//ID') id,
                   extractvalue(value(c), '//NAME') name,
                   extractvalue(value(c), '//TYPE_CODE') type_code
            from   table(
                     xmlsequence(
                       extract(rcd, '/ROWSET/ROW')
                     )
                   ) c
          ) loop
        plsql := plsql ||
            case when c.id > 1 then ', ' end;
        if c.type_code = type_codes.tc_ref_cursor then
          -- Embedded REF CURSOR.
          plsql := plsql || 'rce' || j;
          plsql2 := plsql2 ||
              '    fetched_row.' || ref_cursor_descriptor.quote_identifier(c.name) ||
                     ' := ref_cursor_copy.fetch_data(' ||
                       'rce' || j || ', ' ||
                       'type_name_e' || j ||
                    ');' || chr(10) ||
              case when close_impl like '%S%' and close_impl not like '%d%' then
                '    begin' || chr(10) ||
                '      if rce' || j || '%isopen then' || chr(10) ||
                '        close rce' || j || ';' || chr(10) ||
                '      end if;' || chr(10) ||
                '    exception' || chr(10) ||
                '      when others then' || chr(10) ||
                '        null;' || chr(10) ||
                '    end;' || chr(10)
              end ||
              '    type_name_e' || j || ' := ' ||
                  'ref_cursor_copy.get_collection_type_name(type_name_e' || j ||
                  ');' || chr(10);
          j := j + 1;
        else
          plsql := plsql || 'fetched_row.' || ref_cursor_descriptor.quote_identifier(c.name);
        end if;
        n := n + 1;
        name_list.extend(1);
        name_list(n) := ref_cursor_descriptor.quote_identifier(c.name);
        type_code_list.extend(1);
        type_code_list(n) := c.type_code;
      end loop;

      plsql :=
          plsql || ';' || chr(10) ||
          '    exit when rc1%notfound;' || chr(10) ||
          plsql2 ||
          '    n := n + 1;' || chr(10);
      if plsql_block_after_each_fetch is not null then
        plsql := plsql ||
            '' || chr(10) ||
            plsql_block_after_each_fetch || chr(10) ||
            '' || chr(10);
      end if;
      if upper(destination) = 'M' then
        plsql := plsql ||
            '    fetched_rows.extend(1);' || chr(10) ||
            '    fetched_rows(n) := fetched_row;' || chr(10);
      elsif upper(destination) = 'T' then
        plsql := plsql ||
            '    insert into ' || table_name || ' (' || chr(10) ||
            case when table_impl = 'T' then
              '      "sessionid",' || chr(10)
            end ||
            '      "rownum",' || chr(10);
        for c in 1 .. name_list.count loop
          plsql := plsql ||
              '      ' || name_list(c) ||
              case when c < name_list.count then ', ' end || chr(10);
        end loop;
        plsql := plsql ||
            '    )' || chr(10) ||
            '    values (' || chr(10) ||
            case when table_impl = 'T' then
              '      userenv(''sessionid''),' || chr(10)
            end ||
            '      n,' || chr(10);
        for c in 1 .. name_list.count loop
          plsql := plsql ||
              '      fetched_row.' || name_list(c) ||
              case when c < name_list.count then ', ' end || chr(10);
        end loop;
        plsql := plsql ||
            '    );' || chr(10);
      end if;

      plsql := plsql ||
          '  end loop;' || chr(10) ||
          case when close_impl is null or
              top and
                close_impl not like '%D%'
              or not top and
                close_impl not like '%S%' and
                close_impl not like '%d%' then
            '  begin' || chr(10) ||
            '    if rc1%isopen then' || chr(10) ||
            '      close rc1;' || chr(10) ||
            '    end if;' || chr(10) ||
            '  exception' || chr(10) ||
            '    when others then' || chr(10) ||
            '      null;' || chr(10) ||
            '  end;' || chr(10)
          end ||
          '  :row_count := n;' || chr(10);

      -- Build constructor call to FETCHED_ROW variable.
      plsql := replace(
        plsql,
        '  fetched_rows ' || type_name_c || ' := ' || type_name_c || '();',
        '  fetched_rows ' || type_name_c || ' := ' || type_name_c || '();' || chr(10) ||
           '  fetched_row ' || type_name || ' := ' || type_name || '(' ||
                rtrim(rpad('null, ', 6 * n, 'null, '), ', ') || ');'
      );
    end if;

    if expose_rows then
      if destination = 'M' then
        plsql := plsql ||
          '  ad_fetched_rows := anydata.convertcollection(fetched_rows);' || chr(10);
      elsif destination = 'T' then
        if not table_impl like '%td' then
          plsql := plsql ||
              '  commit;' || chr(10);
        end if;
        plsql := plsql ||
            '  ad_fetched_rows := null;' || chr(10);
      end if;
      plsql := plsql ||
          '  :rows2 := ad_fetched_rows;';
    else
      if destination = 'T' and not table_impl like '%td' then
        plsql := plsql ||
            '  commit;';
      end if;
    end if;

    return plsql;
  end get_fetch_data_plsql;

/**
 * Gets the number of rows fetched in the latest COPY call.
 */

  function get_row_count
  return pls_integer as

  begin
    return row_count;
  end get_row_count;

/**
 * Gets the number of rows fetched in the latest FETCH_DATA call.
 */

  function get_sub_row_count
  return pls_integer as

  begin
    return sub_row_count;
  end get_sub_row_count;

/**
 * Creates two object types and a table matching a description of a REF CURSOR
 * obtained from REF_CURSOR_DESCRIBE.DESCRIBE. The object type has the the name
 * of the form REF_CURSOR_COPY_<sequence>_T. The VARRAY/TABLE collection has a
 * name of the form REF_CURSOR_COPY_<sequence>_C. The table has a name of the
 * form REF_CURSOR_COPY_<sequence>. If such object types and table already exist
 * they are reused.
 * @param   ref_cursor_desc
 *                  REF CURSOR description.
 * @return  Object type name.
 */

  function get_type_name(
    ref_cursor_desc in xmltype
  )
  return user_types.type_name%type as
  pragma autonomous_transaction;

    plsql                varchar2(32767);
    plsql2               varchar2(32767);
    type_name            user_types.type_name%type;
    type_name_c          user_types.type_name%type;
    table_name           user_tables.table_name%type;
    xml                  varchar2(32767);
    description_md5_hash ref_cursor_copy_types.description_md5_hash%type;
    description_length   pls_integer;
    n                    pls_integer;
    nt                   pls_integer := 0;
    m                    pls_integer;
    name_list            sys.odcivarchar2list := sys.odcivarchar2list();
    decl_list            sys.odcivarchar2list := sys.odcivarchar2list();

  begin
    xml := ref_cursor_desc.getstringval;
    description_md5_hash := rawtohex(utl_raw.cast_to_raw(
      dbms_obfuscation_toolkit.md5(input_string => xml))
    );
    description_length := length(xml);

    begin
      -- Do we have a matching object type already?
      select type_name
      into   type_name
      from   ref_cursor_copy_types rcct
      where  rcct.description_md5_hash = get_type_name.description_md5_hash and
             rcct.description_length = get_type_name.description_length;

      type_name_c := get_collection_type_name(type_name);
      table_name := get_table_name(type_name);

      -- Do both still exist?
      select count(ut1.type_name) + count(ut2.table_name)
      into   n
      from   user_types ut1,
             user_tables ut2
      where  ut1.type_name in (
               get_type_name.type_name,
               get_type_name.type_name_c
             ) and
             ut2.table_name = get_type_name.table_name;

      if n < 3 then
        -- We had them, but not anymore. Drop what may have remained and remove
        -- from mapping.
        delete
        from   ref_cursor_copy_types rcct
        where  rcct.type_name = get_type_name.type_name;

        commit;

        begin
          execute immediate 'truncate table ' || table_name;
        exception
          when others then
            null;
        end;
        begin
          execute immediate 'drop table ' || table_name;
        exception
          when others then
            null;
        end;
        begin
          execute immediate 'drop type ' || type_name_c;
        exception
          when others then
            null;
        end;
        begin
          execute immediate 'drop type ' || type_name;
        exception
          when others then
            null;
        end;

        -- Create new ones.
        raise no_data_found;
      end if;
    exception
      when no_data_found then
        -- We don't have a matching object type -- create one.
        select 'REF_CUR_COPY_' || ref_cursor_copy_seq.nextval || '_T'
        into   type_name
        from   dual;

        type_name_c := get_collection_type_name(type_name);
        table_name := get_table_name(type_name);

        plsql :=
            'create type ' || type_name || ' as object (' || chr(10);

        for c in (
              select extractvalue(value(c), '//ID') id,
                     extractvalue(value(c), '//NAME') name,
                     extractvalue(value(c), '//TYPE_CODE') type_code,
                     extractvalue(value(c), '//DECLARATION') declaration,
                     extractvalue(value(c), '//OWNER') owner,
                     extractvalue(value(c), '//TYPE_NAME') type_name
              from   table(
                       xmlsequence(
                         extract(ref_cursor_desc, '/ROWSET/ROW')
                       )
                     ) c
            ) loop
          if c.type_code in (type_codes.tc_long, type_codes.tc_long_raw) then
            raise_application_error(
              -20000,
              'ref_cursor_copy: LONG [RAW] is not supported: Column ' ||
                  c.id || ', name "' || c.name || '"'
            );
          end if;

          plsql := plsql || c.declaration || ',' || chr(10);
          name_list.extend(1);
          name_list(c.id) := ref_cursor_descriptor.quote_identifier(c.name);
          decl_list.extend(1);
          decl_list(c.id) := c.declaration;

          if c.type_code = type_codes.tc_object then
            -- Check whether the type is a nested table.
            select count(text)
            into   m
            from   all_type_versions atv
            where  atv.owner = c.owner and
                   atv.type_name = c.type_name and
                   atv.typecode = 'COLLECTION' and
                   lower(text) like '%table%';

            if m > 0 then
              -- Nested table.
              nt := nt + 1;
              plsql2 := plsql2 || chr(10) ||
                'nested table ' || c.name || ' store as ' || table_name || nt;
            end if;
          end if;
        end loop;

        plsql := rtrim(plsql, ',' || chr(10)) || chr(10) ||
            ');';

        begin
          -- Create object type.
          execute immediate plsql;

          -- Create object type collection.
          plsql :=
            'create type ' || type_name_c || ' as' || chr(10);
          if collection_impl = 'V' then
            plsql := plsql ||
                'varray(2147483647) of ' || type_name || ';';
          elsif collection_impl = 'T' then
            plsql := plsql ||
                'table of ' || type_name || ';';
          end if;

          execute immediate plsql;

          -- Create table.
          plsql :=
            'create ' ||
                case when table_impl in ('td', 'tp') then
                  'global temporary '
                end ||
                'table ' || table_name || ' (' || chr(10) ||
                case when table_impl = 'T' then
                  '  "sessionid" integer,' || chr(10)
                end ||
            '  "rownum" integer,' || chr(10);
          for c in 1 .. decl_list.count loop
            plsql := plsql ||
            '  ' || decl_list(c) ||
                case when c < decl_list.count then ',' else '' end || chr(10);
          end loop;
          plsql := plsql ||
              ')';
          if table_impl = 'td' then
            plsql := plsql || chr(10) ||
                'on commit delete rows';
          elsif table_impl = 'tp' then
            plsql := plsql || chr(10) ||
                'on commit preserve rows';
          end if;

          if plsql2 is not null then
            -- We have nested table attributes. Specify table for these.
            plsql := plsql || chr(10) ||
                plsql2;
          end if;

          execute immediate plsql;

          insert into ref_cursor_copy_types (
            description_md5_hash,
            description_length,
            type_name,
            type_name_collection,
            table_name,
            table_impl
          )
          values (
            description_md5_hash,
            description_length,
            type_name,
            type_name_c,
            table_name,
            table_impl
          );

          commit;
        exception
          when others then
            dbms_output.put_line('ref_cursor_copy.get_type_name: plsql =');
            dbms_output_put_line(plsql);
            dbms_output_put_line(sqlerrm);
            dbms_output_put_line(dbms_utility.format_error_backtrace);
            raise;
        end;
    end;

    return type_name;
  end get_type_name;

/**
 * Returns collection type name for given object type name.
 * @param   type_name
 *                  Object type name.
 * @return  Collection type name.
 */

  function get_collection_type_name(
    type_name in user_types.type_name%type
  )
  return user_types.type_name%type as

  begin
    return substr(type_name, 1, length(type_name) - 1) || 'C';
  end;

/**
 * Returns table name for given object type name.
 * @param   type_name
 *                  Object type name.
 * @return  Table name.
 */

  function get_table_name(
    type_name in user_types.type_name%type
  )
  return user_tables.table_name%type as

  begin
    return substr(type_name, 1, length(type_name) - 2);
  end;

/**
 * Returns table implementation details for given table name.
 * @param   table_name
 *                  Table name.
 * @return  Table implementation used (refer to TABLE_IMPL).
 */

  function get_table_impl(
    table_name in user_types.type_name%type,
    destination in varchar2
  )
  return ref_cursor_copy_types.table_impl%type as

    table_impl ref_cursor_copy_types.table_impl%type;
  begin
    select table_impl
    into   table_impl
    from   ref_cursor_copy_types rcct
    where  rcct.table_name = get_table_impl.table_name;

    return table_impl;
  exception
    when no_data_found then
      if destination = 'M' then
        -- Ignore -- we don't need the table as we're copying to memory.
        null;
      else
        raise;
      end if;
  end get_table_impl;

/**
 * Opens a REF CURSOR for a collection.
 * @param   ad      Collection converted to ANYDATA.
 * @return  REF CURSOR on the collection.
 */

  function rc_from_anydata(
    ad in anydata,
    type_name_collection in varchar2
  )
  return sys_refcursor as

    rc sys_refcursor;

  begin
    open rc for '
    select *
    from   table(cast(:ad as ' || type_name_collection || '))'
    using in ad;

    return rc;
  end rc_from_anydata;

/**
 * Purges data in tables for this session and commits. In general, this is not
 * necessary if global temporary tables are used as data will be purged either
 * when the transaction is committed or at the very latest when the session
 * ends.
 * @return    Number of rows deleted.
 */

  function purge_tables
  return integer as

    dml varchar2(1000);
    n integer := 0;

  begin
    for t in (
          select table_name,
                 table_impl
          from   ref_cursor_copy_types
        ) loop
      dml :=
          'delete ' || chr(10) ||
          'from   ' || t.table_name || ' t' || chr(10);

      if t.table_impl = 'T' then
        -- Not a global temporary table.
        dml := dml ||
            'where  t."sessionid" = userenv(''sessionid'')';
      end if;

      begin
        execute immediate dml;
        commit;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.purge_tables: Purging table "' ||
                  t.table_name || '": ' || sqlerrm
          );
      end;

      n := n + sql%rowcount;
    end loop;

    return n;
  end purge_tables;

/**
 * Drops object types and tables.
 * @return    Number of objects dropped.
 */

  function drop_types_and_tables
  return integer as

    dxl varchar2(1000);
    n integer := 0;

  begin
    for t in (
          select type_name,
                 type_name_collection,
                 table_name
          from   ref_cursor_copy_types
        ) loop
      begin
        dxl := 'drop type ' || t.type_name_collection;
        execute immediate dxl;
        n := n + 1;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.drop_types_and_tables: Dropping type "' ||
                  t.type_name_collection || '": ' || sqlerrm
          );
      end;

      begin
        dxl := 'drop type ' || t.type_name;
        execute immediate dxl;
        n := n + 1;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.drop_types_and_tables: Dropping type "' ||
                  t.type_name || '": ' || sqlerrm
          );
      end;

      -- Empty table first. Necessary if temporary table in order to avoid
      -- ORA-14452: attempt to create, alter or drop an index on temporary table
      -- already in use
      begin
        dxl := 'delete from ' || t.table_name;
        execute immediate dxl;
        n := n + 1;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.drop_types_and_tables: Deleting from table "' ||
                  t.table_name || '": ' || sqlerrm
          );
      end;

      commit;

      -- Truncate table first. Necessary if temporary table in order to avoid
      -- ORA-14452: attempt to create, alter or drop an index on temporary table
      -- already in use
      begin
        dxl := 'truncate table ' || t.table_name;
        execute immediate dxl;
        n := n + 1;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.drop_types_and_tables: Truncating table "' ||
                  t.table_name || '": ' || sqlerrm
          );
      end;

      begin
        dxl := 'drop table ' || t.table_name;
        execute immediate dxl;
        n := n + 1;
      exception
        when others then
          dbms_output.put_line(
              'ref_cursor_copy.drop_types_and_tables: Dropping table "' ||
                  t.table_name || '": ' || sqlerrm
          );
      end;
    end loop;

    delete
    from   ref_cursor_copy_types;

    commit;

    return n;
  end drop_types_and_tables;

/**
 * Determines whether to use DBMS_ODCI.SAVE-/RESTOREREFCURSOR for given context.
 * This can be controlled through the configuration of COPY_IMPL.
 * @param   top     TRUE if top-level, FALSE if embedded REF CURSOR.
 * @return  TRUE if DBMS_ODCI must be used.
 */

  function use_odci(top in boolean)
  return boolean as

  begin
    return top and copy_impl like '%O%' or
        not top and copy_impl like '%o%';
  end use_odci;

/**
 * Safely closes a REF CURSOR. This can be controlled by the value of COPY_IMPL
 * or forced. If FORCE is FALSE this really shouldn't be necessary, but it could
 * help prevent ORA-00600 [17281] [1001].
 * @param   rc      REF CURSOR.
 * @param   force   If TRUE configuration of COPY_IMPL is overridden and RC
 *                  is attempted closed and set to NULL. If FALSE, the value of
 *                  COPY_IMPL controls what is done.
 */

  procedure safely_close(
    rc in out sys_refcursor,
    force in boolean := false
  ) as

  begin
    if rc%isopen then
      if force or close_impl like '%A%' then
        close rc;
      end if;
      if force or close_impl like '%N%' then
        rc := null;
      end if;
    end if;
  exception
    when others then
      null;
  end safely_close;

begin
  copy_impl := '';
  if portable.get_major_minor_version = 10.1 then
    close_impl := 'Dd';
    copy_impl := 'OoR';
  end if;

  if portable.get_major_version = 10 or
      portable.get_full_version like '11.1.0.6%' then
    if portable.is_option_enabled('Java') and
        portable.get_major_minor_version != 10.1 then
      /*
       * Force Java implementation of REF CURSOR description as the PL/SQL
       * implementation doesn't correctly restore the REF CURSOR state, leading
       * to one of the following errors:
       * ORA-00600 [psdmsc: psdinvdef#1]
       * ORA-01001: invalid cursor
       */
      ref_cursor_descriptor.describe_impl := 'J2';
    else
      ref_cursor_descriptor.describe_impl := 'C2';
    end if;
  end if;

  if portable.get_platform != 'Windows' then
    -- Use TABLE for collections instead of VARRAY in order to avoid
    -- ORA-22909: exceeded maximum VARRAY limit.
    collection_impl := 'T' ;
  end if;
end ref_cursor_copy;
/

Examples

Increase EMP.SAL by 10%

In this example we use the PLSQL_BLOCK_AFTER_EACH_FETCH parameter to call a PL/SQL block for each row fetched in order to increase the SAL column by 10%:

set feedback 1

prompt From EMP

select *
from   (
         select empno,
                sal
         from   emp
         order  by empno
       )
where  rownum <= 3;

variable rc2 refcursor

prompt Transformed REF CURSOR copy of EMP

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  type_name user_types.type_name%type;
begin
  open rc1 for
  select *
  from   (
           select empno,
                  sal
           from   emp
           order  by empno
         )
  where  rownum <= 3;

  :rc2 := ref_cursor_copy.to_ref_cursor(
    rc1,
    type_name,
    destination => 'M',
    plsql_block_after_each_fetch => 'fetched_row.sal := fetched_row.sal * 1.10;'
  );
end;
/

print rc2

From EMP

     EMPNO        SAL
---------- ----------
      7369        800
      7499       1600
      7521       1250

3 rows selected.

Transformed REF CURSOR copy of EMP

PL/SQL procedure successfully completed.


     EMPNO        SAL
---------- ----------
      7369        880
      7499       1760
      7521       1375

3 rows selected.

Forcing DBMS_XMLGEN to Generate XML Schema-compliant Date/Time Values Without Side Effects

In this example we use the PLSQL_BLOCK_BEFORE_COPY_OPEN parameter to change various NLS date/time formats in order to make DBMS_XMLGEN generate XML Schema-compliant date/time values. Since we’re doing this after the original REF CURSOR has been fetched, this does not have any side effects on other calls to TO_CHAR in the original query:

set serveroutput on format truncated

alter session set nls_date_format = 'dd-mon-yy';
alter session set nls_timestamp_format = 'dd-mon-rr hh.mi.ssxff am';
alter session set nls_timestamp_tz_format = 'dd-mon-rr hh.mi.ssxff am tzr';

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  type_name user_types.type_name%type;
  context dbms_xmlgen.ctxtype;
  xml xmltype;
begin
  open rc1 for
  select 1 id,
         123.456 n,
         'abcDEF' vc,
         cast(timestamp'2011-02-01 14:25:30' as date) d,
         timestamp'2011-02-01 14:25:30.123456789' t1,
         timestamp'2011-02-01 14:25:30.123456789 +02:00' t2,
         to_char(cast(timestamp'2011-02-01 14:25:30' as date)) d_tc,
         to_char(timestamp'2011-02-01 14:25:30.123456789') t1_tc,
         to_char(timestamp'2011-02-01 14:25:30.123456789 +02:00') t2_tc
  from   dual;

  rc2 := ref_cursor_copy.to_ref_cursor(
    rc1,
    type_name,
    destination => 'M',
    plsql_block_before_copy_open =>
      q'[
        dbms_session.set_nls(
          'nls_date_format', '''yyyy-mm-dd"T"hh24:mi:ss'''
        );
        dbms_session.set_nls(
          'nls_timestamp_format', '''yyyy-mm-dd"T"hh24:mi:ss.ff9'''
        );
        dbms_session.set_nls(
          'nls_timestamp_tz_format', '''yyyy-mm-dd"T"hh24:mi:ss.ff9tzh:tzm'''
        );
      ]'
  );

  context := dbms_xmlgen.newcontext(rc2);
  xml := dbms_xmlgen.getxmltype(context);
  dbms_xmlgen.closecontext(context);
  dbms_output_put_line(xml.getclobval);

  dbms_session.set_nls(
    'nls_date_format', '''dd-mon-yy'''
  );
  dbms_session.set_nls(
    'nls_timestamp_format', '''dd-mon-rr hh.mi.ssxff am'''
  );
  dbms_session.set_nls(
    'nls_timestamp_tz_format', '''dd-mon-rr hh.mi.ssxff am tzr'''
  );
end;
/

Session altered.


Session altered.


Session altered.

<ROWSET>
 <ROW>
  <ID>1</ID>
  <N>123.456</N>
  <VC>abcDEF</VC>
  <D>2011-02-01T14:25:30</D>
  <T1>2011-02-01T14:25:30.123456789</T1>
  <T2>2011-02-01T14:25:30.123456789+02:00</T2>
  <D_TC>01-feb-11</D_TC>
  <T1_TC>01-feb-11 02.25.30.123456789 pm</T1_TC>
  <T2_TC>01-feb-11 02.25.30.123456789 pm +02:00</T2_TC>
 </ROW>
</ROWSET>

PL/SQL procedure successfully completed.

Known Issues

Fragile Cursor Management in Oracle Database 10g Release 1

Cursor management in Oracle Database 10g Release 1 seems extremely fragile. Often when the REF CURSORs are closed Oracle throws ORA-00600 [17281] [1001] when or before you terminate your session. Also when they are closed, Oracle might reuse them in an incorrect manner, leading to errors like ORA-01001: invalid cursor, ORA-01007: variable not in select list etc. The only solution I’ve found on 10.1 is to configure the implementation to not close any of the REF CURSORs, which obviously you need to be careful about. Oracle will close them correctly when your session ends but if you’ve got code that runs for a long time, calling REF_CURSOR_COPY, you might need to increase the init.ora parameter open_cursors or migrate to Oracle 10.2 or newer.
Another issue is that transferring REF CURSORs into and out from dynamic PL/SQL often leads to ORA-01001: invalid cursor. The solution here is to use DBMNS_ODCI.SAVE-/RESTOREREFCURSOR and transfer the obtained cursor number instead.
These issues have been verified on 10.1.0.2.0, 10.1.0.3.0, 10.1.0.4.0 and 10.1.0.5.0.

ORA-01001: invalid cursor in DBMS_SQL

If Oracle throws ORA-01001: invalid cursor in DBMS_SQL try to use either the direct Java or the direct C description implementation as this is a bug in DBMS_SQL. This seems to be related to DBMS_SQL that cannot handle CURSOR expressions in the call to DBMS_SQL.TO_CURSOR_NUMBER.

Example:

set serveroutput on format truncated

declare
  rc1 sys_refcursor;
  rc2 sys_refcursor;
  type_name user_types.type_name%type;
  describe_impls dbms_utility.name_array;
  xml xmltype;
begin
  describe_impls(1) := 'P';
  describe_impls(2) := 'J2';
  describe_impls(3) := 'C2';

  for i in nvl(describe_impls.first, 0) .. nvl(describe_impls.last, -1) loop
    ref_cursor_descriptor.describe_impl := describe_impls(i);

    dbms_output.put_line('ref_cursor_descriptor.describe_impl = ''' ||
        ref_cursor_descriptor.describe_impl || '''');

    open rc1 for
    select d.deptno,
           d.dname,
           cursor (
             select empno,
                    hiredate,
                    ename
             from   emp e
             where  e.deptno = d.deptno and
                    rownum <= 2
           ) emp_cursor
    from   dept d
    where  deptno < 40;

    begin
      rc2 := ref_cursor_copy.to_ref_cursor(
        rc1,
        type_name,
        destination => 'M',
        plsql_block_before_copy_open => 'xml_gen.set_xml_nls_formats;'
      );

      xml := xml_gen.from_ref_cursor(rc2);
      dbms_output_put_line(xml.getclobval);

      xml_gen.set_session_nls_formats;
    exception
      when others then
        dbms_output.put_line(sqlerrm);
        dbms_output.put_line(dbms_utility.format_error_backtrace);
    end;
  end loop;
end;
/

ref_cursor_descriptor.describe_impl = 'P'
ORA-01001: invalid cursor
ORA-06512: at "SYS.DBMS_SQL", line 2610
ORA-06512: at "SCOTT.REF_CURSOR_DESCRIPTOR_PLSQL", line 123
ORA-06512: at "SCOTT.REF_CURSOR_DESCRIPTOR", line 132
ORA-06512: at "SCOTT.REF_CURSOR_DESCRIPTOR", line 83
ORA-06512: at "SCOTT.REF_CURSOR_COPY", line 425
ORA-06512: at "SCOTT.REF_CURSOR_COPY", line 305
ref_cursor_descriptor.describe_impl = 'J2'
<ROWSET>
 <ROW>
  <DEPTNO>10</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7782</EMPNO>
    <HIREDATE>1981-06-09T00:00:00</HIREDATE>
    <ENAME>CLARK</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7839</EMPNO>
    <HIREDATE>1981-11-17T00:00:00</HIREDATE>
    <ENAME>KING</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
 <ROW>
  <DEPTNO>20</DEPTNO>
  <DNAME>RESEARCH</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7369</EMPNO>
    <HIREDATE>1980-12-17T00:00:00</HIREDATE>
    <ENAME>SMITH</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7566</EMPNO>
    <HIREDATE>1981-04-02T00:00:00</HIREDATE>
    <ENAME>JONES</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
 <ROW>
  <DEPTNO>30</DEPTNO>
  <DNAME>SALES</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7499</EMPNO>
    <HIREDATE>1981-02-20T00:00:00</HIREDATE>
    <ENAME>ALLEN</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7521</EMPNO>
    <HIREDATE>1981-02-22T00:00:00</HIREDATE>
    <ENAME>WARD</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
</ROWSET>
ref_cursor_descriptor.describe_impl = 'C2'
<ROWSET>
 <ROW>
  <DEPTNO>10</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7782</EMPNO>
    <HIREDATE>1981-06-09T00:00:00</HIREDATE>
    <ENAME>CLARK</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7839</EMPNO>
    <HIREDATE>1981-11-17T00:00:00</HIREDATE>
    <ENAME>KING</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
 <ROW>
  <DEPTNO>20</DEPTNO>
  <DNAME>RESEARCH</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7369</EMPNO>
    <HIREDATE>1980-12-17T00:00:00</HIREDATE>
    <ENAME>SMITH</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7566</EMPNO>
    <HIREDATE>1981-04-02T00:00:00</HIREDATE>
    <ENAME>JONES</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
 <ROW>
  <DEPTNO>30</DEPTNO>
  <DNAME>SALES</DNAME>
  <EMP_CURSOR>
   <EMP_CURSOR_ROW>
    <EMPNO>7499</EMPNO>
    <HIREDATE>1981-02-20T00:00:00</HIREDATE>
    <ENAME>ALLEN</ENAME>
   </EMP_CURSOR_ROW>
   <EMP_CURSOR_ROW>
    <EMPNO>7521</EMPNO>
    <HIREDATE>1981-02-22T00:00:00</HIREDATE>
    <ENAME>WARD</ENAME>
   </EMP_CURSOR_ROW>
  </EMP_CURSOR>
 </ROW>
</ROWSET>

Supporting Oracle Database 9i Release 2

The techniques in REF_CURSOR_COPY and the related blog can be made to work for Oracle 9.2 as well but you would need to remove calls to DBMS_ODCI.SAVE-/RESTOREREFCURSOR and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE as these procedures are not available in Oracle 9.2.
I haven’t used Conditional Compilation techniques for this as it would exclude Oracle versions 10.1.0.2.0 and 10.1.0.3.0.

Improvements

The code can be improved in the following ways:

• Usage of BULK COLLECT in the fetch process. I didn't bother implementing this because it's not possible to use BULK COLLECT with embedded REF CURSORs because we cannot declare an associative array type that holds REF CURSORs (you get PLS-00990: Index Tables of Cursor Variables are disallowed). So implementing BULK COLLECT would only be used if the incoming REF CURSOR doesn't have embedded REF CURSORs.
• Commit interval in the code generated when DESTINATION is 'T' (table), preventing errors in case the rollback segment is not large enouogh to hold all rows.
• The GET_TYPE_NAME function that creates object types and table for handling a given REF CURSOR could be improved such that it’s guarded by a critical section/region, preventing that multiple sessions try to create identical object types (with different names through the sequence) with the same MD5 hash, all but the first failing with a DUP_VALUE_ON_INDEX exception.

Source Code

You can download the source code here.
The PL/SQL code depends on the PL/SQL, Java and C code developed in my blog post Describing a REF CURSOR in Oracle 10g+ Using PL/SQL, Java and C.

Installation

Install the code for the other blog post first. Then run grant.sql with SQL*Plus logged on as a DBA, specifying the schema you would like to install in as a parameter and then run install.sql with SQL*Plus, logged on to the schema you want the code to be installed in.

Conclusion

The code shown here has been fairly well tested with the following Oracle versions and editions:

• Oracle Database 10g Release 1 10.1.0.2.0 Personal Edition on Windows XP SP3.
• Oracle Database 10g Release 1 10.1.0.3.0 Enterprise Edition on Intel Solaris 10.
• Oracle Database 10g Release 2 10.2.0.1.0 Express Edition on Windows XP SP3.
• Oracle Database 10g Release 2 10.2.0.1.0 Express Edition on Red Hat Enterprise Linux 5.3.
• Oracle Database 10g Release 2 10.2.0.2.0 Enterprise Edition on Intel Solaris 10.
• Oracle Database 11g Release 1 11.1.0.6.0 Personal Edition on Windows XP SP3.
• Oracle Database 11g Release 1 11.1.0.6.0 Enterprise Edition on SPARC Solaris 10.
• Oracle Database 11g Release 2 11.2.0.1.0 Personal Edition on Windows 7.

Feel free to use the code at your own risk. I welcome your feedback and suggestions for improvements but the code as such is not supported.

Use case: You need to be able to describe any PL/SQL REF CURSOR in a uniform way across all editions of Oracle Database from 10g Release 1 and newer. Performance of the solution is not important. Potentially, you need to fetch the REF CURSOR into transient or persistent storage — I’ll come back to this in a future blog post.

Challenges: Description of a REF CURSOR directly in PL/SQL requires Oracle 11g Release 1 or newer. Java stored procedures are not supported by Oracle Database 10g Release 2 Express Edition.

Solution: I’ve developed code that uses a combination of Java, C and PL/SQL in order to implement the use case. This blog post describes the code, which has taken a significant amount of time to develop.

Alternatives

Basically, we have the following alternatives:

• On Oracle Database 11g Release 1 and newer you can use DBMS_SQL.TO_CURSOR_NUMBER to convert the REF CURSOR to a DBMS_SQL cursor and then use DBMS_SQL.DESCRIBE_COLUMNS3 to get a descriptions of the columns.
• On Oracle Database 10g Release 1 and 2 you can write a Java stored procedure that receives the REF CURSOR as a parameter, which Oracle converts to a java.sql.ResultSet, that you can describe through java.sql.ResultSetMetaData.
• On Oracle Database 10g Release 2 Express Edition (in fact with Oracle Database 9i Release 2 as well) you can write an external procedure in C and use Oracle Call Interface (OCI) to describe the REF CURSOR.

We’ll look at the alternatives and implement each of them in the following. I’ve chosen to implement the functionality such that the REF CURSOR description returned from each of the 3 functions is a VARCHAR2 with XML on the form:

<ROWSET generator="PL/SQL|Java|C">
  <ROW><!-- Column 1. -->
    <ID>column_id_1</ID>
    <NAME>column_name_1</NAME>
    <TYPE_CODE>data_type_code_1</TYPE_CODE>
    <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
    <!-- Character set form: 1: Database. 2: National. -->
    <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
    <NAME>column_name_1</NAME>
    <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
    <LENGTH>column_length</LENGTH>
    <!-- For CHAR, VARCHAR2. -->
    <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
    <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
    <PRECISION>precision_1</PRECISION>
    <!-- NUMBER, INTERVAL DAY TO SECOND. -->
    <SCALE>scale_1</SCALE>
    <!-- For object types, including collections. -->
    <OWNER>type_owner_1</LENGTH_SEMANTICS>
    <!-- For object types, including collections. -->
    <TYPE_NAME>type_name_1</TYPE_NAME>
    <DECLARATION>declaration_1</DECLARATION>
  </ROW>
  <!-- Columns 2 through n-1. -->
  <ROW><!-- Column n. -->
    <ID>column_id_n</ID>
    <NAME>column_name_n</NAME>
    <TYPE_CODE>data_type_code_n</TYPE_CODE>
    <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
    <!-- Character set form: 1: Database. 2: National. -->
    <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
    <NAME>column_name_n</NAME>
    <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
    <LENGTH>column_length</LENGTH>
    <!-- For CHAR, VARCHAR2. -->
    <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
    <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
    <PRECISION>precision_n</PRECISION>
    <!-- NUMBER, INTERVAL DAY TO SECOND. -->
    <SCALE>scale_n</SCALE>
    <!-- For object types, including collections. -->
    <OWNER>type_owner_n</LENGTH_SEMANTICS>
    <!-- For object types, including collections. -->
    <TYPE_NAME>type_name_n</TYPE_NAME>
    <DECLARATION>declaration_n</DECLARATION>
  </ROW>
</ROWSET>

Type Code Mappings

Oracle uses different type codes in different contexts so we need to be able to map them from the native type codes used in PL/SQL, Java and C to a “uniform” type code, in order to make usages of the code we develop here portable across the various Oracle Database editions and versions. I’ve described such a “uniform” type code mapping in my blog post Oracle Type Code Mappings.

Implementations

We’ll have a look at the various implementations in the following sections. I’ve chosen not to use Conditional Compilation for separation of the language-specific variants of the describe functionality. This is in order to be able to support Oracle 10.1.0.2.0 and 10.1.0.3.0 as Conditional Compilation wasn’t introduced for Oracle 10g until 10.1.0.4.0.

We use a trick to be able to transfer REF CURSOR to a Java stored procedure and C external procedure: We obtain a cursor number for the REF CURSOR through a call to DBMS_ODCI.SAVEREFCURSOR, transfer the cursor number to Java/C and there obtain the REF CURSOR based on the number with DBMS_ODCI.RESTOREREFCURSOR. This shouldn’t be necessary for Java but experience shows that transferring the REF CURSOR directly fails with Oracle Database 10g Release 1 (Oracle sometimes throws various “invalid cursor” errors or provides a null java.sql.ResultSet) and furthermore, it’s not possible to transfer a REF CURSOR directly from PL/SQL to a C external procedure. Well, the latter turns out not to be true but more about that later.

PL/SQL: REF_CURSOR_DESCRIPTOR_PLSQL

This is the pure PL/SQL implementation (ref_cursor_descriptor_plsql.fnc):

create or replace function ref_cursor_descriptor_plsql(rc in out sys_refcursor)
return varchar2 as

/**
 * PL/SQL function that describes a REF CURSOR. Requires Oracle 11g Release 1 or
 * newer. Description is returned in the following XML format (without the XML
 * comments):
 * <ROWSET generator="PL/SQL">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *     <DECLARATION>declaration_1</DECLARATION>
 *   </ROW>
 *   ...
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *     <DECLARATION>declaration_n</DECLARATION>
 *   </ROW>
 * </ROWSET>
 * Feel free to use at your own risk.
 * @param   rc      REF CURSOR.
 * @return  XML describing the REF CURSOR.
 * @version   $Revision: 1 $
 * @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.
 */

  rcc binary_integer;
  result varchar2(32767);
  column_count number;
  column_metadata dbms_sql.desc_tab3;
  type_code pls_integer;
  native_type_code pls_integer;
  xml varchar2(32767);

/**
 * Maps type code from native type code returned by DBMS_SQL.DESCRIBE_COLUMNS3
 * to "uniform" type code as described in blog article
 * http://ellebaek.wordpress.com/2011/02/25/oracle-type-code-mappings/
 * @param   type_code
 *                  Type code returned by DBMS_SQL.DESCRIBE_COLUMNS3.
 * @return  "Uniform" type code.
 */

  function map_type_code(type_code in pls_integer)
  return pls_integer as

  begin
    case type_code
      when 11 then
        -- ROWID.
        return type_codes.tc_rowid;
      when 100 then
        -- BINARY_FLOAT.
        return type_codes.tc_binary_float;
      when 101 then
        -- BINARY_DOUBLE.
        return type_codes.tc_binary_double;
      when 111 then
        -- Object REF.
        return type_codes.tc_ref;
      else
        return type_code;
    end case;
  end map_type_code;

/**
 * Append XML element, indented to given level (2 spaces per level, starting
 * from 1).
 * @param   level   Level for indentation.
 * @param   name    Element name.
 * @param   value   Element numeric value.
 */

  procedure append(
    level in pls_integer,
    name in varchar2,
    value in varchar2
  ) as

  begin
    xml := xml ||
      rpad(' ', (level - 1) * 2) ||
      '<' || name || '>' ||
      value ||
      '</' || name || '>' || chr(10);
  end append;

begin
  -- Convert the REF CURSOR to a DBMS_SQL cursor. Only possible with Oracle
  -- Database 11g Release 1 and newer.
  rcc := dbms_sql.to_cursor_number(rc);

  -- Describe the columns.
  dbms_sql.describe_columns3(
    c => rcc,
    col_cnt => column_count,
    desc_t => column_metadata
  );

  if column_count > 0 then
    xml := '<ROWSET generator="PL/SQL">' || chr(10);

    for i in 1 .. column_count loop
      native_type_code := column_metadata(i).col_type;
      type_code := map_type_code(native_type_code);
      xml := xml ||
          '  <ROW>' || chr(10);
      append(3, 'ID', i);
      append(3, 'NAME', column_metadata(i).col_name);
      append(3, 'TYPE_CODE', type_code);
      append(3, 'NATIVE_TYPE_CODE', native_type_code);
      if type_code in (
          type_codes.tc_char,
          type_codes.tc_varchar2,
          type_codes.tc_clob
        ) then
        -- Text.
        append(3, 'CHARSET_FORM', column_metadata(i).col_charsetform);
      end if;
      if type_code in (
          type_codes.tc_char,
          type_codes.tc_varchar2,
          type_codes.tc_raw,
          type_codes.tc_urowid
        ) then
        -- Text (not CLOB), RAW, UROWID.
        append(3, 'LENGTH', column_metadata(i).col_max_len);
      end if;
      if type_code in (
            type_codes.tc_char,
            type_codes.tc_varchar2
          )
          and column_metadata(i).col_charsetform = 1 then
        -- Database character set.
        append(3, 'LENGTH_SEMANTICS', 'BYTE');
      end if;
      if type_code in (
          type_codes.tc_number,
          type_codes.tc_interval_ym,
          type_codes.tc_interval_ds
        ) then
        -- NUMBER/INTERVAL YEAR TO MONTH/INTERVAL DAY TO SECOND.
        append(3, 'PRECISION', column_metadata(i).col_precision);
      end if;
      if type_code in (
          type_codes.tc_timestamp,
          type_codes.tc_timestamp_tz,
          type_codes.tc_timestamp_ltz
        ) then
        -- TIMESTAMP%.
        append(3, 'PRECISION', column_metadata(i).col_scale);
      end if;
      if type_code in (
          type_codes.tc_number,
          type_codes.tc_interval_ds
        ) then
        -- NUMBER/INTERVAL DAY TO SECOND.
        append(3, 'SCALE', column_metadata(i).col_scale);
      end if;
      if type_code = type_codes.tc_object then
        -- Object type/collection.
        append(3, 'OWNER', column_metadata(i).col_schema_name);
        append(3, 'TYPE_NAME', column_metadata(i).col_type_name);
      end if;
      xml := xml || '  </ROW>' || chr(10);
    end loop;

    xml := xml || '</ROWSET>';
  end if;

  -- Convert the DBMS_SQL cursor back to a REF CURSOR. Only possible with Oracle
  -- Database 11g Release 1 and newer. This is necessary in order to continue
  -- to use the REF CURSOR.
  rc := dbms_sql.to_refcursor(rcc);

  return xml;
end ref_cursor_descriptor_plsql;
/

Java Stored Procedure: com.appatra.blog.RefCursorDescriptor + REF_CURSOR_DESCRIPTOR_JAVA

This is the Java stored procedure com.appatra.blog.RefCursorDescriptor (RefCursorDescriptor.sql):

create or replace and resolve java source named
    "com/appatra/blog/RefCursorDescriptor" as
package com.appatra.blog;

/**
 * Java class useful for describing REF CURSORs in PL/SQL. Requires Oracle 10g
 * Release 1 or newer. Does not work with Oracle 10g Release 2 Express Edition
 * as this doesn't support Java stored procedures.
 * Feel free to use at your own risk.
 * When installing, substitution must be switched off ("set scan off") or the
 * define character must be set to something different than & ("set define ^").
 * @version   $Revision: 1 $
 * @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.
 */

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.CallableStatement;
import java.sql.ResultSet;
import java.sql.Types;

import java.lang.StringBuffer;

import java.sql.ResultSetMetaData;
import java.sql.SQLException;

import oracle.jdbc.OracleTypes;
import oracle.jdbc.OracleResultSetMetaData;

public class RefCursorDescriptor {
  /// Buffer.
  private static StringBuffer result;

/**
 * Java method that describes a REF CURSOR. Description is returned in the
 * following XML format (without the XML comments):
 * <ROWSET generator="Java">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *   </ROW>
 *   <!-- Columns 2 through n-1. -->
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *   </ROW>
 * </ROWSET>
 * @param   rS      Result set. Oracle translates a PL/SQL REF CURSOR to a Java
 *                  result set when this method is called. However, this only
 *                  works from Oracle 10g Release 2. For 10g Release 1, you need
 *                  to use the overloaded method that takes a REF CURSOR number
 *                  as input.
 * @return  XML describing the REF CURSOR.
 */

  public static String describe(java.sql.ResultSet rS)
  throws SQLException {
    result = new StringBuffer(10000);

    int typeCode, nativeTypeCode;
    String typeName;
    int precision;
    int scale;

    if (rS != null) {
      OracleResultSetMetaData rSMD = (OracleResultSetMetaData)rS.getMetaData();

      result.append("<ROWSET generator=\"Java\">\n");
      for (int i = 1; i <= rSMD.getColumnCount(); i++) {
        result.append("  <ROW>\n");
        append("ID", i);
        append("NAME", rSMD.getColumnName(i));
        typeName = rSMD.getColumnTypeName(i);
        nativeTypeCode = rSMD.getColumnType(i);
        typeCode = mapTypeCode(nativeTypeCode, typeName);
        append("TYPE_CODE", typeCode);
        append("NATIVE_TYPE_CODE", nativeTypeCode);
        if (typeCode == 1 ||
            typeCode == 96 ||
            typeCode == 112) {
          append("CHARSET_FORM", rSMD.isNCHAR(i) ? "2" : "1");
        }
        if (typeCode == 109 || typeCode == 111) {
          append("OWNER", typeName.substring(0, typeName.indexOf('.')));
          append("TYPE_NAME", typeName.substring(typeName.indexOf('.') + 1));
        }
        if (typeCode == 1 ||
            typeCode == 96 ||
            typeCode == 23 ||
            typeCode == 208) {
          append("LENGTH", rSMD.getColumnDisplaySize(i));
          if (typeCode != 23 && typeCode != 208 && ! rSMD.isNCHAR(i))
            append("LENGTH_SEMANTICS", "CHAR");
        }

        precision = 0;
        if (typeCode == 2 ||
            typeCode == 182 ||
            typeCode == 183)
          precision = rSMD.getPrecision(i);

        scale = 0;
        if (typeCode == 2 ||
            typeCode == 180 ||
            typeCode == 181 ||
            typeCode == 183 ||
            typeCode == 231)
          scale = rSMD.getScale(i);

        if (typeCode == 2 ||
            typeCode == 183) {
          // Number, INTERVAL DS
          append("PRECISION", precision);
          append("SCALE", scale);
        }
        if (typeCode == 182) {
          // INTERVAL YM.
          append("PRECISION", precision);
        }
        if (typeCode == 180 ||
            typeCode == 181 ||
            typeCode == 231) {
          // TIMESTAMP%,
          append("PRECISION", scale);
        }
        result.append("  </ROW>\n");
      }
      result.append("</ROWSET>");

      return result.toString();
    }
    else result.append("<ROWSET/>");

    return null;
  }

/**
 * Java method that describes a REF CURSOR. Description is returned in the
 * following XML format (without the XML comments):
 * <ROWSET generator="Java">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *   </ROW>
 *   <!-- Columns 2 through n-1. -->
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *   </ROW>
 * </ROWSET>
 * @param   rCN     REF CURSOR number for the REF CURSOR that is to be
 *                  described. This must have been obtained through a call to
 *                  DBMS_ODCI.SAVEREFCURSOR. Result set. This "trick" is
 *                  required for Oracle 10g Release 1. For For 10g Release 2 and
 *                  newer, you can use the overloaded method that takes a Java
 *                  result set as input.
 * @return  XML describing the REF CURSOR.
 */

  public static String describe(int rCN)
  throws SQLException {
    Connection connection =
        DriverManager.getConnection("jdbc:default:connection:");

    CallableStatement cS =
        connection.prepareCall("begin dbms_odci.restorerefcursor(:rc, :rcn); end;");

    ResultSet rS;

    cS.registerOutParameter(1, OracleTypes.CURSOR);
    cS.setInt(2, rCN);
    cS.execute();
    rS = (ResultSet)cS.getObject(1);

    return describe(rS);
  }

/**
 * Append XML element, indented to given level 3 (2 spaces per level).
 * @param   name    Element name.
 * @param   value   Element string value.
 */

  private static void append(String name, String value) {
    result.append("    <" + name + ">" + value + "</" + name + ">\n");
  }

/**
 * Append XML element, indented to given level 3 (2 spaces per level).
 * @param   name    Element name.
 * @param   value   Element integer value.
 */

  private static void append(String name, int value) {
    append(name, "" + value);
  }

/**
 * Maps from JDBC type code to "uniform" type code, which differs in many
 * cases. Refer to blog post
 * http://ellebaek.wordpress.com/2011/02/25/oracle-type-code-mappings/
 * @param   typeCode
 *                  JDBC type code.
 * @param   typeName
 *                  Type name, required to distinguish DATE and TIMESTAMP.
 * @return  "Uniform" type code.
 */

  private static int mapTypeCode(int typeCode, String typeName) {
    switch (typeCode) {
      case Types.VARCHAR:
        // VARCHAR2.
        return 1;
      case Types.NUMERIC:
        // NUMBER.
        return 2;
      case Types.LONGVARCHAR:
        // LONG.
        return 8;
      case Types.DATE:
        // DATE.
        return 12;
      case 100:
        // BINARY_FLOAT.
        return 21;
      case 101:
        // BINARY_DOUBLE.
        return 22;
      case Types.TIMESTAMP:
        // TIMESTAMP/DATE.
        if (typeName.equals("DATE"))
          return 12;
        return 180;
      case -101:
        // TIMESTAMP WITH TIME ZONE.
        return 181;
      case -102:
        // TIMESTAMP WITH LOCAL TIME ZONE.
        return 231;
      case -103:
        // INTERVAL YEAR TO MONTH.
        return 182;
      case -104:
        // INTERVAL DAY TO SECOND.
        return 183;
      case Types.VARBINARY:
        // RAW.
        return 23;
      case Types.LONGVARBINARY:
        // LONG RAW.
        return 24;
      case -8:
        // ROWID/UROWID, map to UROWID.
        return 208;
      case Types.CHAR:
        // CHAR
        return 96;
      case Types.CLOB:
        // CLOB.
        return 112;
      case Types.BLOB:
        // BLOB.
        return 113;
      case -13:
        // BFILE.
        return 114;
      case Types.STRUCT:
      case Types.ARRAY:
      case 2007:
        // Object type, collection etc.
        return 109;
      case 2006:
        // Object REF.
        return 111;
      case -10:
        // REF CURSOR.
        return 102;
    }

    return typeCode;
  }
};
/

create or replace function ref_cursor_descriptor_java(rcn in pls_integer)
return varchar2 as
language java
name 'com.appatra.blog.RefCursorDescriptor.describe(int) return java.lang.String';
/

create or replace function ref_cursor_descriptor_java2(rc in sys_refcursor)
return varchar2 as
language java
name 'com.appatra.blog.RefCursorDescriptor.describe(java.sql.ResultSet) return java.lang.String';
/

C External Procedures: refcurdesc and refcurdesc2

According to the Oracle documentation it’s not possible to pass a REF CURSOR to a C external procedure — but with Oracle Database 9i Release 2 and newer you can in fact transfer a REF CURSOR directly to your C external procedure and it will be received as OCIStmt **. With Oracle Database 9i Release 1 you get an error if you try this when you call the function (ORA-28577: argument 2 of external procedure refcurdesc2 has unsupported datatype UNKNOWN). With Oracle Database 8 and 8i you get an internal error.

I discovered this when I was working on my followup to this blog post Copying/Transforming a REF CURSOR in Oracle 10g+. Here I found that the DBMS_ODCI trick somehow changed the REF CURSOR such that it wasn’t possible to use it with DBMS_XMLGEN afterwards (ORA-24374: define not done before fetch or execute and fetch errors) and also it didn’t work consistently with sub REF CURSORs, rendering them impossible to fetch from after the describe (ORA-01001: invalid cursor, ORA-01008: not all variables bound errors etc).

I’ve kept the original implementation of the refcurdesc function and added a new function refcurdesc2 that uses the direct approach.

Here’s the implementation of the PL/SQL interfaces to the C external procedures, first ref_cursor_descriptor_c.fnc (indirect):

create or replace function ref_cursor_descriptor_c(
  rcn in pls_integer
)
return varchar2 as
language c
name "refcurdesc"
library refcurdesc
with context
parameters (
  context,
  rcn int,
  rcn indicator short,
  return indicator short,
  return length short,
  return string
);
/

create or replace function ref_cursor_descriptor_c2(
  rc in sys_refcursor
)
return varchar2 as
language c
name "refcurdesc2"
library refcurdesc
with context
parameters (
  context,
  rc,
  return indicator short,
  return length short,
  return string
);
/

Here’s the implementation of the C external procedure (refcurdesc.c):

/**
 * External C procedure used to describe a REF CURSOR. This is useful for
 * pre-Oracle 11.1 (with Oracle 11.1+ you can convert the REF CURSOR to a
 * DBMS_SQL cursor and then describe). It's also possible to write a Java
 * stored procedure, but obviously this is not supported with 10.2 XE.
 * refcurdesc() requires Oracle 10g Release 1 or newer.
 * refcurdesc2() requires Oracle 9i Release 2 or newer.
 * Feel free to use at your own risk.
 * @version   $Revision: 2 $
 * @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.
 */

#if defined(_WINDOWS)
#define _CRT_SECURE_NO_WARNINGS 1
#endif

#define USE_OCI_STMT_PREPARE 1

#include <stdio.h>
#include <string.h>
#include <oci.h>
#include <ociextp.h>

static text temp[32600];
static text temp2[1000];
static text indent[1000];
static text declare[100];
static sword oci_status;

#define INDENT_PER_LEVEL 2

static OCIExtProcContext *oci_ctx;
static OCIEnv            *envhp;
static OCISvcCtx         *svchp;
static OCIError          *errhp;
static OCIParam          *pard;

static int line;
static int is_cursor_expression;

void describe_statement(int level, OCIStmt *refcur);
void append_element(int level, text *name, text *value);
void append_element2(int level, text *name, sb2 value);
void append2(int level, text *t);
void append(text *t);
int map_type_code(int type_code);
void disable_prefetch(OCIStmt *stmt);
void checkerr(OCIError *errhp, sword status, int line);

/**
 * OCI error handling: Check for returned errors and don't execute future calls
 * if an OCI error is raised. Any errors will be raised in an exception.
 * @param   call    Function call to execute if no previous calls have failed.
 */

#define CHECKERR(call) { line = __LINE__; if (oci_status == OCI_SUCCESS) checkerr(errhp, call, line); }

/**
 * Describe REF CURSOR and return description in XML form in a VARCHAR2.
 * Description is returned in the following XML format (without the XML
 * comments):
 * <ROWSET generator="C">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *   </ROW>
 *   <!-- Columns 2 through n-1. -->
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *   </ROW>
 * </ROWSET>
 * Limitations: 32KB limitation on the return value. The implementation is not
 * thread safe.
 * @param   ctx     OCI context provided by Oracle.
 * @param   refcurno
 *                  REF CURSOR number obtained on the original REF CURSOR
 *                  through a call to DBMS_ODCI.SAVEREFCURSOR. This is required
 *                  as according to the Oracle documentation we can't directly
 *                  transfer the REF CURSOR to this function.
 * @param   refcurno_i
 *                  NULL indicator for refcurno.
 * @param   result_i
 *                  NULL indicator for the VARCHAR2 we're returning.
 * @param   result_l
 *                  Length of the VARCHAR2 we're returning.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

text *refcurdesc(
  OCIExtProcContext *ctx,
  int refcurno,
  short refcurno_i,
  short *result_i,
  short *result_l
)

{
  text *result;

  static OCIStmt *stmhp = (OCIStmt *)0;
  char *plsql_block = "begin dbms_odci.restorerefcursor(:rc, :rcn); end;";
  char *plsql_block2 = "begin dbms_odci.saverefcursor(:rc, :rcn); end;";
  OCIStmt *refcur = NULL;
  OCIBind *bndp1 = (OCIBind *)0;
  OCIBind *bndp2 = (OCIBind *)0;

  temp[0] = '\0';

  oci_ctx = ctx;
  oci_status = OCI_SUCCESS;

  if (refcurno_i == OCI_IND_NULL) {
    *result_i = (short)OCI_IND_NULL;
    // PL/SQL has no notion of a NULL ptr, so return a zero-byte string.
    result = (text *)OCIExtProcAllocCallMemory(ctx, 1);
    result[0] = '\0';
  }
  else {
    // Get the OCI handles from our Oracle session.
    CHECKERR(OCIExtProcGetEnv(ctx, &envhp, &svchp, &errhp))

    is_cursor_expression = 0;
#if USE_OCI_STMT_PREPARE
    // Convert the REF CURSOR number back to a REF CURSOR.
    CHECKERR(
      OCIHandleAlloc(
        (dvoid *)envhp, (dvoid **)&stmhp,
        (ub4)OCI_HTYPE_STMT, (size_t)0, (dvoid **)0
      )
    )
    //disable_prefetch(stmhp);
    // Prepare PL/SQL block.
    CHECKERR(
      OCIStmtPrepare(
        stmhp, errhp, (unsigned char *)plsql_block, (ub4)strlen(plsql_block),
        OCI_NTV_SYNTAX, OCI_DEFAULT
      )
    )
#else
    CHECKERR(
      OCIStmtPrepare2(
        svchp, &stmhp, errhp, (unsigned char *)plsql_block, (ub4)strlen(plsql_block),
        NULL, 0,
        OCI_NTV_SYNTAX, OCI_DEFAULT
      )
    )
#endif

    // Bind.
    CHECKERR(
      OCIHandleAlloc(envhp, (void **)&refcur, OCI_HTYPE_STMT, 0, NULL)
    )
    // Position 1: Output REF CURSOR.
    CHECKERR(
      OCIBindByPos(
        stmhp, &bndp1, errhp, (ub4)1,
        (void *)&refcur, (sb4)0, SQLT_RSET, (void *)0, (ub2 *)0,
        (ub2 *)0, (ub4)0, (ub4)0, (ub4)OCI_DEFAULT
      )
    )
    // Position 2: Input REF CURSOR number, previously saved with
    // DBMS_ODCI.SAVEREFCURSOR.
    CHECKERR(
      OCIBindByPos(
        stmhp, &bndp2, errhp, (ub4)2,
        (void *)&refcurno, sizeof(int), SQLT_INT, (void *)&refcurno_i, (ub2 *)0,
        (ub2 *)0, (ub4)0, (ub4)0, (ub4)OCI_DEFAULT
      )
    )

    // Execute.
    CHECKERR(OCIStmtExecute(svchp, stmhp, errhp, 1, 0, NULL, NULL, OCI_DEFAULT))

#if USE_OCI_STMT_PREPARE
    // Free PL/SQL statement.
    CHECKERR(OCIHandleFree(stmhp, OCI_HTYPE_STMT))
    stmhp = NULL;
#endif

    if (oci_status == OCI_SUCCESS)
      describe_statement(1, refcur);

    // Set length.
    *result_l = (short)strlen((char *)temp);
    // Set NULL indicator.
    if (*result_l > 0)
      *result_i = (short)OCI_IND_NOTNULL;
    else *result_i = (short)OCI_IND_NULL;

    // Allocate memory for result string, including NULL terminator.
    result = (text *)OCIExtProcAllocCallMemory(ctx, *result_l + 1);
    strcpy((char *)result, (char *)temp);

    // Free REF CURSOR handle: Seems to close the original REF CURSOR, rendering
    // this useless (ORA-01001: invalid cursor).
    //CHECKERR(OCIHandleFree(refcur, OCI_HTYPE_STMT))
    refcur = NULL;
  }

  // Return pointer, which PL/SQL frees later.
  return result;
}

/**
 * Describe REF CURSOR and return description in XML form in a VARCHAR2.
 * Description is returned in the following XML format (without the XML
 * comments):
 * <ROWSET generator="C">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *   </ROW>
 *   <!-- Columns 2 through n-1. -->
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *   </ROW>
 * </ROWSET>
 * Limitations: 32KB limitation on the return value. The implementation is not
 * thread safe.
 * @param   ctx     OCI context provided by Oracle.
 * @param   refcur
 *                  REF CURSOR (pointer to OCIStmt *). According to the Oracle
 *                  documentation we can't directly transfer a PL/SQL REF
 *                  CURSOR to this function but in fact we can with Oracle 9.2
 *                  and newer.
 * @param   result_i
 *                  NULL indicator for the VARCHAR2 we're returning.
 * @param   result_l
 *                  Length of the VARCHAR2 we're returning.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

text *refcurdesc2(
  OCIExtProcContext *ctx,
  OCIStmt **refcur,
  short *result_i,
  short *result_l
)

{
  text *result;

  temp[0] = '\0';

  oci_ctx = ctx;
  oci_status = OCI_SUCCESS;

  CHECKERR(OCIExtProcGetEnv(ctx, &envhp, &svchp, &errhp))

  if (oci_status == OCI_SUCCESS)
    describe_statement(1, *refcur);

  // Set length.
  *result_l = (short)strlen((char *)temp);
  // Set NULL indicator.
  if (*result_l > 0)
    *result_i = (short)OCI_IND_NOTNULL;
  else *result_i = (short)OCI_IND_NULL;

  // Allocate memory for result string, including NULL terminator.
  result = (text *)OCIExtProcAllocCallMemory(ctx, *result_l + 1);
  strcpy((char *)result, (char *)temp);

  return result;
}

/**
 * Describes given ref cursor statement handle and appends to global description
 * XML.
 */

void describe_statement(int level, OCIStmt *refcur) {
  text column_name[31];

  ub2 type_code, native_type_code;
  ub2 col_width, charset_form, char_semantics, charset_id;
  text *s;
  ub4 slen;
  sb4 param_status;
  sb2 /*ub1*/ precision;
  sb1 scale;

  ub4 column_count;
  int i;

  text owner[31];
  text type_name[129];

  // Number of columns.
  column_count = 0;
  CHECKERR(
    OCIAttrGet(
      (void *)refcur, (ub4)OCI_HTYPE_STMT, (void *)&column_count,
      (ub4 *)0, (ub4)OCI_ATTR_PARAM_COUNT, errhp
    )
  )

  if (column_count > 0)
    append2(level, "<ROWSET generator=\"C\">\n");

  for (i = 1; i <= (int)column_count; i++) {
    // For each column.
    append2(level + 1, "<ROW>\n");

    pard = (OCIParam *)0;

    // Get the column.
    param_status = OCIParamGet(
      (dvoid *)refcur, OCI_HTYPE_STMT, errhp,
      (dvoid **)&pard, (ub4)i
    );

    // Datatype.
    CHECKERR(
      OCIAttrGet(
        (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
        (dvoid *)&native_type_code, (ub4 *)0, (ub4)OCI_ATTR_DATA_TYPE,
        (OCIError *)errhp
      )
    )
    type_code = map_type_code(native_type_code);

    // Column name.
    slen = 0;
    CHECKERR(
      OCIAttrGet(
        (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
        (dvoid **)&s, (ub4 *)&slen, (ub4)OCI_ATTR_NAME,
        (OCIError *)errhp
      )
    )
    strncpy((char *)column_name, (char *)s, slen);
    column_name[slen] = '\0';

    append_element2(level + 2, "ID", i);
    append_element(level + 2, "NAME", column_name);
    append_element2(level + 2, "TYPE_CODE", type_code);
    append_element2(level + 2, "NATIVE_TYPE_CODE", native_type_code);

    // Character set.
    if (type_code == SQLT_AFC ||
        type_code == SQLT_CLOB ||
        type_code == SQLT_CHR ||
        type_code == SQLT_VCS) {
      CHECKERR(
        OCIAttrGet(
          (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&charset_form, (ub4 *)0, (ub4)OCI_ATTR_CHARSET_FORM,
          (OCIError *)errhp
        )
      )
      append_element2(level + 2, "CHARSET_FORM", charset_form);
      if (charset_form == SQLCS_EXPLICIT) {
        CHECKERR(
          OCIAttrGet(
            (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
            (dvoid *)&charset_id, (ub4 *)0, (ub4)OCI_ATTR_CHARSET_ID,
            (OCIError *)errhp
          )
        )
        append_element2(level + 2, "CHARSET_ID", charset_id);
      }
      else charset_id = 0;
    }
    else charset_form = 0;

    // Number/Interval YM/Interval DS: Precision.
    if (type_code == SQLT_NUM ||
        type_code == 182 ||
        type_code == 183) {
      CHECKERR(
        OCIAttrGet(
          (dvoid*)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&precision, (ub4 *)0,
          (ub4)OCI_ATTR_PRECISION, (OCIError *)errhp
        )
      )

      sprintf(temp2, "precision = %d", (int)precision);
      append_element2(level + 2, "PRECISION", precision);
    }
    // Number: Scale.
    if (type_code == SQLT_NUM) {
      CHECKERR(
        OCIAttrGet(
          (dvoid*)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&scale, (ub4 *)0,
          (ub4)OCI_ATTR_SCALE, (OCIError *)errhp
        )
      )
      append_element2(level + 2, "SCALE", scale);
    }
#ifdef OCI_ATTR_FSPRECISION
    // Timestamp/interval DS: Scale.
    if (type_code == 180 ||
        type_code == 181 ||
        type_code == 231 ||
        type_code == 183) {
      CHECKERR(
        OCIAttrGet(
          (dvoid*)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&scale, (ub4 *)0,
          (ub4)OCI_ATTR_FSPRECISION, (OCIError *)errhp
        )
      )
      if (type_code == 183)
        append_element2(level + 2, "SCALE", scale);
      else append_element2(level + 2, "PRECISION", scale);
    }
#endif

    // Length, length semantics.
    char_semantics = 0;
    if (type_code == SQLT_AFC ||
        type_code == SQLT_CHR ||
        type_code == SQLT_VCS ||
        type_code == SQLT_BIN ||
        type_code == 208) {
#ifdef OCI_ATTR_CHAR_USED
      CHECKERR(
        OCIAttrGet(
          (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&char_semantics,  (ub4 *)0, (ub4)OCI_ATTR_CHAR_USED,
          (OCIError *)errhp
        )
      )
      col_width = 0;
      if (char_semantics)
        // Column width in characters.
        CHECKERR(
          OCIAttrGet(
            (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
            (dvoid *)&col_width, (ub4 *)0, (ub4)OCI_ATTR_CHAR_SIZE,
            (OCIError *)errhp
          )
        )
      else
#else
      char_semantics = 2;
#endif
        // Column width in bytes.
        CHECKERR(
          OCIAttrGet(
            (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
            (dvoid *)&col_width, (ub4 *)0, (ub4)OCI_ATTR_DATA_SIZE,
            (OCIError *)errhp
          )
        )

      append_element2(level + 2, "LENGTH", col_width);
      if (charset_form == 1 && type_code != 208)
        append_element(level + 2, "LENGTH_SEMANTICS", char_semantics == 2 ? "" : (char_semantics ? "CHAR" : "BYTE"));
    }

    // Object type/collection or REF to object type/collection.
    if (type_code == 109 || type_code == 111) {
      // Schema name.
      slen = 0;
      CHECKERR(
        OCIAttrGet(
          (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&s, (ub4 *)&slen,
          (ub4)OCI_ATTR_SCHEMA_NAME, (OCIError *)errhp
        )
      )
      strncpy((char *)owner, (char *)s, slen);
      owner[slen] = '\0';
      append_element(level + 2, "OWNER", owner);

      // Type name.
      slen = 0;
      CHECKERR(
        OCIAttrGet(
          (dvoid *)pard, (ub4)OCI_DTYPE_PARAM,
          (dvoid *)&s, (ub4 *)&slen,
          (ub4)OCI_ATTR_TYPE_NAME, (OCIError *)errhp
        )
      )
      strncpy((char *)type_name, (char *)s, slen);
      type_name[slen] = '\0';
      append_element(level + 2, "TYPE_NAME", type_name);
    }

    append2(level + 1, "</ROW>\n");
  }

  if (column_count > 0)
    append2(level, "</ROWSET>");
}

/**
 * Append XML element, indented to column 5.
 * @param   level   Level for indentation.
 * @param   name    Element name.
 * @param   value   Element text value.
 */

void append_element(int level, text *name, text *value) {
  text temp[100];

  sprintf(temp, "%*s<%s>%s</%s>\n", (level - 1) * INDENT_PER_LEVEL, "", name, value, name);
  append(temp);
}

/**
 * Append XML element, indented to given level (2 spaces per level, starting
 * from 1).
 * @param   level   Level for indentation.
 * @param   name    Element name.
 * @param   value   Element numeric value.
 */

void append_element2(int level, text *name, sb2 value) {
  text temp[100];

  sprintf(temp, "%*s<%s>%d</%s>\n", (level - 1) * INDENT_PER_LEVEL, "", name, value, name);
  append(temp);
}

/**
 * Appends given text to internal buffer, indented to given level (2 spaces per
 * level, starting from 1).
 * @param   level   Level for indentation.
 * @param   t       Text to append.
 */

void append2(int level, text *t) {
  sprintf(indent, "%*s", (level - 1) * INDENT_PER_LEVEL, "");
  append(indent);
  append(t);
}

/**
 * Appends given text to internal buffer.
 * @param   t       Text to append.
 */

void append(text *t) {
  if (strlen(t) + strlen(temp) > sizeof(temp))
    // Too large, raise an exception.
    OCIExtProcRaiseExcpWithMsg(
      oci_ctx,
      20000,
      "refcurdesc: Too many columns, description is larger than 32KB, which isn't supported",
      0
    );

  strcat(temp, t);
}

/**
 * Maps from OCI type code to "uniform" type code, which differs in a few
 * cases. Refer to blog post
 * http://ellebaek.wordpress.com/2011/02/25/oracle-type-code-mappings/
 * @param   type_code
 *                  OCI type code.
 * @return  "Uniform" type code.
 */

int map_type_code(int type_code) {
  switch (type_code) {
#ifdef SQLT_IBFLOAT
    case SQLT_IBFLOAT:
      // BINARY_FLOAT.
      return 21;
#endif
#ifdef SQLT_IBDOUBLE
    case SQLT_IBDOUBLE:
      // BINARY_DOUBLE.
      return 22;
#endif
#ifdef SQLT_TIMESTAMP
    case SQLT_TIMESTAMP:
      // TIMESTAMP.
      return 180;
#endif
#ifdef SQLT_TIMESTAMP_TZ
    case SQLT_TIMESTAMP_TZ:
      // TIMESTAMP WITH TIME ZONE.
      return 181;
#endif
#ifdef SQLT_TIMESTAMP_LTZ
    case SQLT_TIMESTAMP_LTZ:
      // TIMESTAMP WITH LOCAL TIME ZONE.
      return 231;
#endif
#ifdef SQLT_INTERVAL_YM
    case SQLT_INTERVAL_YM:
      // INTERVAL YEAR TO MONTH.
      return 182;
#endif
#ifdef SQLT_INTERVAL_DS
    case SQLT_INTERVAL_DS:
      // INTERVAL DAY TO SECOND.
      return 183;
#endif
#ifdef SQLT_RDD
    case SQLT_RDD:
      // ROWID/UROWID, map to UROWID.
      return 208;
#endif
    case SQLT_NTY:
    case SQLT_NCO:
      // Object type or collection.
      return 109;
    case SQLT_REF:
      // REF to object type or collection.
      return 111;
    case SQLT_RSET:
      // REF CURSOR.
      return 102;
  }

  return type_code;
}

/**
 * OCI error checker. If any error is found, information about this is raised
 * in an exception and the global variable oci_status is updated with the value
 * of the error code, which will prevent any OCI calls wrapped up in CHECKERR
 * from being executed.
 * @param   errhp   Error handle.
 * @parem   status  OCI call return code.
 * @param   line    Line on which error occurred.
 */

void checkerr(OCIError *errhp, sword status, int line) {
  text errbuf[512];
  text message[1000] = "";
  text message2[1000] = "";
  sb4 errcode = 0;

  switch (status) {
    case OCI_SUCCESS:
      break;
    case OCI_SUCCESS_WITH_INFO:
      strcpy(message, "Error: OCI_SUCCESS_WITH_INFO");
      break;
    case OCI_NEED_DATA:
      strcpy(message, "Error: OCI_NEED_DATA");
      break;
    case OCI_NO_DATA:
      strcpy(message, "Error: OCI_NO_DATA");
      break;
    case OCI_ERROR:
      OCIErrorGet(
        (dvoid *)errhp, (ub4)1, (text *)NULL, &errcode,
        errbuf, (ub4) sizeof(errbuf), OCI_HTYPE_ERROR
      );
      sprintf(message, "Error: %.*s", 512, errbuf);
      break;
    case OCI_INVALID_HANDLE:
      strcpy(message, "Error: OCI_INVALID_HANDLE");
      break;
    case OCI_STILL_EXECUTING:
      strcpy(message, "Error: OCI_STILL_EXECUTING");
      break;
    case OCI_CONTINUE:
      strcpy(message, "Error: OCI_CONTINUE");
      break;
    default:
      sprintf(message, "Error: %d", (int)status);
      break;
  }

  if (status != OCI_SUCCESS) {
    oci_status = status;
    sprintf(message2, "refcurdesc.c line %d: %s", line, message);
    OCIExtProcRaiseExcpWithMsg(oci_ctx, 20001, message2, 0);
  }
}

void disable_prefetch(OCIStmt *stmt) {
  ub4 prefetch = 0;

  // Set prefetch to 0 in order not to prefetch anything from embedded REF
  // CURSORs.
  CHECKERR(
    OCIAttrSet(
      (dvoid *)stmt, (ub4)OCI_HTYPE_STMT, (dvoid *)&prefetch,
      0, (ub4)OCI_ATTR_PREFETCH_ROWS, errhp
    )
  );
  CHECKERR(
    OCIAttrSet(
      (dvoid *)stmt, (ub4)OCI_HTYPE_STMT, (dvoid *)&prefetch,
      0, (ub4)OCI_ATTR_PREFETCH_MEMORY, errhp
    )
  );
}

LIBRARY refcurdesc

EXPORTS
refcurdesc
refcurdesc2

Compilation Using Microsoft Visual Studio 2010

You can use the following commands to create refcurdesc.dll on Windows (32-bit, assuming that you have called vcvars32.bat to set up PATH, INCLUDE, LIB, etc and also that ORACLE_HOME is set, lines wrapped for legibility):

@setlocal
set INCLUDE=%INCLUDE%;%ORACLE_HOME%\oci\include
set LIB=%LIB%;%ORACLE_HOME%\oci\lib\msvc
cl /c /Zi /nologo /W3 /WX- /O2 /Oi /Oy- /GL /D WIN32 /D NDEBUG /D _WINDOWS /D
  _USRDLL /D REFCURDESC_EXPORTS /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MT
  /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"Release\\"
  /Fd"Release\vc100.pdb" /Gd /TC /analyze- /errorReport:queue refcurdesc.c
link /ERRORREPORT:QUEUE /OUT:"Release\refcurdesc.dll" /INCREMENTAL:NO /NOLOGO
  oci.lib /DEF:"refcurdesc.def" /MANIFEST
  /ManifestFile:"Release\refcurdesc.dll.intermediate.manifest"
  /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG
  /PDB:"Release\refcurdesc.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /LTCG
  /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"Release\refcurdesc.lib" /MACHINE:X86
  /DLL Release\refcurdesc.obj
@endlocal

Compilation Using GCC

You can use the following commands to create refcurdesc.so on Linux (32-bit, assuming that ORACLE_HOME is set):

gcc -I $ORACLE_HOME/rdbms/public -fPIC -c refcurdesc.c
gcc -shared -static-libgcc -o refcurdesc.so refcurdesc.o

Installation

The compiled refcurdesc.dll/refcurdesc.so must be placed somewhere on the database server’s file system such that it can be referred from within Oracle. This is typically done in %ORACLE_HOME%\BIN on Windows and $ORACLE_HOME/bin on Linux. Please be aware that the directory name used for the Oracle library is case sensitive, even on Windows.

Configuration of Extproc Listener

If you get the following error when trying to execute the external procedure

ORA-28595: Extproc agent : Invalid DLL Path

you probably need to amend the configuration of the extproc listener to include a setting like

(ENVS="EXTPROC_DLLS=ANY")

or a more specific

(ENVS="EXTPROC_DLLS=ONLY:C:\ORACLE\O112\BIN\refcurdesc.dll")

Example:

SID_LIST_O112 =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = O112)
      (ORACLE_HOME = c:\ORACLE\O112)
      (SID_NAME = O112)
    )
    (SID_DESC =
      (SID_NAME = PLSExtProcO112)
      (ORACLE_HOME = C:\oracle\O112)
      (PROGRAM = extproc)
      (ENVS="EXTPROC_DLLS=ANY")
    )
    (SID_DESC =
      (SID_NAME = CLRExtProcO112)
      (ORACLE_HOME = C:\oracle\O112)
      (PROGRAM = extproc)
    )
  )

Main Package

The main package REF_CURSOR_DESCRIBE dispatches the incoming call to the specific implementation based on the following precedence order:

1. If Oracle Database 11g Release 1 or newer: Use pure PL/SQL.
2. If Oracle Database 10g Release 1 or 2: Use Java if Java option enabled.
3. If Oracle Database 10g Release 1 or 2: Use C.

If you have problems with this precedence order, you can control which implementation is used through setting of a package variable DESCRIBE_IMPL.
REF_CURSOR_DESCRIBE also adds an extra XML element for each column: DECLARATION, which can be used for declaring a placeholder for fetching the given column. The datatype used for the declaration of a placeholder for an embedded REF CURSOR is ANYDATA, which makes it possible to hold the value in an object type attribute.

Here’s the implementation of the REF_CURSOR_DESCRIBE package specification (ref_cursor_describe.pks):

create or replace package ref_cursor_descriptor
authid current_user as
/**
 * Functionality for describing a REF CURSOR in Oracle Database 10g Release 1 or
 * newer.
 * Feel free to use at your own risk.
 * @version   $Revision: 2 $
 * @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.
 */

  /**
   * Overrides which implementation is used:
   * 'P':  PL/SQL.
   * 'J':  Java using DBMS_ODCI.
   * 'J2': Java direct.
   * 'C':  C using DBMS_ODCI.SAVE-/RESTOREREFCURSOR.
   * 'C2': C direct (not possible according to documentation).
   * '':   PL/SQL for Oracle 11.1 and newer, otherwise Java if Java option
   *       enabled (Java direct on 10.2 and newer, Java indirect on 10.1) and C
   *       if Java is not enabled (C direct).
   */
  describe_impl varchar2(2);

  function describe(rc in out sys_refcursor)
  return xmltype;

  function describe_plsql(rc in out sys_refcursor)
  return xmltype;
  function describe_java(rc in out sys_refcursor)
  return xmltype;
  function describe_c(rc in out sys_refcursor)
  return xmltype;

  function get_declaration(
    name in varchar2,
    type_code in pls_integer,
    charset_form in pls_integer,
    length_semantics in varchar2,
    precision in pls_integer,
    scale in pls_integer,
    length in pls_integer,
    owner in varchar2,
    type_name in varchar2
  )
  return varchar2;

  function quote_identifier(identifier in varchar2)
  return varchar2;
end;
/

create or replace package body ref_cursor_descriptor as
  --- $Revision: 2 $

  --- REF CURSOR number.
  rcn pls_integer;
  --- XML to be returned.
  xml varchar2(32767);
  --- Valid identifier characters.
  identifier_chars constant varchar2(40) :=
      'ABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890_$#';

  function add_declarations(xml in xmltype)
  return xmltype;

/**
 * Describe REF CURSOR and return description in XML form in a VARCHAR2.
 * Description is returned in the following XML format (without the XML
 * comments):
 * <ROWSET generator="PL/SQL|Java|C">
 *   <ROW><!-- Column 1. -->
 *     <ID>column_id_1</ID>
 *     <NAME>column_name_1</NAME>
 *     <TYPE_CODE>data_type_code_1</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_1</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_1</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_1</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_1</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_1</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_1</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_1</TYPE_NAME>
 *     <DECLARATION>declaration_1</DECLARATION>
 *   </ROW>
 *   <!-- Columns 2 through n-1. -->
 *   <ROW><!-- Column n. -->
 *     <ID>column_id_n</ID>
 *     <NAME>column_name_n</NAME>
 *     <TYPE_CODE>data_type_code_n</TYPE_CODE>
 *     <NATIVE_TYPE_CODE>native_data_type_code_n</NATIVE_TYPE_CODE>
 *     <!-- Character set form: 1: Database. 2: National. -->
 *     <CHARSET_FORM>native_data_type_code_n</CHARSET_FORM>
 *     <!-- For [N]CHAR, [N]VARCHAR2, RAW, UROWID. -->
 *     <LENGTH>column_length</LENGTH>
 *     <!-- For CHAR, VARCHAR2. -->
 *     <LENGTH_SEMANTICS>column_name_n</LENGTH_SEMANTICS>
 *     <!-- NUMBER, TIMESTAMP%, INTERVAL%. -->
 *     <PRECISION>precision_n</PRECISION>
 *     <!-- NUMBER, INTERVAL DAY TO SECOND. -->
 *     <SCALE>scale_n</SCALE>
 *     <!-- For object types, including collections. -->
 *     <OWNER>type_owner_n</LENGTH_SEMANTICS>
 *     <!-- For object types, including collections. -->
 *     <TYPE_NAME>type_name_n</TYPE_NAME>
 *     <DECLARATION>declaration_n</DECLARATION>
 *   </ROW>
 * </ROWSET>
 * Main entry. Dispatches to:
 * 1. PL/SQL if Oracle Database version >= 11.
 * 2. Java if Oracle Database version is 10 and Java option is enabled.
 * 3. C if Oracle Database version is 10.
 * Otherwise, an exception is raised.
 * @param   rc      REF CURSOR.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

  function describe(rc in out sys_refcursor)
  return xmltype as

    result xmltype;

  begin
    case nvl(upper(describe_impl), 'D')
      when 'P' then
        result := describe_plsql(rc);
      when 'J' then
        result := describe_java(rc);
      when 'J2' then
        result := describe_java(rc);
      when 'C' then
        result := describe_c(rc);
      when 'C2' then
        result := describe_c(rc);
      when 'D' then
        -- Default.
        if portable.get_major_version >= 11 then
          result := describe_plsql(rc);
        elsif portable.get_major_version = 10 then
          if portable.is_option_enabled('Java') then
            result := describe_java(rc);
          else
            result := describe_c(rc);
          end if;
        else
          raise_application_error(
            -20000,
            'ref_cursor_descriptor.describe: ' ||
                'Oracle Database 10g Release 1 or newer is required'
          );
        end if;
      else
        raise_application_error(
          -20002,
          'ref_cursor_descriptor.describe: ' ||
              'Invalid value of DESCRIBE_IMPL: "' || describe_impl || '"'
        );
    end case;

    return add_declarations(result);
  end describe;

/**
 * At least Oracle 11.1. Pure PL/SQL solution.
 * @param   rc      REF CURSOR.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

  function describe_plsql(rc in out sys_refcursor)
  return xmltype as

  begin
    xml := ref_cursor_descriptor_plsql(rc);

    if xml is not null then
      return xmltype(xml);
    else
      return null;
    end if;
  end describe_plsql;

/**
 * At least Oracle 10.1 and Java option enabled. Java solution.
 * @param   rc      REF CURSOR.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

  function describe_java(rc in out sys_refcursor)
  return xmltype as

  begin
    if upper(describe_impl) = 'J' then
      /*
       * Necessary to make it work for Oracle 10.1 as Oracle on some platforms
       * will transfer the ResultSet as null.
       */
      dbms_odci.saverefcursor(rc, rcn);
      xml := ref_cursor_descriptor_java(rcn);
      dbms_odci.restorerefcursor(rc, rcn);
    else
      -- 10.2 or newer, no need for using DBMS_ODCI, in fact, for 10.2 and 11.1
      -- this can create problems.
      xml := ref_cursor_descriptor_java2(rc);
    end if;

    if xml is not null then
      return xmltype(xml);
    else
      return null;
    end if;
  end describe_java;

/**
 * At least Oracle 9.2. C solution.
 * @param   rc      REF CURSOR.
 * @return  Description of columns in REF CURSOR on XML form.
 *          An exception is raised if the result is larger than
 *          32KB.
 */

  function describe_c(rc in out sys_refcursor)
  return xmltype as

  begin
    if nvl(describe_impl, 'C2') = 'C2' then
      /*
       * Pass REF CURSOR directly even though this is not possible according to
       * documentation. Works on Oracle 9.2 and newer.
       */
      xml := ref_cursor_descriptor_c2(rc);
    elsif describe_impl = 'C' then
      /*
       * Necessary to make it work as we cannot pass on REF CURSORs to external
       * procedures according to documentation.
       */
      rcn := null;
      dbms_odci.saverefcursor(rc, rcn);
      xml := ref_cursor_descriptor_c(rcn);
      dbms_odci.restorerefcursor(rc, rcn);
    end if;

    if xml is not null then
      return xmltype(xml);
    else
      return null;
    end if;
  exception
    when others then
      if sqlcode in (-31011, -19202, -1001) then
        dbms_output.put_line(xml);
        raise;
      else
        raise;
      end if;
  end describe_c;

/**
 * Builds the declaration for given column. ANYDATA is used for REF CURSOR,
 * which makes it possible to fetch the REF CURSOR into an object type attribute
 * of data type ANYDATA (REF CURSOR is not supported for object type
 * attributes).
 * @param   name    Column name.
 * @param   type_code
 *                  "Uniform" type code.
 * @param   charset_form
 *                  Character set form: 1: Database CS, 2: National CS.
 * @param   length_semantics
 *                  Length semantics: 'BYTE' or 'CHAR'.
 * @param   precision
 *                  Precision.
 * @param   scale   Scale.
 * @param   length  Length.
 * @param   owner   Datatype owner (for object types, incl. collections).
 * @param   type_name
 *                  Datatype name (for object types, incl. collections).
 * @return  Declaration of the form '<name> <datatype>'.
 */

  function get_declaration(
    name in varchar2,
    type_code in pls_integer,
    charset_form in pls_integer,
    length_semantics in varchar2,
    precision in pls_integer,
    scale in pls_integer,
    length in pls_integer,
    owner in varchar2,
    type_name in varchar2
  )
  return varchar2 as

    declaration varchar2(1000);

  begin
    case type_code
      when type_codes.tc_varchar2 then
        declaration :=
            case when charset_form = 2 then 'N' else '' end ||
            'VARCHAR2(' || rtrim(length || ' ' || length_semantics) || ')';
      when type_codes.tc_number then
        -- TODO: Special cases.
        if precision = 0 /*and scale = -127*/ then
          declaration := 'NUMBER';
        else
          declaration := 'NUMBER(' || precision;
          if scale != 0 then
            declaration := declaration || ', ' || scale;
          end if;
          declaration := declaration || ')';
        end if;
      when type_codes.tc_long then
        declaration := 'LONG';
      when type_codes.tc_date then
        declaration := 'DATE';
      when type_codes.tc_binary_float then
        declaration := 'BINARY_FLOAT';
      when type_codes.tc_binary_double then
        declaration := 'BINARY_DOUBLE';
      when type_codes.tc_timestamp then
        declaration := 'TIMESTAMP(' || precision || ')';
      when type_codes.tc_timestamp_tz then
        declaration := 'TIMESTAMP(' || precision || ') WITH TIME ZONE';
      when type_codes.tc_timestamp_ltz then
        declaration := 'TIMESTAMP(' || precision || ') WITH LOCAL TIME ZONE';
      when type_codes.tc_interval_ym then
        declaration := 'INTERVAL YEAR(' || precision || ') TO MONTH';
      when type_codes.tc_interval_ds then
        declaration :=
            'INTERVAL DAY(' || precision || ') ' ||
            'TO SECOND(' || scale || ')';
      when type_codes.tc_raw then
        declaration := 'RAW(' || length || ')';
      when type_codes.tc_long_raw then
        declaration := 'LONG RAW';
      when type_codes.tc_rowid then
        -- Map ROWID to VARCHAR2(18), ROWID object type attributes not allowed.
        declaration := 'VARCHAR2(18)';
      when type_codes.tc_urowid then
        -- Map UROWID to VARCHAR2(18), UROWID object type attributes not allowed.
        declaration := 'VARCHAR2(18)';
      when type_codes.tc_char then
        declaration :=
            case when charset_form = 2 then 'N' else '' end ||
            'CHAR(' || length || ' ' || length_semantics || ')';
      when type_codes.tc_clob then
        declaration :=
            case when charset_form = 2 then 'N' else '' end || 'CLOB';
      when type_codes.tc_blob then
        declaration := 'BLOB';
      when type_codes.tc_bfile then
        declaration := 'BFILE';
      when type_codes.tc_object then
        declaration :=
            quote_identifier(owner) ||
            '.' ||
            quote_identifier(type_name);
      when type_codes.tc_ref then
        declaration :=
            'REF ' ||
            quote_identifier(owner) ||
            '.' ||
            quote_identifier(type_name);
      when type_codes.tc_ref_cursor then
        -- REF CURSOR, to be saved in ANYDATA.
        declaration := 'SYS.ANYDATA';
      else
        raise_application_error(
          -20003,
          'ref_cursor_descriptor.get_declaration: ' || type_code || ': ' ||
              'Unknown datatype'
        );
    end case;

    return quote_identifier(name) || ' ' || declaration;
  end get_declaration;

/**
 * Adds DECLARATION element for each column.
 * @param   xml     Description XML.
 * @return  Description XML with DECLARATION element for each column.
 */

  function add_declarations(xml in xmltype)
  return xmltype as

    declaration varchar2(200);
    result xmltype;

    doc dbms_xmldom.domdocument;
    row dbms_xmldom.domnode;
    rows dbms_xmldom.domnodelist;
    decl_element dbms_xmldom.domelement;
    decl_text_element dbms_xmldom.domtext;
    decl_node dbms_xmldom.domnode;
    decl_text_node dbms_xmldom.domnode;
    root dbms_xmldom.domelement;

  begin
    doc := dbms_xmldom.newdomdocument(xml);
    root := dbms_xmldom.getdocumentelement(doc);
    rows := dbms_xmldom.getchildrenbytagname(root, 'ROW');

    for r in (
          select extractvalue(value(r), '//ID') id,
                 extractvalue(value(r), '//NAME') name,
                 extractvalue(value(r), '//TYPE_CODE') type_code,
                 extractvalue(value(r), '//CHARSET_FORM') charset_form,
                 extractvalue(value(r), '//LENGTH') length,
                 extractvalue(value(r), '//LENGTH_SEMANTICS') length_semantics,
                 extractvalue(value(r), '//PRECISION') precision,
                 extractvalue(value(r), '//SCALE') scale,
                 extractvalue(value(r), '//OWNER') OWNER,
                 extractvalue(value(r), '//TYPE_NAME') type_name
          from   table(
                   xmlsequence(
                     extract(xml, '/ROWSET/ROW')
                   )
                 ) r
        ) loop
      -- Get declaration.
      declaration := get_declaration(
        r.name,
        r.type_code,
        r.charset_form,
        r.length_semantics,
        r.precision,
        r.scale,
        r.length,
        r.owner,
        r.type_name
      );

      row := dbms_xmldom.item(rows, r.id - 1);
      -- New element.
      decl_element := dbms_xmldom.createelement(doc, 'DECLARATION');
      decl_node := dbms_xmldom.makenode(decl_element);
      -- New text node.
      decl_text_element := dbms_xmldom.createtextnode(doc, declaration);
      -- Append text node to element node.
      decl_text_node := dbms_xmldom.appendchild(
        decl_node,
        dbms_xmldom.makenode(decl_text_element)
      );
      -- Insert new element in document.
      row := dbms_xmldom.appendchild(row, decl_node);

      dbms_xmldom.freenode(decl_text_node);
      dbms_xmldom.freenode(decl_node);

/* Requires 10.2 or newer.
      -- Append new DECLARATION element to column with given ID.
      result := result.appendchildxml(
        '/ROWSET/ROW[' || r.id || ']',
        xmltype('<DECLARATION>' || declaration || '</DECLARATION>')
      );
 */
    end loop;

    result := dbms_xmldom.getxmltype(doc);
    dbms_xmldom.freedocument(doc);

    return result;
  end add_declarations;

/**
 * Double quotes given identifier if it's necessary.
 * @param   identifier
 *                  Identifier.
 * @return  Identifier, potentially double quoted.
 */

  function quote_identifier(identifier in varchar2)
  return varchar2 as

  begin
    if identifier != upper(identifier) or
        ascii(substr(identifier, 1, 1)) not between ascii('A') and ascii('Z') or
        translate(identifier, ' ' || identifier_chars, ' ') is not null then
      return '"' || identifier || '"';
    end if;

    return identifier;
  end quote_identifier;

begin
  if portable.get_major_minor_version = 10.1 then
    /*
     * Necessary to make it work for Oracle 10.1 as Oracle on some platforms
     * will transfer the ResultSet as null or empty.
     */
    describe_impl := 'J';
  end if;
end ref_cursor_descriptor;
/

Installation

You need to provide the following grants to the schema in which you’re going to install the code:

grant create library to &&1;
grant execute on sys.dbms_system to &&1;
grant create procedure to &&1;

An alternative to the CREATE LIBRARY privilege is to let your DBA create the library and grant EXECUTE privilege on the library to your schema.

The following installation script is assumed to run within SQL*Plus (install.sql). You should run this again if you migrate this solution to another database version or edition as the foundation for the installation discovered whilst installing could potentially have changed:

-- Installation script assmed to be run with SQL*Plus connected to the schema
-- that will own the database objects. You will need write access to the current
-- directory as dynamic scripts are created during the installation.
-- Feel free to use at your own risk.
-- @version   $Revision: 1 $
-- @author    Finn Ellebaek Nielsen, Ellebaek Consulting ApS.

set serveroutput on format truncated
set trimspool on
set linesize 100

@portable.pks
@portable.pkb
@@type_codes.pks

prompt Creating library...

declare
  oracle_home varchar2(255);
  file_path varchar2(255);
  refcurdesc varchar2(14);
  path_separator char(1);
  bin varchar2(3);
  ddl varchar2(1000);
begin
  sys.dbms_system.get_env('ORACLE_HOME', oracle_home);

  if lower(dbms_utility.port_string) like '%win%' then
    refcurdesc := 'refcurdesc.dll';
    path_separator := '\';
    bin := 'BIN';
  else
    refcurdesc := 'refcurdesc.so';
    path_separator := '/';
    bin := 'bin';
  end if;
  file_path := oracle_home || path_separator || bin;

  begin
    ddl :=
        'create or replace library refcurdesc as ''' ||
            file_path || path_separator || refcurdesc ||
            '''';
    execute immediate ddl;

    dbms_output.put_line(
      'Library REFCURDESC created for "' ||
          file_path || path_separator || refcurdesc ||
          '".'
    );

    dbms_output.put_line('Please place ' || refcurdesc || ' in "' || file_path || '".');
  exception
    when others then
      dbms_output.put_line(ddl || ': ' || sqlerrm);
      raise;
  end;
end;
/

prompt done

prompt Creating PL/SQL stored procedure

set feedback off
spool install_plsql.sql

declare
  ddl varchar2(32767);
begin
  if portable.get_major_version >= 11 then
    dbms_output.put_line('@@ref_cursor_descriptor_plsql.fnc');
  else
    dbms_output.put_line('prompt Older than Oracle 11.1');
    dbms_output.put_line('');
    dbms_output.put_line('create or replace function ref_cursor_descriptor_plsql(rc in out sys_refcursor)');
    dbms_output.put_line('return varchar2 as');
    dbms_output.put_line('begin');
    dbms_output.put_line('  raise_application_error(-20002, ''Requires Oracle 11.1 or newer'');');
    dbms_output.put_line('end;');
    dbms_output.put_line('/');
    dbms_output.put_line('');
    dbms_output.put_line('show err');
  end if;
end;
/

spool off
set feedback 1

@@install_plsql.sql

prompt done

prompt Creating Java stored procedure

set feedback off
spool install_java.sql

declare
  ddl varchar2(32767);
begin
  if portable.is_option_enabled('Java') then
    dbms_output.put_line('set scan off');
    dbms_output.put_line('');
    dbms_output.put_line('@@RefCursorDescriptor.sql');
    dbms_output.put_line('');
    dbms_output.put_line('set scan on');
    dbms_output.put_line('');
    dbms_output.put_line('@@ref_cursor_descriptor_java.fnc');
    dbms_output.put_line('@@ref_cursor_descriptor_java2.fnc');
  else
    dbms_output.put_line('prompt Java option not enabled');
    dbms_output.put_line('create or replace function ref_cursor_descriptor_java(rcn in pls_integer)');
    dbms_output.put_line('return varchar2 as');
    dbms_output.put_line('begin');
    dbms_output.put_line('  raise_application_error(-20003, ''Java option not enabled'');');
    dbms_output.put_line('end;');
    dbms_output.put_line('/');
    dbms_output.put_line('');
    dbms_output.put_line('show err');
    dbms_output.put_line('');
    dbms_output.put_line('create or replace function ref_cursor_descriptor_java2(rc in sys_refcursor)');
    dbms_output.put_line('return varchar2 as');
    dbms_output.put_line('begin');
    dbms_output.put_line('  raise_application_error(-20003, ''Java option not enabled'');');
    dbms_output.put_line('end;');
    dbms_output.put_line('/');
    dbms_output.put_line('');
    dbms_output.put_line('show err');
  end if;
end;
/

spool off
set feedback 1

@@install_java.sql

prompt done

prompt Creating C external procedure

@@ref_cursor_descriptor_c.fnc
@@ref_cursor_descriptor_c2.fnc

prompt done

@@ref_cursor_descriptor.pks
@@ref_cursor_descriptor.pkb

@dbms_output_put_line.prc

Example

Here’s an example:

set long 50000

variable xml clob

prompt SELECT * FROM EMP

declare
  rc sys_refcursor;
begin
  open rc for
  select *
  from   emp;

  :xml := ref_cursor_descriptor.describe(rc).getclobval;
end;
/

print xml

<ROWSET generator="C">
  <ROW>
    <ID>1</ID>
    <NAME>EMPNO</NAME>
    <TYPE_CODE>2</TYPE_CODE>
    <NATIVE_TYPE_CODE>2</NATIVE_TYPE_CODE>
    <PRECISION>4</PRECISION>
    <SCALE>0</SCALE>
    <DECLARATION>EMPNO NUMBER(4)</DECLARATION>
  </ROW>
  <ROW>
    <ID>2</ID>
    <NAME>ENAME</NAME>
    <TYPE_CODE>1</TYPE_CODE>
    <NATIVE_TYPE_CODE>1</NATIVE_TYPE_CODE>
    <CHARSET_FORM>1</CHARSET_FORM>
    <LENGTH>10</LENGTH>
    <LENGTH_SEMANTICS>BYTE</LENGTH_SEMANTICS>
    <DECLARATION>ENAME VARCHAR2(10 BYTE)</DECLARATION>
  </ROW>
  <ROW>
    <ID>3</ID>
    <NAME>JOB</NAME>
    <TYPE_CODE>1</TYPE_CODE>
    <NATIVE_TYPE_CODE>1</NATIVE_TYPE_CODE>
    <CHARSET_FORM>1</CHARSET_FORM>
    <LENGTH>9</LENGTH>
    <LENGTH_SEMANTICS>BYTE</LENGTH_SEMANTICS>
    <DECLARATION>JOB VARCHAR2(9 BYTE)</DECLARATION>
  </ROW>
  <ROW>
    <ID>4</ID>
    <NAME>MGR</NAME>
    <TYPE_CODE>2</TYPE_CODE>
    <NATIVE_TYPE_CODE>2</NATIVE_TYPE_CODE>
    <PRECISION>4</PRECISION>
    <SCALE>0</SCALE>
    <DECLARATION>MGR NUMBER(4)</DECLARATION>
  </ROW>
  <ROW>
    <ID>5</ID>
    <NAME>HIREDATE</NAME>
    <TYPE_CODE>12</TYPE_CODE>
    <NATIVE_TYPE_CODE>12</NATIVE_TYPE_CODE>
    <DECLARATION>HIREDATE DATE</DECLARATION>
  </ROW>
  <ROW>
    <ID>6</ID>
    <NAME>SAL</NAME>
    <TYPE_CODE>2</TYPE_CODE>
    <NATIVE_TYPE_CODE>2</NATIVE_TYPE_CODE>
    <PRECISION>7</PRECISION>
    <SCALE>2</SCALE>
    <DECLARATION>SAL NUMBER(7, 2)</DECLARATION>
  </ROW>
  <ROW>
    <ID>7</ID>
    <NAME>COMM</NAME>
    <TYPE_CODE>2</TYPE_CODE>
    <NATIVE_TYPE_CODE>2</NATIVE_TYPE_CODE>
    <PRECISION>7</PRECISION>
    <SCALE>2</SCALE>
    <DECLARATION>COMM NUMBER(7, 2)</DECLARATION>
  </ROW>
  <ROW>
    <ID>8</ID>
    <NAME>DEPTNO</NAME>
    <TYPE_CODE>2</TYPE_CODE>
    <NATIVE_TYPE_CODE>2</NATIVE_TYPE_CODE>
    <PRECISION>2</PRECISION>
    <SCALE>0</SCALE>
    <DECLARATION>DEPTNO NUMBER(2)</DECLARATION>
  </ROW>
</ROWSET>

Subtleties

I’ve found the following subtleties whilst developing the code:

• DBMS_SQL.DESCRIBE_COLUMN% always uses byte length semantics for data types CHAR and VARCHAR2, no matter how they were declared. Eg, if you have declared a table column with VARCHAR2(10 CHAR) in a database with characterset AL32UTF8 this will be reported as VARCHAR2(40 BYTE).
• Inversely, java.sql.ResultSetMetaData always uses character length semantics.
• It’s not possible to distinguish ROWID and UROWID in Java and C so both are mapped to UROWID.

Known Issues

• If you convert a REF CURSOR to a DBMS_SQL cursor and later invoke DBMS_XMLGEN.NEWCONTEXT on that same cursor (after you’ve converted it back to a REF CURSOR) you may encounter the following Oracle errors: ORA-01001: invalid cursor, ORA-24338: statement handle not executed and ORA-00600: internal error code, arguments: [psdmsc: psdinvdef#1]. I’ve only encountered these errors in Oracle 11.1.0.6.0 as they seems to have been fixed in 11.1.0.7.0 and 11.2.0.1.0.
• If you describe a REF CURSOR and then call either Oracle code that is implemented in C (not PL/SQL) or another external procedure written in C you’ll probably get Oracle errors like ORA-24374: define not done before fetch or execute and fetch. The problem seems to be that something on the REF CURSOR OCI statement handle is marked as processed when we obtain it in our C external procedure, which means that eg DBMS_XMLGEN doesn’t define the columns for its fetch, which results in an error. The workaround is to use the direct C implementation (not the indirect one using DBMS_ODCI) or fetch the REF CURSOR to either memory or a table, open a new REF CURSOR on that copy and then invoke DBMS_XMLGEN on the copy. I’ll come back to this in a future blog post.

Supporting Oracle Database 9i Release 2

The techniques in REF_CURSOR_COPY and the related blog can be made to work for Oracle 9.2 as well but you would need to remove calls to DBMS_ODCI.SAVE-/RESTOREREFCURSOR and DBMS_UTILITY.FORMAT_ERROR_BACKTRACE as these procedures are not available in Oracle 9.2.
I haven’t used Conditional Compilation techniques for this as it would exclude Oracle versions 10.1.0.2.0 and 10.1.0.3.0.

Improvements

The following improvements could be implemented:

• Lift the 32KB limitation by using a CLOB instead of a VARCHAR2.
• Make the C implementation thread safe.

Conclusion

The code shown here has been fairly well tested with the following Oracle versions and editions:

• Oracle Database 10g Release 1 10.1.0.2.0 Personal Edition on Windows XP SP3.
• Oracle Database 10g Release 1 10.1.0.3.0 Enterprise Edition on Intel Solaris 10.
• Oracle Database 10g Release 2 10.2.0.1.0 Express Edition on Windows XP SP3.
• Oracle Database 10g Release 2 10.2.0.1.0 Express Edition on Red Hat Enterprise Linux 5.3.
• Oracle Database 10g Release 2 10.2.0.2.0 Enterprise Edition on Intel Solaris 10.
• Oracle Database 11g Release 1 11.1.0.6.0 Personal Edition on Windows XP SP3.
• Oracle Database 11g Release 1 11.1.0.7.0 Personal Edition on Windows XP SP3.
• Oracle Database 11g Release 1 11.1.0.6.0 Enterprise Edition on SPARC Solaris 10.
• Oracle Database 11g Release 2 11.2.0.1.0 Personal Edition on Windows 7.

Feel free to use the code at your own risk. I welcome your feedback and suggestions for improvements but the code as such is not supported.

Source Code

You can download the source code here.

Challenge: Oracle uses different type codes in different contexts. This is fair enough if they need to adhere to standards (eg JDBC), but there’s really no excuse for using different type codes in the documentation, in PL/SQL, in DBMS_TYPES and Oracle Call Interface (OCI). In my view it’s a bit of a mess.

Solution: I’ve spent quite some time mapping between these slightly different type codes and also chosen a “uniform type code” across these. My rule of thumb has been that I use the type code from the Oracle Database 11g Release 2 documentation (Data Types). If the documentation doesn’t provide one, I use the one returned by one of the DBMS_SQL.DESCRIBE_COLUMNS% procedures as this is probably what most Oracle developers have used over time.

I’ve listed the mappings in the table below. If a cell is empty it’s because it’s missing or not supported for that particular type system (eg DUMP doesn’t work with all data types). I’ve underlined where the type codes differ from the Oracle documentation or between DBMS_SQL and the other type systems.

¹ Probably because in Java a true Date cannot hold a time value.

² XMLTYPE, ANYDATA, ANYDATASET and ANYTYPE are not “normal object types”. They are all declared like “CREATE TYPE <OWNER>.<OBJECT_TYPE> AS OPAQUE VARYING (*) USING LIBRARY...“.

The type codes from PL/SQL, Java and C have been verified through creation of a table or SELECT statement using the various data types and description of these through:

• PL/SQL: DBMS_SQL.DESCRIBE_COLUMNS3 on a DBMS_SQL cursor representing the above mentioned table/SELECT.
• Java: java.sql.ResultSetMetaData obtained through the java.sql.ResultSet representing the above mentioned table/SELECT.
• C: OCIParamGet(…, OCI_HTYPE_STMT, …) + OCIAttrGet(…, OCI_ATTR_DATA_TYPE, …) on a statement handle representing the above mentioned table/SELECT.

The type codes have been verified on Oracle Database 11g Release 2 11.2.0.1.0 Personal Edition on Windows 7. When I have more time I’ll create an automated test suite in order to verify my findings on Oracle Database 10g Release 1, Release 2 and Oracle Database 11g Release 1 and also verify on other platforms like Linux and Solaris.

I welcome your feedback and suggestions for improvements and corrections.

This blog post is about comparing a specific type of XML documents with the purpose of obtaining details about their differences, within an Oracle database. This specific type is documents representing table data/resultsets conforming to Oracle’s canonical form, ie XML with the following structure (the comments <!-- ... --> are only included for illustration):

<ROWSET>
  <ROW><!-- Row 1 -->
    <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
    <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
    <!-- ... -->
    <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
  </ROW>
  <ROW><!-- Row 2 -->
    <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
    <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
    <!-- ... -->
    <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
  </ROW>
  <!-- ... -->
  <ROW><!-- Row n -->
    <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
    <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
    <!-- ... -->
    <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
   </ROW>
</ROWSET>

So basically there’s an XML element for each column and the name of the element corresponds to the column name. There’s a <ROW> fragment for each row in the resultset and the root element is called <ROWSET> (all XML documents must have at most one root element).

The diffing method I’m going to describe and implement in this blog post is focusing on a limited version of Oracle canonical form, namely it is assumed that:

• None of the columns in the table are using object types, including collections and XMLTYPE.
• Object tables are not used.
• XML element attributes are not used.
• NULL values are represented by an empty tag, eg <ENAME/> or <ENAME></ENAME>.
• We regard the document order to be significant — like comparing sets in SQL.

This means that we can assume that all the column values are atomic, single and simple values, which means that the comparison implementation can be simplified.

The primary use case for this XML is comparison (or XML diffing) for automated unit/regression tests that need to verify that resultsets from the Program Under Test (PUT) are identical to a given expected outcome. It’s a convenient way to specify the expected outcome in XML and the PUT resultsets can then be converted to XML and compared. My blog post Converting Between Oracle Data and XML describes how to convert the Oracle data to XML.

The differ should return NULL in case there are no differences and it must support node values larger than 32KB.

The code in this blog post requires Oracle Database 11g Release 1 or newer and has been tested on the following versions (both on Windows XP Service Pack 3 32-bit):

• Oracle Database 11g Release 1 11.1.0.6.0.
• Oracle Database 11g Release 2 11.2.0.1.0.

In a future follow up to this blog post I’ll implement a more generic XML differ that will work with Oracle Database 10g Release 1 or newer.

Example

Here’s an example of what I would like to be able to do. Given the following two XML documents:

<ROWSET>
  <ROW>
    <DEPTNO>10</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <LOC>NEW YORK</LOC>
  </ROW>
  <ROW>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <LOC>DALLAS</LOC>
  </ROW>
  <ROW>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <LOC>CHICAGO</LOC>
  </ROW>
  <ROW>
    <DEPTNO>40</DEPTNO>
    <DNAME>OPERATIONS</DNAME>
    <LOC>BOSTON</LOC>
  </ROW>
</ROWSET>

and

<ROWSET>
  <ROW>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <LOC>DALLAS</LOC>
  </ROW>
  <ROW>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <LOC>CHICAGO</LOC>
  </ROW>
  <ROW>
    <DEPTNO>40</DEPTNO>
    <DNAME>XOPERATIONSETC</DNAME>
    <LOC>NEW YORK</LOC>
  </ROW>
  <ROW>
    <DEPTNO>50</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <LOC>ALBANY</LOC>
  </ROW>
</ROWSET>

I would like an XML differ to pick up the following differences and only those:

• Department 10 has been removed.
• Department 50 has been added.
• Department 40′s name and location have changed.

The XML differ should return this in a structured form, preferably in XML, perhaps something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Differencing export generated by Altova® DiffDog® version 2009 sp1
  Enterprise Edition - for more information please visit www.altova.com -->
<diff_result>
  <diff_info comparison_mode="text_or_xml">
    <source_left name="c:\temp\xml1.xml" uri="file:///c:/temp/xml1.xml"/>
    <source_right name="c:\temp\xml2.xml" uri="file:///c:/temp/xml2.xml"/>
  </diff_info>
  <xml_diff>
    <left_location>
      <parent xpath="/ROWSET"/>
      <position>1</position>
    </left_location>
    <right_location>
      <parent xpath="/ROWSET"/>
      <position>1</position>
    </right_location>
    <left_content>
      <element>
        <ROW>
          <DEPTNO>10</DEPTNO>
          <DNAME>ACCOUNTING</DNAME>
          <LOC>NEW YORK</LOC>
        </ROW>
      </element>
    </left_content>
  </xml_diff>
  <xml_diff>
    <left_location>
      <parent xpath="/ROWSET/ROW[4]/DNAME"/>
      <position>1</position>
    </left_location>
    <right_location>
      <parent xpath="/ROWSET/ROW[3]/DNAME"/>
      <position>1</position>
    </right_location>
    <left_content>
      <element>OPERATIONS</element>
    </left_content>
    <right_content>
      <element>XOPERATIONSETC</element>
    </right_content>
  </xml_diff>
  <xml_diff>
    <left_location>
      <parent xpath="/ROWSET/ROW[4]/LOC"/>
      <position>1</position>
    </left_location>
    <right_location>
      <parent xpath="/ROWSET/ROW[3]/LOC"/>
      <position>1</position>
    </right_location>
    <left_content>
      <element>BOSTON</element>
    </left_content>
    <right_content>
      <element>NEW YORK</element>
    </right_content>
  </xml_diff>
  <xml_diff>
    <left_location>
      <parent xpath="/ROWSET"/>
      <position>5</position>
    </left_location>
    <right_location>
      <parent xpath="/ROWSET"/>
      <position>4</position>
    </right_location>
    <right_content>
      <element>
        <ROW>
          <DEPTNO>50</DEPTNO>
          <DNAME>ACCOUNTING</DNAME>
          <LOC>ALBANY</LOC>
        </ROW>
      </element>
    </right_content>
  </xml_diff>
</diff_result>

which is what Altova DiffDog 2009 returns, or even better:

<DIFFERENCES>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="10"]]]></XPATH>
    <OPERATION>DELETE</OPERATION>
    <VALUE1><![CDATA[<ROW>
  <DEPTNO>10</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>NEW YORK</LOC>
</ROW>]]></VALUE1>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="50"]]]></XPATH>
    <OPERATION>INSERT</OPERATION>
    <VALUE2><![CDATA[<ROW>
  <DEPTNO>50</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>ALBANY</LOC>
</ROW>]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="40"]/DNAME]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[OPERATIONS]]></VALUE1>
    <VALUE2><![CDATA[XOPERATIONSETC]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="40"]/LOC]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[BOSTON]]></VALUE1>
    <VALUE2><![CDATA[NEW YORK]]></VALUE2>
  </DIFFERENCE>
</DIFFERENCES>

which provides the same information but in a more compact and useful format.

Alternatives

For XML diffing within the Oracle database we have various alternatives:

• Open source PL/SQL XML Compare Project: DBI_XML_COMPAREDBI_XML_COMPARE.GET_XML_DIFFERENCES.
• Commercial software Quest Code Tester for Oracle’s built-in XML differ: QU_XML.GET_XML_DIFFERENCES.
• Open source Java code that could be loaded into the database as Java stored procedures, eg XMLUnit.
• Oracle’s XMLDIFF function that was introduced in Oracle Database 11g Release 1.
• Write your own from scratch.

Let’s look at each of these alternatives in turn.

PL/SQL XML Compare Project

The function returns a collection of records describing the nodes of the two documents that differ. You can then traverse the result to understand the differences and you could also convert this to XML. Let’s look at what DBI_XML_COMPAREDBI_XML_COMPARE.GET_XML_DIFFERENCES returns when converted to XML using the following code snippet:

create or replace function xmlcdata2(
  c in clob
)
return clob as

/**
 * Wraps CLOB value up in a CDATA section. Oracle's built-in XMLCDATA function
 * was not introduced until 10.2, doesn't supporting "chaining" CDATA sections
 * and doesn't support CLOBs.
 * @param   c       CLOB to be wrapped up in a CDATA section. CDATA section
 *                  chaining is used if C contains the sequence ']]>'.
 * @return  CDATA section.
 */

  result clob;

begin
  dbms_lob.createtemporary(result, false, dbms_lob.call);
  dbms_lob.append(result, '<![CDATA[');
  if c is not null then
    -- Append the value. Replace embedded ]]> -- CDATA chaining.
    dbms_lob.append(result, replace(c, ']]>', ']]]]><![CDATA[>'));
  end if;
  dbms_lob.append(result, ']]>');

  return result;
end xmlcdata2;
/

create or replace function xmldiff_dbi_xml_compare(
  xml1 in xmltype,
  xml2 in xmltype
)
return xmltype as

/**
 * Compares two XML documents and returns the differences in XML. Based on
 * DBI_XML_COMPARE.
 * @param   xml1    First XML document.
 * @param   xml2    Second XML document.
 * @return  Document describing differences based on an XML version of what is
 *          return from DBI_XML_COMPARE.GET_XML_DIFFERENCES. NULL if no
 *          differences.
 */

  result clob;
  differences dbi_xml_compare.t_nodes;

begin
  differences := dbi_xml_compare.get_xml_differences(
    xml1,
    xml2,
    p_ignore_whitespace => dbi_xml_compare.gc_whitespace_normalize
  );

  if differences is not null and differences.count > 0 then
    dbms_lob.createtemporary(result, true, dbms_lob.call);
    dbms_lob.append(result, '<dbi_xml_compare>');
  end if;

  for i in nvl(differences.first, 0) .. nvl(differences.last, -1) loop
    dbms_lob.append(result, '<difference>');
    dbms_lob.append(result, '<document_id>');
    dbms_lob.append(result, xmlcdata2(to_char(differences(i).document_id)));
    dbms_lob.append(result, '</document_id>');
    dbms_lob.append(result, '<node_id>');
    dbms_lob.append(result, xmlcdata2(differences(i).node_id));
    dbms_lob.append(result, '</node_id>');
    dbms_lob.append(result, '<node_values>');

    for j in
        nvl(differences(i).node_values.first, 0) ..
        nvl(differences(i).node_values.last, -1) loop
      dbms_lob.append(result, '<node_value id="' || j || '">');
      dbms_lob.append(result, xmlcdata2(differences(i).node_values(j)));
      dbms_lob.append(result, '</node_value>');
    end loop;

    dbms_lob.append(result, '</node_values>');
    dbms_lob.append(
      result,
      '<difference_type>' ||
        case differences(i).difference_type
          when dbi_xml_compare.gc_identical_value then
            'dbi_xml_compare.gc_identical_value'
          when dbi_xml_compare.gc_different_value then
            'dbi_xml_compare.gc_different_value'
          when dbi_xml_compare.gc_no_equivalent_node then
            'dbi_xml_compare.gc_no_equivalent_node'
          when dbi_xml_compare.gc_ambiguous_element then
            'dbi_xml_compare.gc_ambiguous_element'
        end ||
      '</difference_type></difference>'

    );
  end loop;

  if differences is not null and differences.count > 0 then
    dbms_lob.append(result, '</dbi_xml_compare>');
  end if;

  if result is not null then
    return xmltype(result);
  end if;

  return null;
end xmldiff_dbi_xml_compare;
/

we would get the following differences for our example (abbreviated for legibility, a total of 48 differences were found):

<dbi_xml_compare>
  <difference>
    <document_id><![CDATA[1]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{1}/LOC{NEW YORK}]]></node_id>
    <node_values/>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
  <difference>
    <document_id><![CDATA[1]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{4}/LOC{BOSTON}]]></node_id>
    <node_values/>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
  <!-- snip -->
  <difference>
    <document_id><![CDATA[1]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{1}/DEPTNO{10}]]></node_id>
    <node_values/>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
  <difference>
    <document_id><![CDATA[2]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{4}/DNAME{ACCOUNTING}/#text]]></node_id>
    <node_values>
      <node_value id="1"><![CDATA[ACCOUNTING]]></node_value>
    </node_values>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
  <difference>
    <document_id><![CDATA[2]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{4}/LOC{ALBANY}]]></node_id>
    <node_values/>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
  <!-- snip -->
  <difference>
    <document_id><![CDATA[2]]></document_id>
    <node_id><![CDATA[/#document/ROWSET{1}/ROW{4}/DNAME{ACCOUNTING}]]></node_id>
    <node_values/>
    <difference_type>dbi_xml_compare.gc_no_equivalent_node</difference_type>
  </difference>
</dbi_xml_compare>

which clearly is not what we need — basically the differ got so confused that it couldn’t match anything.

This was tested with DBI_XML_COMPARE version 1.3. DBI_XML_COMPARE does not support element values larger than 32KB.

Quest Code Tester for Oracle

Like DBI_XML_COMPAREDBI_XML_COMPARE.GET_XML_DIFFERENCES, QU_XML.GET_XML_DIFFERENCES returns a collection of records. Using the following snippet to convert this to XML:

create or replace function xmldiff_qu_xml(
  xml1 in xmltype,
  xml2 in xmltype
)
return xmltype as

/**
 * Compares two XML documents and returns the differences in XML. Based on
 * QU_XML.
 * @param   xml1    First XML document.
 * @param   xml2    Second XML document.
 * @return  Document describing differences based on an XML version of what is
 *          return from QU_XML.GET_XML_DIFFERENCES. NULL if no differences.
 */

  nodes qu_xml.t_nodes;
  value varchar2(32767);

  result clob;

begin
  nodes := qu_xml.get_xml_differences(xml1, xml2);

  if nodes.count > 0 then
    dbms_lob.createtemporary(result, true, dbms_lob.call);
    dbms_lob.append(result, '<qu_xml>');
    for i in nvl(nodes.first, 0) .. nvl(nodes.last, -1) loop
      if nodes(i).node_values.count > 0 then
        value := nodes(i).node_values(nodes(i).node_values.first);
      else
        value := '';
      end if;
      dbms_lob.append(result, '<diff_node>');
      dbms_lob.append(result, '<name>');
      dbms_lob.append(result, xmlcdata2(nodes(i).node_id));
      dbms_lob.append(result, '</name>');
      dbms_lob.append(result, '<value>');
      dbms_lob.append(result, xmlcdata2(value));
      dbms_lob.append(result, '</value>');
      dbms_lob.append(result, '<difference>' ||
          case nodes(i).difference_type
            when 1 then
              'Value difference'
            when 2 then
              'Node removed/added'
            when 3 then
              'Ambiguous'
          end ||
          '</difference></diff_node>'
      );
    end loop;
    dbms_lob.append(result, '</qu_xml>');
  end if;

  if result is not null then
    return xmltype(result);
  end if;

  return null;
end xmldiff_qu_xml;
/

we would get the following differences (abbreviated for legibility, a total of 48 differences were found):

<qu_xml>
  <diff_node>
    <name><![CDATA[/#document/ROWSET{1}/ROW{1}/LOC{NEW YORK}]]></name>
    <value><![CDATA[]]></value>
    <difference>Node removed/added</difference>
  </diff_node>
  <diff_node>
    <name><![CDATA[/#document/ROWSET{1}/ROW{4}/LOC{BOSTON}]]></name>
    <value><![CDATA[]]></value>
    <difference>Node removed/added</difference>
  </diff_node>
  <!-- snip -->
  <diff_node>
    <name><![CDATA[/#document/ROWSET{1}/ROW{4}/DNAME{ACCOUNTING}]]></name>
    <value><![CDATA[]]></value>
    <difference>Node removed/added</difference>
  </diff_node>
</qu_xml>

This was tested with Quest Code Tester for Oracle 1.9.1.505. QU_XML does not support element values larger than 32KB.

Open Source Java Code

It would be convenient if we could find an open source Java project that meets our requirements. If one was found, we could load it into the database using loadjava, assuming that the JDK used by the Java code is compatible with the JDK version of our Oracle database. One such open source project to investigate is XMLUnit, which is an extension to JUnit — the Java unit testing framework.

Here’s some Java code that uses XMLUnit for comparison of our example (setup of the XML documents has been left out):

package com.appatra.blog;

import com.topologi.diffx.DiffXException;

import java.io.IOException;

import java.util.List;

import javax.xml.transform.TransformerConfigurationException;

import org.custommonkey.xmlunit.DetailedDiff;
import org.custommonkey.xmlunit.Difference;
import org.custommonkey.xmlunit.XMLTestCase;

import org.jdom.JDOMException;

import org.xml.sax.SAXException;

/**
 * Test of XMLUnit's capabilities of XML comparison.
 */

public class XmlDiffXmlUnit extends XMLTestCase {
  public XmlDiffXmlUnit() {
    super("XmlDiffXmlUnit");
  }

  private void test()
  throws SAXException, IOException {
    String xml1 = /* Left out... */;
    String xml2 = /* Left out... */;

    DetailedDiff myDiff = new DetailedDiff(compareXML(xml2, xml1));
    List<Difference> allDifferences = (List<Difference>)myDiff.getAllDifferences();

    for (Difference diff : allDifferences) {
      System.err.println(diff);
    }
  }

  public static void main(String[] args)
  throws SAXException, IOException {
    XmlDiffXmlUnit xmlDiffXmlUnit = new XmlDiffXmlUnit();

    xmlDiffXmlUnit.test();
  }
}

This produces the following output (12 differences, wrapped for improved legibility):

Expected text value '20' but was '10' -
  comparing <DEPTNO ...>20</DEPTNO> at
  /ROWSET[1]/ROW[1]/DEPTNO[1]/text()[1] to
  <DEPTNO ...>10</DEPTNO> at /ROWSET[1]/ROW[1]/DEPTNO[1]/text()[1]
Expected text value 'RESEARCH' but was 'ACCOUNTING' -
  comparing <DNAME ...>RESEARCH</DNAME> at
  /ROWSET[1]/ROW[1]/DNAME[1]/text()[1] to
  <DNAME ...>ACCOUNTING</DNAME> at /ROWSET[1]/ROW[1]/DNAME[1]/text()[1]
Expected text value 'DALLAS' but was 'NEW YORK' -
  comparing <LOC ...>DALLAS</LOC> at
  /ROWSET[1]/ROW[1]/LOC[1]/text()[1] to
  <LOC ...>NEW YORK</LOC> at /ROWSET[1]/ROW[1]/LOC[1]/text()[1]
Expected text value '30' but was '20' -
  comparing <DEPTNO ...>30</DEPTNO> at
  /ROWSET[1]/ROW[2]/DEPTNO[1]/text()[1] to
  <DEPTNO ...>20</DEPTNO> at /ROWSET[1]/ROW[2]/DEPTNO[1]/text()[1]
Expected text value 'SALES' but was 'RESEARCH' -
  comparing <DNAME ...>SALES</DNAME> at
  /ROWSET[1]/ROW[2]/DNAME[1]/text()[1] to
  <DNAME ...>RESEARCH</DNAME> at /ROWSET[1]/ROW[2]/DNAME[1]/text()[1]
Expected text value 'CHICAGO' but was 'DALLAS' -
  comparing <LOC ...>CHICAGO</LOC> at
  /ROWSET[1]/ROW[2]/LOC[1]/text()[1] to
  <LOC ...>DALLAS</LOC> at /ROWSET[1]/ROW[2]/LOC[1]/text()[1]
Expected text value '40' but was '30' -
  comparing <DEPTNO ...>40</DEPTNO> at
  /ROWSET[1]/ROW[3]/DEPTNO[1]/text()[1] to
  <DEPTNO ...>30</DEPTNO> at /ROWSET[1]/ROW[3]/DEPTNO[1]/text()[1]
Expected text value 'XOPERATIONSETC' but was 'SALES' -
  comparing <DNAME ...>XOPERATIONSETC</DNAME> at
  /ROWSET[1]/ROW[3]/DNAME[1]/text()[1] to
  <DNAME ...>SALES</DNAME> at /ROWSET[1]/ROW[3]/DNAME[1]/text()[1]
Expected text value 'NEW YORK' but was 'CHICAGO' -
  comparing <LOC ...>NEW YORK</LOC> at
  /ROWSET[1]/ROW[3]/LOC[1]/text()[1] to
  <LOC ...>CHICAGO</LOC> at /ROWSET[1]/ROW[3]/LOC[1]/text()[1]
Expected text value '50' but was '40' -
  comparing <DEPTNO ...>50</DEPTNO> at
  /ROWSET[1]/ROW[4]/DEPTNO[1]/text()[1] to
  <DEPTNO ...>40</DEPTNO> at /ROWSET[1]/ROW[4]/DEPTNO[1]/text()[1]
Expected text value 'ACCOUNTING' but was 'OPERATIONS' -
  comparing <DNAME ...>ACCOUNTING</DNAME> at
  /ROWSET[1]/ROW[4]/DNAME[1]/text()[1] to
  <DNAME ...>OPERATIONS</DNAME> at /ROWSET[1]/ROW[4]/DNAME[1]/text()[1]
Expected text value 'ALBANY' but was 'BOSTON' -
  comparing <LOC ...>ALBANY</LOC> at
  /ROWSET[1]/ROW[4]/LOC[1]/text()[1] to
  <LOC ...>BOSTON</LOC> at /ROWSET[1]/ROW[4]/LOC[1]/text()[

This was tested with XMLUnit version 1.3.

XMLDIFF

Oracle introduced a new function XMLDIFF in Oracle Database 11g Release 1. This function generates an XML document with the differences conforming to an Xdiff schema. Let's look at how this function works on our example:

create or replace function xmldiff_xmldiff(
  xml1 in xmltype,
  xml2 in xmltype
)
return xmltype as

/**
 * Compares two XML documents and returns the differences in XML. Based on
 * XMLDIFF, hence only supported with Oracle Database 11g Release 1 and newer.
 * @param   xml1    First XML document.
 * @param   xml2    Second XML document.
 * @return  Document describing differences based on an XML version of what is
 *          return from QU_XML.GET_XML_DIFFERENCES. NULL if no differences.
 */

  diff xmltype;

begin
  select xmldiff(xml1, xml2)
  into   diff
  from   dual;
  if diff is not null then
    if diff.extract('/*/*') is not null then
      return diff;
    end if;
  end if;

  return null;
end xmldiff_xmldiff;
/

which returns the following based on our example (once again, 12 differences):

<xd:xdiff
    xsi:schemaLocation="http://xmlns.oracle.com/xdb/xdiff.xsd
    http://xmlns.oracle.com/xdb/xdiff.xsd"
    xmlns:xd="http://xmlns.oracle.com/xdb/xdiff.xsd"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <?oracle-xmldiff operations-in-docorder="true"
    output-model="snapshot"
    diff-algorithm="global"?>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[1]/DEPTNO[1]/text()[1]">
    <xd:content>20</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[1]/DNAME[1]/text()[1]">
    <xd:content>RESEARCH</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[1]/LOC[1]/text()[1]">
    <xd:content>DALLAS</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[2]/DEPTNO[1]/text()[1]">
    <xd:content>30</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[2]/DNAME[1]/text()[1]">
    <xd:content>SALES</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[2]/LOC[1]/text()[1]">
    <xd:content>CHICAGO</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[3]/DEPTNO[1]/text()[1]">
    <xd:content>40</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[3]/DNAME[1]/text()[1]">
    <xd:content>XOPERATIONSETC</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[3]/LOC[1]/text()[1]">
    <xd:content>NEW YORK</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[4]/DEPTNO[1]/text()[1]">
    <xd:content>50</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[4]/DNAME[1]/text()[1]">
    <xd:content>ACCOUNTING</xd:content>
  </xd:update-node>
  <xd:update-node xd:node-type="text"
      xd:xpath="/ROWSET[1]/ROW[4]/LOC[1]/text()[1]">
    <xd:content>ALBANY</xd:content>
  </xd:update-node>
</xd:xdiff>

Common Issue

A common problem with these alternatives is that none of these differs has a concept of a primary key for an XML fragment. In our example, the XML is Oracle’s canoncial form representing the data in the SCOTT.DEPT table. The primary key of this table is the DEPTNO column. If the differs had known this or had spent time analyzing the XML and detected it, they would have produced a much more useful output, instead of getting very confused and thinking that everything has changed.

The last one of the alternatives was to write your own XML differ from scratch and I’ll come back to this in a later blog post. For now, let’s look at how we can improve on XMLDIFF by writing a wrapper around it that detects primary key usage and internally uses XMLDIFF to diff the fragments for each primary key value found.

XMLDIFF2

The algorithm in our extended XMLDIFF function that we’re going to call XMLDIFF2 is:

1. Prepare CLOB variable for constructing the difference XML.
2. Find minimum number of leading elements that form primary key values (unique lookup).
3. Look for primary key values in XML1 not present in XML2: These primary key values have been deleted.
4. Look for primary key values in XML2 not present in XML1: These primary key values have been inserted.
5. Diff all common primary key values, ie in both XML1 and XML2.
6. Return NULL if no differences found.

The implementation uses the XMLCDATA2 function that I’ve listed earlier. Here is the listing of XMLDIFF2:

create or replace function xmldiff2(
  xml1 in xmltype,
  xml2 in xmltype
)
return xmltype as

/**
 * Extension to Oracle's XMLDIFF function, available with Oracle 11.1 and newer.
 * This version is to be used primarily with simple XML documents in Oracles
 * "canonical form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Assumptions: It is assumed that none of the column elements in the XML
 * contain XML, such that the maximum level of XML elements is 3. It is also
 * assumed that datetime XML elements are passed in using XML Schema compliant
 * formats ('yyyy-mm-dd"T"hh24:mi:ss' for DATE, 'yyyy-mm-dd"T"hh24:mi:ss.ff9'
 * for TIMESTAMP and 'yyyy-mm-dd"T"hh24:mi:ss.ff9tzh:tzm' for TIMESTAMP WITH
 * [LOCAL] TIME ZONE).
 * Extensions: This version understands a concept of leading elements at level 3
 * forming a primary key and ignores document element order.
 * Differences are returned in the following form:
 * <DIFFERENCES>
 *   <DIFFERENCE>
 *     <XPATH><![CDATA[<XPATH1>]]></XPATH>
 *     <OPERATION>(INSERT|DELETE|UPDATE)</OPERATION>
 *     <VALUE1><![CDATA[<VALUE_FROM_XML1>]]></VALUE1>
 *     <VALUE2><![CDATA[<VALUE_FROM_XML2>]]></VALUE2>
 *   </DIFFERENCE>
 *   <DIFFERENCE>
 *     <XPATH><![CDATA[<XPATH1>]]></XPATH>
 *     <OPERATION>(INSERT|DELETE|UPDATE)</OPERATION>
 *     <VALUE1><![CDATA[<VALUE_FROM_XML1>]]></VALUE1>
 *     <VALUE2><![CDATA[<VALUE_FROM_XML2>]]></VALUE2>
 *   </DIFFERENCE>
 *   ...
 *   <DIFFERENCE>
 *     <XPATH><![CDATA[<XPATH1>]]></XPATH>
 *     <OPERATION>(INSERT|DELETE|UPDATE)</OPERATION>
 *     <VALUE1><![CDATA[<VALUE_FROM_XML1>]]></VALUE1>
 *     <VALUE2><![CDATA[<VALUE_FROM_XML2>]]></VALUE2>
 *   </DIFFERENCE>
 * </DIFFERENCES>
 * <VALUE1> is only present for DELETE and UPDATE operations. <VALUE2> is only
 * present for INSERT and UPDATE operations.
 * Limitations. Namespace and attributes are not supported. Works only with
 * Oracle 11.1 and newer.
 * @param   xml1    First XML document.
 * @param   xml2    Second XML document.
 * @return  Differences, NULL if none found.
 */

  ---  Xdiff namespace.
  xmldiff_ns constant varchar2(48) :=
      'xmlns:xd="http://xmlns.oracle.com/xdb/xdiff.xsd"';

  --- Element overview, ie name and count.
  type element_overview_t is record (
    name   varchar2(4000),
    occurs integer
  );
  type level2_elements_c is
  table of element_overview_t
  index by binary_integer;

  --- Level 2 element overview for XML1.
  level2_elements1 level2_elements_c;
  --- Level 2 element overview for XML2.
  level2_elements2 level2_elements_c;

  -- Primary key XPath expressions.
  type varchar2_4000_lookup is
  table of boolean
  index by varchar2(4000);
  pkvs varchar2_4000_lookup;

  --- Number of elements required to form primary key.
  pk_element_count integer := null;

  --- Assuming Oracle canonica form: Root element name.
  element1_name constant varchar2(6) := 'ROWSET';
  --- Assuming Oracle canonica form: Element name for each row (level 2).
  element2_name constant varchar2(3) := 'ROW';

  ---
  type pk_element_t is record (
    name  varchar2(4000),
    value varchar2(4000)
  );
  type pk_element_c is
  table of pk_element_t
  index by binary_integer;
  pk_elements pk_element_c;

  -- Extract of XML1.
  xml1e xmltype;
  -- Extract of XML2.
  xml2e xmltype;
  --- Result from SYS.XMLDIFF.
  diff2 xmltype;
  --- Return result.
  result xmltype;

  --- Column value for XML1.
  cv1 clob;
  --- Column value for XML2.
  cv2 clob;
  --- Primary key XPath.
  pk_xpath varchar2(32767);
  --- Primary key XPath for XML1.
  pk_xpath1 varchar2(4000);
  --- Primary key XPath for XML2.
  pk_xpath2 varchar2(4000);
  --- Primary key value.
  pkv varchar2(4000);
  --- Column name.
  cname varchar2(32767);

  -- XPath.
  xpath varchar2(4000);

  --- Difference document.
  diff clob;
  --- XML update fragment.
  xmlu clob;

  procedure diff_nodes;

/**
 * Parses the level 2 elements in given XML document.
 * @param   xml   XML document.
 * @return  Collection of level 2 elements.
 */

  function get_level2_elements(
    xml in xmltype
  )
  return level2_elements_c as

    level2_elements level2_elements_c;

  begin
    select name,
           count(*)
    bulk collect into level2_elements
    from   (
             select x.getrootelement() name
             from   table(xmlsequence(extract(xml, '/*/*'))) x
           )
    group  by name;

    return level2_elements;
  end get_level2_elements;

/**
 * Finds the number of leading elements it takes to form a unique lookup, ie the
 * minimum number of elements that are required for a primary key.
 * @param   xml     XML document.
 * @param   level2_elements
 *                  Collection of level 2 elements.
 * @return  Number of elements in primary key for this document.
 */

  function get_pk_element_count(
    xml in xmltype,
    level2_elements in level2_elements_c
  )
  return pls_integer as

    j pls_integer;
    n pls_integer := 1;
    m pls_integer;
    xpath varchar2(32767);
    pk_count pls_integer;
    done boolean := false;

  begin
    for i in 1 .. level2_elements(1).occurs loop
      -- For each element in the collection.
      j := 1;
      xpath := '/' || element1_name || '/' || element2_name || '[';
      for pk in (
            select value(x) element
            from   table(
                     xmlsequence(
                       extract(xml, '/' || element1_name || '/' ||
                           element2_name || '[' || i || ']/*'
                       )
                     )
                   ) x
          ) loop
        -- For each element for the ith element of the collection.
        pk_elements(j).name := pk.element.getrootelement;
        pk_elements(j).value := pk.element.extract('//text()').getstringval;

        if j > 1 then
          xpath := rtrim(xpath, ']') || ' and ';
        end if;
        xpath := xpath || pk_elements(j).name || '="' ||
            pk_elements(j).value || '"';
        xpath := xpath || ']';

        if j = n then
          select count(*)
          into   pk_count
          from   table(xmlsequence(extract(xml, xpath)));

          done := pk_count = 1 or j = 32;
          exit when done;

          n := n + 1;
        end if;

        j := j + 1;
      end loop;
    end loop;

    return n;
  end get_pk_element_count;

/**
 * Appends a fragment to the differences XML document.
 */

  procedure append_fragment(
    fragment in clob
  ) as

  begin
    dbms_lob.append(diff, fragment);
  end append_fragment;

/**
 * Check that each fragment from first document identified through each primary
 * key can be found in the second document.
 * @param   operation
 *                  Operation to use for difference fragment, ie 'DELETE' if
 *                  it's missing or 'INSERT' if it has been inserted.
 * @param   xml1    First XML document.
 * @param   xml2    Second XML document.
 */

  procedure check_existence(
    operation in varchar2,
    xml1 in xmltype,
    xml2 in xmltype
  ) as

    relative_xpath varchar2(32767);
    pk_exists boolean;
    value_id pls_integer;

  begin
    for collection_elements in (
          select value(x) node
          from   table(
                   xmlsequence(
                     extract(
                       xml1, '/' || element1_name || '/' || element2_name
                     )
                   )
                 ) x
        ) loop
      relative_xpath := '/' || element1_name || '/' || element2_name || '[';
      for i in 1 .. pk_element_count loop
        if i > 1 then
          relative_xpath := relative_xpath || ' and ';
        end if;
        relative_xpath := relative_xpath || pk_elements(i).name || '="' ||
            collection_elements.node.extract(
              '/*/*[' || i || ']/text()'
            ).getstringval || '"';
      end loop;
      relative_xpath := relative_xpath || ']';

      pk_exists := pkvs.exists(relative_xpath);
      if pk_exists or xml2.existsnode(relative_xpath) = 1 then
        pkvs(relative_xpath) := true;
      else
        dbms_lob.createtemporary(xmlu, true, dbms_lob.call);
        dbms_lob.append(
          xmlu,
          '<DIFFERENCE>' ||
            '<XPATH>' || xmlcdata2(relative_xpath) || '</XPATH>' ||
            '<OPERATION>' || operation || '</OPERATION>'
        );
        if operation = 'INSERT' then
          value_id := 2;
        elsif operation = 'DELETE' then
          value_id := 1;
        end if;
        dbms_lob.append(xmlu, '<VALUE' || value_id || '>');
        dbms_lob.append(
          xmlu,
          xmlcdata2(rtrim(xml1.extract(relative_xpath).getclobval, chr(10)))
        );
        dbms_lob.append(xmlu, '</VALUE' || value_id || '>');
        dbms_lob.append(xmlu, '</DIFFERENCE>');
        append_fragment(xmlu);
      end if;
    end loop;
  end check_existence;

/**
 * Diff the fragments found for the primary key values common between XML1 and
 * XML2. This is done by calling Oracle's XMLDIFF function.
 */

  procedure diff_common_pks as

  begin
    pkv := pkvs.first;
    while pkv is not null loop
      xpath := pkv;
      -- Note the extra GETCLOBVAL and XMLTYPE. Seem that Oracle XMLTYPE has a
      -- bug here as XML1E = XML1 if not done!
      xml1e := xmltype(xml1.extract(xpath).getclobval);
      xml2e := xmltype(xml2.extract(xpath).getclobval);

      select xmldiff(xml1e, xml2e)
      into   diff2
      from   dual;

      diff_nodes;

      pkv := pkvs.next(pkv);
    end loop;
  end diff_common_pks;

/**
 * Inspect the diff result from Oracle's XMLDIFF and append to our difference
 * document.
 */

  procedure diff_nodes as

    node_type varchar2(100);
    operation varchar2(6);
    diff_type varchar2(18);
    xpath xmltype;
    content xmltype;

  begin
    for cv in (
          select value(x) v
          from   table(xmlsequence(extract(diff2, '/xd:xdiff/*', xmldiff_ns))) x
        ) loop
      -- For each difference found.
      xpath := cv.v.extract('//@xd:xpath', xmldiff_ns);
      if xpath is null then
        xpath := cv.v.extract('//@xd:parent-xpath', xmldiff_ns);
      end if;
      cname := xpath.getstringval;
      cname := substr(cname, instr(cname, '/', 2) + 1);
      cname := substr(cname, 1, instr(cname, '[') - 1);
      node_type := cv.v.extract('//@xd:node-type', xmldiff_ns).getstringval;

      diff_type := cv.v.getrootelement;
      if diff_type like '%delete-node' then
        -- Node was deleted. Was it an element value or an element?
        cv1 := xml1e.extract('//' || cname || '/text()').getclobval;
        cv2 := '';
        if node_type = 'text' then
          -- Was non-NULL but is now NULL.
          operation := 'UPDATE';
        else
          -- Node has been removed.
          operation := 'DELETE';
        end if;
      elsif diff_type like '%append-node' then
        -- Node has been added. Was it an element value or an element?
        cv1 := '';
        cv2 := xml2e.extract('//' || cname || '/text()').getclobval;
        if node_type = 'text' then
          operation := 'UPDATE';
        else
          -- Node has been inserted.
          operation := 'INSERT';
        end if;
      elsif diff_type like '%insert-node-before' then
        -- Node has been added, find which.
        content := cv.v.extract('/*/xd:content/*', xmldiff_ns);
        cname := content.getrootelement;
        cv1 := '';
        cv2 := content.extract('//text()').getclobval;
        operation := 'INSERT';
      elsif diff_type like '%update-node' then
        -- Node has been updated.
        cv1 := xml1e.extract('//' || cname || '/text()').getclobval;
        cv2 := cv.v.extract('//xd:content/text()', xmldiff_ns).getclobval;
        operation := 'UPDATE';
      end if;

      dbms_lob.createtemporary(xmlu, false, dbms_lob.call);
      dbms_lob.append(
        xmlu,
        '<DIFFERENCE>' ||
          '<XPATH>' || xmlcdata2(pkv || '/' || cname) || '</XPATH>' ||
          '<OPERATION>' || operation || '</OPERATION>'
      );

      if operation in ('DELETE', 'UPDATE') then
        dbms_lob.append(xmlu, '<VALUE1>');
        dbms_lob.append(xmlu, xmlcdata2(cv1));
        dbms_lob.append(xmlu, '</VALUE1>');
      end if;
      if operation in ('INSERT', 'UPDATE') then
        dbms_lob.append(xmlu, '<VALUE2>');
        dbms_lob.append(xmlu, xmlcdata2(cv2));
        dbms_lob.append(xmlu, '</VALUE2>');
      end if;

      dbms_lob.append(xmlu, '</DIFFERENCE>');
      append_fragment(xmlu);
    end loop;
  end diff_nodes;

begin
  pk_xpath1 := substr(pk_xpath, 1, instr(pk_xpath, '/', -1) - 1);
  pk_xpath2 := substr(pk_xpath, length(pk_xpath1) + 2);

  dbms_lob.createtemporary(diff, true, dbms_lob.call);
  dbms_lob.append(diff, '<DIFFERENCES>');
  -- TODO: NULL handling.

  level2_elements1 := get_level2_elements(xml1);
  level2_elements2 := get_level2_elements(xml2);

  -- Find minimum number of leading elements that form PK values.
  pk_element_count := greatest(
    get_pk_element_count(xml1, level2_elements1),
    get_pk_element_count(xml2, level2_elements2)
  );

  -- Look for PKs in XML1 not present in XML2: DELETE fragment.
  check_existence('DELETE', xml1, xml2);

  -- Look for PKs in XML2 not present in XML1: INSERT fragment.
  check_existence('INSERT', xml2, xml1);

  -- Diff all common PKs: UPDATE fragment.
  diff_common_pks;

  dbms_lob.append(diff, '</DIFFERENCES>');

  if length(diff) = 13 + 14 and diff = '<DIFFERENCES></DIFFERENCES>' then
    -- No differences.
    result := null;
  else
    result := xmltype(diff);
  end if;

  return result;
end xmldiff2;
/

Examples

Here are a few examples on how to use XMLDIFF2.

The first example is the example used for comparison of the various alternatives described earlier:

declare
  xml1 xmltype := xmltype('
<ROWSET>
  <ROW>
    <DEPTNO>10</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <LOC>NEW YORK</LOC>
  </ROW>
  <ROW>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <LOC>DALLAS</LOC>
  </ROW>
  <ROW>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <LOC>CHICAGO</LOC>
  </ROW>
  <ROW>
    <DEPTNO>40</DEPTNO>
    <DNAME>OPERATIONS</DNAME>
    <LOC>BOSTON</LOC>
  </ROW>
</ROWSET>');
  xml2 xmltype := xmltype('
<ROWSET>
  <ROW>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <LOC>DALLAS</LOC>
  </ROW>
  <ROW>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <LOC>CHICAGO</LOC>
  </ROW>
  <ROW>
    <DEPTNO>40</DEPTNO>
    <DNAME>XOPERATIONSETC</DNAME>
    <LOC>NEW YORK</LOC>
  </ROW>
  <ROW>
    <DEPTNO>50</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <LOC>ALBANY</LOC>
  </ROW>
</ROWSET>');
  diff xmltype;
begin
  diff := xmldiff2(xml1, xml2);
  dbms_output.put_line(diff.getclobval(0, 2));
end;
/

which results in:

<DIFFERENCES>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="10"]]]></XPATH>
    <OPERATION>DELETE</OPERATION>
    <VALUE1><![CDATA[<ROW>
  <DEPTNO>10</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>NEW YORK</LOC>
</ROW>]]></VALUE1>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="50"]]]></XPATH>
    <OPERATION>INSERT</OPERATION>
    <VALUE2><![CDATA[<ROW>
  <DEPTNO>50</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>ALBANY</LOC>
</ROW>]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="40"]/DNAME]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[OPERATIONS]]></VALUE1>
    <VALUE2><![CDATA[XOPERATIONSETC]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[DEPTNO="40"]/LOC]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[BOSTON]]></VALUE1>
    <VALUE2><![CDATA[NEW YORK]]></VALUE2>
  </DIFFERENCE>
</DIFFERENCES>

Another example, this time with just one row in the resultset:

declare
  xml1 xmltype := xmltype('
<ROWSET>
  <ROW>
    <EMPNO>7934</EMPNO>
    <ENAME>MILLER</ENAME>
    <JOB>CLERK</JOB>
    <MGR>7782</MGR>
    <HIREDATE>1982-01-23T00:00:00</HIREDATE>
    <SAL>1300</SAL>
    <COMM/>
    <DEPTNO>10</DEPTNO>
  </ROW>
</ROWSET>');
  xml2 xmltype := xmltype('
<ROWSET>
  <ROW>
    <EMPNO>7934</EMPNO>
    <ENAME>MILLER</ENAME>
    <JOB>ANALYST</JOB>
    <MGR>7782</MGR>
    <HIREDATE>1982-01-23T00:00:00</HIREDATE>
    <SAL>1500</SAL>
    <COMM>100</COMM>
    <DEPTNO>20</DEPTNO>
  </ROW>
</ROWSET>');
  diff xmltype;
begin
  diff := xmldiff2(xml1, xml2);
  dbms_output.put_line(diff.getclobval(0, 2));
end;
/

which results in the following:

<DIFFERENCES>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/JOB]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[CLERK]]></VALUE1>
    <VALUE2><![CDATA[ANALYST]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/SAL]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[1300]]></VALUE1>
    <VALUE2><![CDATA[1500]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/COMM]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[]]></VALUE1>
    <VALUE2><![CDATA[100]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/DEPTNO]]></XPATH>
    <OPERATION>UPDATE</OPERATION>
    <VALUE1><![CDATA[10]]></VALUE1>
    <VALUE2><![CDATA[20]]></VALUE2>
  </DIFFERENCE>
</DIFFERENCES>

And a final example, this time an element that is removed and an element that is added:

declare
  xml1 xmltype := xmltype('
<ROWSET>
  <ROW>
    <EMPNO>7934</EMPNO>
    <ENAME>MILLER</ENAME>
    <JOB>CLERK</JOB>
    <MGR>7782</MGR>
    <HIREDATE>1982-01-23T00:00:00</HIREDATE>
    <REMOVED>abc</REMOVED>
    <SAL>1300</SAL>
    <COMM/>
    <DEPTNO>10</DEPTNO>
  </ROW>
</ROWSET>');
  xml2 xmltype := xmltype('
<ROWSET>
  <ROW>
    <EMPNO>7934</EMPNO>
    <ENAME>MILLER</ENAME>
    <JOB>CLERK</JOB>
    <MGR>7782</MGR>
    <ADDED>xyz</ADDED>
    <HIREDATE>1982-01-23T00:00:00</HIREDATE>
    <SAL>1300</SAL>
    <COMM/>
    <DEPTNO>10</DEPTNO>
  </ROW>
</ROWSET>');
  diff xmltype;
begin
  diff := xmldiff2(xml1, xml2);
  dbms_output.put_line(diff.getclobval(0, 2));
end;
/

which gives the following result:

<DIFFERENCES>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/ADDED]]></XPATH>
    <OPERATION>INSERT</OPERATION>
    <VALUE2><![CDATA[xyz]]></VALUE2>
  </DIFFERENCE>
  <DIFFERENCE>
    <XPATH><![CDATA[/ROWSET/ROW[EMPNO="7934"]/REMOVED]]></XPATH>
    <OPERATION>DELETE</OPERATION>
    <VALUE1><![CDATA[abc]]></VALUE1>
  </DIFFERENCE>
</DIFFERENCES>

Conclusion

I’ve described different options for comparing XML documents in the Oracle database and implemented an enhanced version of Oracle’s XMLDIFF that can be used to compare SQL datasets for relational tables not using object types, when represented on Oracle canonical form.

References

• Altova DiffDog: http://www.altova.com/diffdog.
• PL/SQL XML Compare Utility: http://sourceforge.net/projects/plsql-xml-diff/.
• Quest Code Tester for Oracle: http://www.quest.com/code-tester-for-oracle/.
• XMLUnit: http://xmlunit.sourceforge.net/.

This blog post describes various mechanisms for converting between Oracle data and XML, and vice versa, focusing on the former. Many blog posts have been written on this subject. Most are very short and only demonstrate the usage but not the associated issues and I’ll try to dig a little deeper than most others and also mention the issues I’ve encountered or are aware of using these techniques.

I find combining Oracle data and XML useful for the following use cases:

• Extraction, Transformation and Load (ETL) for a datawarehouse or other types of transfers: Data is on a structured form with widespread tool (XML editors), programming language and standard support (eg XSLT, JAX, DOM etc) and is easier to transform in the process.
• Easier to manipulate in a decent XML editor with expand/collapse (hiding irrelevant data sets) and XPath functionality (used for extracting fragments of the XML document).
• Comparison: Comparing data sets for equality, eg in automated unit or regression testing. It’s easier to write a good, generic XML differ than a good, generic SQL result set differ. My next blog post will cover this.

The examples in this blog post assume that you are logged on as SCOTT and that you work in a SQL*Plus environment with the following settings and data in DEPT and EMP tables intact from installation:

set serveroutput on format truncated
set long 100000
set pagesize 50000
set linesize 1000

alter session set nls_date_format = 'dd.mm.yyyy hh24:mi:ss';
alter session set nls_timestamp_format = 'dd.mm.yyyy hh24:mi:ss.ff';
alter session set nls_timestamp_tz_format = 'dd.mm.yyyy hh24:mi:ss.ff tzh:tzm';

Oracle Functionality

In the following sections I’ll give an overview of the functionality Oracle provides in terms of converting between Oracle data and XML. Most of this functionality works on what Oracle calls canonical form, which is XML with the following structure (the comments <!-- ... --> are only included for illustration):

<ROWSET>
 <ROW><!-- Row 1 -->
  <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
  <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
  <!-- ... -->
  <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 </ROW>
 <ROW><!-- Row 2 -->
 <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
  <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
  <!-- ... -->
  <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 </ROW>
 <!-- ... -->
 <ROW><!-- Row n -->
  <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
  <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
  <!-- ... -->
  <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 </ROW>
</ROWSET>

In this blog post I’ll focus on DBMS_XMLGEN and XMLTYPE.

DBMS_XMLQUERY

DBMS_XMLQUERY provides functionality for obtaining XML data based on a dynamic SQL SELECT statement, optionally using bind parameters (all of datatype VARCHAR2). It’s possible to control names of tags generated and you can also control the date/time format (same format for all) but unfortunately ownly down to milliseconds as internally, Java’s java.text.SimpleDateFormat is used, which doesn’t support higher precision than milliseconds. Also, it’s inconvenient that it’s not possible to have empty elements with no attributes for NULL values, only either no element or an empty element with an attribute NULL="YES". XML is returned as CLOB.

There are two ways of using DBMS_XMLQUERY:

• Quick and easy in one call, but less flexible: GETXML that works on a SQLQUERY parameter of type VARCHAR2 or CLOB and returning a CLOB.
• Fully flexible by a series of calls, all working on a context handle:
  • NEWCONTEXT called with SQLQUERY parameter of type VARCHAR2 or CLOB and returning a context handle of type CTXTYPE. This handle must be used in subsequent calls.
  • One or more calls that set various options controlling the XML generation (the SET% procedures), such as NULL value handling, number of rows to process, bind variable values, tag names, etc.
  • CLOSECONTEXT called with the context handle, which closes the context and frees resources.

DBMS_XMLQUERY was introduced with Oracle 8i Release 3 and it has been superseded by DBMS_XMLGEN. DBMS_XMLQUERY is implemented in Java and hence is not supported with Oracle Database Express Edition.

DBMS_XMLGEN

DBMS_XMLGEN was introduced with Oracle Database 9i Release 1. It supersedes DBMS_XMLQUERY, is implemented in C (hence faster) and is a little more flexible than DBMS_XMLQUERY in some ways so in general you should use DBMS_XMLGEN over DBMS_XMLQUERY. Bind variable support (all of datatype VARCHAR2), XSLT support and pretty printing was introduced in Oracle Database 10g Release 1.

In general, DBMS_XMLGEN mimics the functionality in DBMS_XMLQUERY with subtle differences. One of these is that it’s not possible to pass in a query larger than 32KB, but contrary to DBMS_XMLQUERY you can also pass in a SYS_REFCURSOR, so you can use the OPEN-FOR statement that from Oracle Database 11g Release 1 supports CLOB. It’s also possible to obtain the XML as an XMLTYPE, not only a CLOB.

Here are a few examples:

Selecting all rows and columns from SCOTT.DEPT will produce the following result:

select dbms_xmlgen.getxml('select * from dept order by deptno') xml
from   dual;

XML
---------------------------
<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <DEPTNO>10</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>NEW YORK</LOC>
 </ROW>
 <ROW>
  <DEPTNO>20</DEPTNO>
  <DNAME>RESEARCH</DNAME>
  <LOC>DALLAS</LOC>
 </ROW>
 <!-- Abbreviated... -->
 <ROW>
  <DEPTNO>40</DEPTNO>
  <DNAME>OPERATIONS</DNAME>
  <LOC>BOSTON</LOC>
 </ROW>
</ROWSET>

select dbms_xmlgen.getxml('select * from emp order by empno') xml
from   dual;

XML
--------------------------------
<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <EMPNO>7369</EMPNO>
  <ENAME>SMITH</ENAME>
  <JOB>CLERK</JOB>
  <MGR>7902</MGR>
  <HIREDATE>17.12.1980 00:00:00</HIREDATE>
  <SAL>800</SAL>
  <DEPTNO>20</DEPTNO>
 </ROW>
 <ROW>
  <EMPNO>7499</EMPNO>
  <ENAME>ALLEN</ENAME>
  <JOB>SALESMAN</JOB>
  <MGR>7698</MGR>
  <HIREDATE>20.02.1981 00:00:00</HIREDATE>
  <SAL>1600</SAL>
  <COMM>300</COMM>
  <DEPTNO>30</DEPTNO>
 </ROW>
 <!-- Abbreviated... -->
 <ROW>
  <EMPNO>7934</EMPNO>
  <ENAME>MILLER</ENAME>
  <JOB>CLERK</JOB>
  <MGR>7782</MGR>
  <HIREDATE>23.01.1982 00:00:00</HIREDATE>
  <SAL>1300</SAL>
  <DEPTNO>10</DEPTNO>
 </ROW>
</ROWSET>

variable xml clob

declare
  context dbms_xmlgen.ctxtype;
begin
  context := dbms_xmlgen.newcontext('select * from emp where empno = 7934');
  dbms_xmlgen.setnullhandling(context, dbms_xmlgen.empty_tag);
  :xml := dbms_xmlgen.getxml(context);
  dbms_xmlgen.closecontext(context);
end;
/

print :xml

XML
------------------------------------------
<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <EMPNO>7934</EMPNO>
  <ENAME>MILLER</ENAME>
  <JOB>CLERK</JOB>
  <MGR>7782</MGR>
  <HIREDATE>23.01.1982 00:00:00</HIREDATE>
  <SAL>1300</SAL>
  <COMM/>
  <DEPTNO>10</DEPTNO>
 </ROW>
</ROWSET>

variable xml clob

declare
  context dbms_xmlgen.ctxtype;
begin
  context := dbms_xmlgen.newcontext('select * from emp where empno = 7934');
  dbms_xmlgen.setnullhandling(context, dbms_xmlgen.null_attr);
  dbms_xmlgen.setindentationwidth(context, 2);
  :xml := dbms_xmlgen.getxml(context);
  dbms_xmlgen.closecontext(context);
end;
/

print :xml

XML
----------------------------------------------------------------
<?xml version="1.0"?>
<ROWSET xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance">
  <ROW>
    <EMPNO>7934</EMPNO>
    <ENAME>MILLER</ENAME>
    <JOB>CLERK</JOB>
    <MGR>7782</MGR>
    <HIREDATE>23.01.1982 00:00:00</HIREDATE>
    <SAL>1300</SAL>
    <COMM xsi:nil = "true"/>
    <DEPTNO>10</DEPTNO>
  </ROW>
</ROWSET>

If you work with LONG columns it’s worth knowing that DBMS_XMLGEN truncates values of LONG columns to 32KB. For more information, please refer to Adrian Billington’s blog post mentioned at the end of this blog post. I’m also describing a slightly different technique for converting LONG to CLOB in my blog post Converting a LONG Column to a CLOB on the fly.

XMLELEMENT, XMLFOREST, XMLAGG etc

Oracle provides a collections of functions that you can use to construct the XML based on your relational (optionally object-relational) data. Here’s an example on a hierarchical XML construction with employees under departments:

select xmlelement(
         "DEPARTMENTS",
         xmlagg(
           xmlelement(
             "DEPARTMENT",
             xmlforest(
               d.deptno,
               d.dname,
               (
                 select xmlagg(
                          xmlelement(
                            "EMPLOYEE",
                             xmlforest(
                               e.empno,
                               xmlcdata(e.ename) ename,
                               e.hiredate
                             )
                           )
                        )
                 from   emp e
                 where  e.deptno = d.deptno and
                        e.empno in (7369, 7499, 7934)
               ) employees
             )
           )
         )
       ) x
from   dept d;

(output wrapped and indented for improved readability)

<DEPARTMENTS>
  <DEPARTMENT>
    <DEPTNO>10</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7934</EMPNO>
        <ENAME><![CDATA[MILLER]]></ENAME>
        <HIREDATE>1982-01-23</HIREDATE>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7369</EMPNO>
        <ENAME><![CDATA[SMITH]]></ENAME>
        <HIREDATE>1980-12-17</HIREDATE>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7499</EMPNO>
        <ENAME><![CDATA[ALLEN]]></ENAME>
        <HIREDATE>1981-02-20</HIREDATE>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>40</DEPTNO>
    <DNAME>OPERATIONS</DNAME>
  </DEPARTMENT>
</DEPARTMENTS>

select xmlelement(
         "DEPARTMENTS",
         xmlagg(
           xmlelement(
             "DEPARTMENT",
             xmlforest(
               d.deptno,
               d.dname,
               (
                 select xmlagg(
                          xmlelement(
                            "EMPLOYEE",
                             xmlelement("EMPNO", e.empno),
                             xmlelement("ENAME", xmlcdata(e.ename)),
                             xmlelement("HIREDATE", e.hiredate),
                             xmlelement("COMM", e.comm)
                           )
                        )
                 from   emp e
                 where  e.deptno = d.deptno and
                        e.empno in (7369, 7499, 7934)
               ) employees
             )
           )
         )
       ).getclobval() x
from   dept d;

(output wrapped and indented for improved readability)

<DEPARTMENTS>
  <DEPARTMENT>
    <DEPTNO>10</DEPTNO>
    <DNAME>ACCOUNTING</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7934</EMPNO>
        <ENAME><![CDATA[MILLER]]></ENAME>
        <HIREDATE>1982-01-23</HIREDATE>
        <COMM></COMM>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>20</DEPTNO>
    <DNAME>RESEARCH</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7369</EMPNO>
        <ENAME><![CDATA[SMITH]]></ENAME>
        <HIREDATE>1980-12-17</HIREDATE>
        <COMM></COMM>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>30</DEPTNO>
    <DNAME>SALES</DNAME>
    <EMPLOYEES>
      <EMPLOYEE>
        <EMPNO>7499</EMPNO>
        <ENAME><![CDATA[ALLEN]]></ENAME>
        <HIREDATE>1981-02-20</HIREDATE>
        <COMM>300</COMM>
      </EMPLOYEE>
    </EMPLOYEES>
  </DEPARTMENT>
  <DEPARTMENT>
    <DEPTNO>40</DEPTNO>
    <DNAME>OPERATIONS</DNAME>
  </DEPARTMENT>
</DEPARTMENTS>

Please be aware that DATE values with time parts are truncated to the date value only, which is very unfortunate:

select xmlelement(
         "DATETIME",
         xmlforest(
           trunc(sysdate) date_no_time,
           sysdate date_with_time,
           systimestamp timestamp_with_tz,
           localtimestamp timestamp_local_tz
         )
       ) xml
from   dual;

(output wrapped and indented for improved readability)

<DATETIME>
  <DATE_NO_TIME>2011-01-26</DATE_NO_TIME>
  <DATE_WITH_TIME>2011-01-26</DATE_WITH_TIME>
  <TIMESTAMP_WITH_TZ>2011-01-26T10:47:38.026000+01:00</TIMESTAMP_WITH_TZ>
  <TIMESTAMP_LOCAL_TZ>2011-01-26T10:47:38.026000</TIMESTAMP_LOCAL_TZ>
</DATETIME>

select xmlelement(
         "DATETIME",
         xmlforest(
           trunc(sysdate) date_no_time,
           cast(sysdate as timestamp) date_with_time,
           systimestamp timestamp_with_tz,
           localtimestamp timestamp_local_tz
         )
       ) xml
from   dual;

(output wrapped and indented for improved readability)

<DATETIME>
  <DATE_NO_TIME>2011-01-26</DATE_NO_TIME>
  <DATE_WITH_TIME>2011-01-26T10:50:32.000000</DATE_WITH_TIME>
  <TIMESTAMP_WITH_TZ>2011-01-26T10:47:38.026000+01:00</TIMESTAMP_WITH_TZ>
  <TIMESTAMP_LOCAL_TZ>2011-01-26T10:47:38.026000</TIMESTAMP_LOCAL_TZ>
</DATETIME>

XMLTYPE

The XMLTYPE object type has various means of conversions, eg from any object type instance to corresponding XML (CREATEXML static function and constructor taking a parameter of type "<ADT_1>") and vice versa, ie from the XMLTYPE instance to an object type instance (member procedure TOOBJECT).

Here's an example:

select xmltype(sys.aq$_agent('name', 'address', 1)).getclobval() xml
from   dual;

XML
--------------------------------------------------------------------------------
<AQ_x0024__AGENT><NAME>name</NAME><ADDRESS>address</ADDRESS><PROTOCOL>1</PROTOCO
L></AQ_x0024__AGENT>

select xmltype(sys.aq$_agent('name', 'address', 1)).getclobval(0, 2) xml
from   dual;

XML
----------------------------
<AQ_x0024__AGENT>
  <NAME>name</NAME>
  <ADDRESS>address</ADDRESS>
  <PROTOCOL>1</PROTOCOL>
</AQ_x0024__AGENT>

DBMS_XMLSAVE

DBMS_XMLSAVE can be used to INSERT, UPDATE and DELETE data in a table based on XML in Oracle canonical form. It works on a context handle that you obtain, set options, call either INSERTXML, UPDATEXML or DELETEXML and then close the context again.

Example:

select *
from   dept
order  by deptno;

declare
  context dbms_xmlsave.ctxtype;
  row_count integer;
  xml_insert xmltype := xmltype('
<ROWSET>
 <ROW>
  <DEPTNO>50</DEPTNO>
  <DNAME>ACCOUNTING</DNAME>
  <LOC>ALBANY</LOC>
 </ROW>
 <ROW>
  <DEPTNO>60</DEPTNO>
  <DNAME>TO_BE_DELETED</DNAME>
  <LOC>N/A</LOC>
 </ROW>
</ROWSET>
  ');
  xml_update xmltype := xmltype('
<ROWSET>
 <ROW>
  <DEPTNO>40</DEPTNO>
  <DNAME>XOPERATIONSETC</DNAME>
  <LOC>NEW YORK</LOC>
 </ROW>
</ROWSET>
  ');
  xml_delete xmltype := xmltype('
<ROWSET>
 <ROW>
  <DEPTNO>60</DEPTNO>
 </ROW>
</ROWSET>
  ');
begin
  -- Create new context for INSERT.
  context := dbms_xmlsave.newcontext('SCOTT.DEPT');
  -- Insert rows.
  row_count := dbms_xmlsave.insertxml(context, xml_insert.getclobval);
  -- Close the context.
  dbms_xmlsave.closecontext(context);
  dbms_output.put_line(row_count || ' row(s) inserted.');

  -- Create new context for UPDATE.
  context := dbms_xmlsave.newcontext('SCOTT.DEPT');
  -- Set primary key column for UPDATE to work.
  dbms_xmlsave.clearupdatecolumnlist(context);
  dbms_xmlsave.setkeycolumn(context, 'DEPTNO');
  -- UPDATE the document.
  row_count := dbms_xmlsave.updatexml(context, xml_update.getclobval);
  -- Close the context.
  dbms_xmlsave.closecontext(context);
  dbms_output.put_line(row_count || ' row(s) updated.');

  -- Create new context for DELETE.
  context := dbms_xmlsave.newcontext('SCOTT.DEPT');
  -- Set primary key column for DELETE to work.
  dbms_xmlsave.clearupdatecolumnlist(context);
  dbms_xmlsave.setkeycolumn(context, 'DEPTNO');
  -- DELETE the document.
  row_count := dbms_xmlsave.deletexml(context, xml_delete.getclobval);
  -- Close the context.
  dbms_xmlsave.closecontext(context);
  dbms_output.put_line(row_count || ' row(s) deleted.');
end;
/

select *
from   dept
order  by deptno;

rollback;

    DEPTNO DNAME          LOC
---------- -------------- -------------
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 OPERATIONS     BOSTON

2 row(s) inserted.
1 row(s) updated.
1 row(s) deleted.

PL/SQL procedure successfully completed.

    DEPTNO DNAME          LOC
---------- -------------- -------------
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 XOPERATIONSETC NEW YORK
        50 ACCOUNTING     ALBANY

Rollback complete.

DBMS_XMLSTORE

DBMS_XMLSTORE was introduced with Oracle Database 10g Release 1 and should be used instead of DBMS_XMLSAVE in order to work around limitations in the latter, such as unsupported TIMESTAMP fractional seconds with precisions larger than 3 decimal places and also that the time part of DATE values is mandatory. DBMS_XMLSTORE is written in C and provides better performance than DBMS_XMLSAVE. DBMS_XMLSTORE uses NLS settings for date/time values so you must be careful to set these for XML Schema compliance as described elsewhere in this blog post.

Example: As the example for DBMS_XMLSAVE, simply replace DBMS_XMLSAVE with DBMS_XMLSTORE.

DBMS_METADATA

DBMS_METADATA can be used to obtain DDL for your database objects. This can either be returned as DDL or XML. Example:

select dbms_metadata.get_xml('TABLE', 'DEPT') xml
from   dual;

(Output abbreviated for legibility)

DBMS_METADATA.GET_XML('TABLE','DEPT')
------------------------------------------------------------
<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <TABLE_T>
   <VERS_MAJOR>1</VERS_MAJOR>
   <VERS_MINOR>3 </VERS_MINOR>
   <OBJ_NUM>73194</OBJ_NUM>
   <SCHEMA_OBJ>
    <OBJ_NUM>73194</OBJ_NUM>
    <DATAOBJ_NUM>73194</DATAOBJ_NUM>
    <OWNER_NUM>84</OWNER_NUM>
    <OWNER_NAME>SCOTT</OWNER_NAME>
    <NAME>DEPT</NAME>
    <NAMESPACE>1</NAMESPACE>
    <!-- Snip -->
   </SCHEMA_OBJ>
   <STORAGE>
    <FILE_NUM>4</FILE_NUM>
    <BLOCK_NUM>130</BLOCK_NUM>
    <TYPE_NUM>5</TYPE_NUM>
    <TS_NUM>4</TS_NUM>
    <BLOCKS>8</BLOCKS>
    <EXTENTS>1</EXTENTS>
    <INIEXTS>8</INIEXTS>
    <MINEXTS>1</MINEXTS>
    <MAXEXTS>2147483645</MAXEXTS>
    <EXTSIZE>128</EXTSIZE>
    <!-- Snip -->
   </STORAGE>
   <COL_LIST>
    <COL_LIST_ITEM>
     <OBJ_NUM>73194</OBJ_NUM>
     <COL_NUM>1</COL_NUM>
     <INTCOL_NUM>1</INTCOL_NUM>
     <SEGCOL_NUM>1</SEGCOL_NUM>
     <PROPERTY>0</PROPERTY>
     <NAME>DEPTNO</NAME>
     <TYPE_NUM>2</TYPE_NUM>
     <LENGTH>22</LENGTH>
     <PRECISION_NUM>2</PRECISION_NUM>
     <SCALE>0</SCALE>
     <NOT_NULL>1</NOT_NULL>
     <!-- Snip -->
    </COL_LIST_ITEM>
   </COL_LIST>
  </TABLE_T>
 </ROW>
</ROWSET>

XSQL

Oracle XSQL pages publishing framework is an extensible platform for publishing XML in multiple formats. It can also be used to call PL/SQL APIs, INSERT/UPDATE/DELETE data based on one (unfortunately only one) posted document. It has been available since Oracle Database 8i Release 3 and is accessible through either a Java Servlet hosted on your application server, the command line (xsql.bat batch file/xsql shell script) or your own Java program. XSQL is not available from inside the database, but you can load the oraclexsql.jar into the database with loadjava and write your own PL/SQL wrapper for it. Example of how to insert two rows into DEPT and query all the rows afterwards:
Contents of dept.xsql:

<page connection="scott" xmlns:xsql="urn:oracle-xsql">
  <xsql:insert-request table="dept"/>
  <xsql:query>
    select *
    from   dept
  </xsql:query>
</page>

<ROWSET>
 <ROW>
  <DEPTNO>-1</DEPTNO>
  <DNAME>Dept A</DNAME>
  <LOC>Loc A</LOC>
 </ROW>
 <ROW>
  <DEPTNO>-2</DEPTNO>
  <DNAME>Dept B</DNAME>
  <LOC>Loc B</LOC>
 </ROW>
</ROWSET>

@setlocal
rem %ORACLE_HOME%\xdk\admin\XSQLConfig.xml must be configured with a connection called "scott".
rem Set all CP in C:\Oracle\O112\BIN\xsql.bat
set ORACLE_HOME=c:\oracle\o112
set JAVA_HOME=%ORACLE_HOME%\jdk
set CP=%ORACLE_HOME%\RDBMS\jlib\xdb.jar
xsql.bat dept.xsql posted-xml=dept.xml
@endlocal

<page>
 <xsql-status action="xsql:insert-request" rows="2"/>
 <ROWSET>
  <ROW num="1">
   <DEPTNO>-1</DEPTNO>
   <DNAME>Dept A</DNAME>
   <LOC>Loc A</LOC>
  </ROW>
  <ROW num="2">
   <DEPTNO>-2</DEPTNO>
   <DNAME>Dept B</DNAME>
   <LOC>Loc B</LOC>
  </ROW>
  <ROW num="3">
   <DEPTNO>10</DEPTNO>
   <DNAME>ACCOUNTING</DNAME>
   <LOC>NEW YORK</LOC>
  </ROW>
  <ROW num="4">
   <DEPTNO>20</DEPTNO>
   <DNAME>RESEARCH</DNAME>
   <LOC>DALLAS</LOC>
  </ROW>
  <ROW num="5">
   <DEPTNO>30</DEPTNO>
   <DNAME>SALES</DNAME>
   <LOC>CHICAGO</LOC>
  </ROW>
  <ROW num="6">
   <DEPTNO>40</DEPTNO>
   <DNAME>OPERATIONS</DNAME>
   <LOC>BOSTON</LOC>
  </ROW>
 </ROWSET>
</page>

XML Schema Compliance

In order to be compliant with XML Schema, all DATE and TIMESTAMP values must adhere to the following Oracle formats:

• DATE: 'yyyy-mm-dd"T"hh24:mi:ss' (for time part as well, note the T between date and time part) or 'yyyy-mm-dd' without time part.
• TIMESTAMP: 'yyyy-mm-dd"T"hh24:mi:ss.ff9' or 'yyyy-mm-dd"T"hh24:mi:ss.ff'.
• TIMESTAMP WITH [LOCAL] TIME ZONE: 'yyyy-mm-dd"T"hh24:mi:ss.ff9tzh:tzm' or 'yyyy-mm-dd"T"hh24:mi:ss.fftzh:tzm'.

Unfortunately, Oracle's DBMS_XMLGEN and XMLTYPE (plus XMLELEMENT, XMLFOREST etc prior to Oracle Database 10g Release 2) do not adhere to the XML Schema standard and instead by default they generate the values according to the current NLS settings. Your session's NLS_DATE_FORMAT, NLS_TIMESTAMP_FORMAT and NLS_TIMESTAMP_TZ_FORMAT must be set to one of the values mentioned above in order to work around this.

XML Character Entities

The default behavior is that characters that have a special meaning in XML are converted to their corresponding XML character entities as it would otherwise be invalid XML. Such characters are <, >, " and ', which are converted to <, >, " and ' respectively:

select dbms_xmlgen.getxml(q'[select '<>"''' test from dual]') xml
from   dual;

XML
-----------------------------------
<?xml version="1.0"?>
<ROWSET>
 <ROW>
  <TEST>&lt;&gt;&quot;&apos;</TEST>
 </ROW>
</ROWSET>

Encoding

Encoding of an XML document can be specified in the document prolog, such as:

<?xml version="1.0" encoding="ISO-8859-1"?>
<blogs>
  <blog>
    <url>ellebaek.wordpress.com</url>
    <author>Finn Ellebæk Nielsen</author>
  </blog>
</blogs>

Code

I've written a PL/SQL package in order to demonstrate how to generate XML based on Oracle data, using DBMS_XMLGEN, XMLTYPE and a few tricks. My primary goal was to achieve the the following functionality:

• Get XML from dynamic SQL query with optional support for bind variables.
• Get XML from open REF CURSOR.
• Get XML from object type instance.
• Get XML from collection.
• Get XML from ANYDATA instance.
• Get XML from ANYDATASET instance.
• Get XML from ANYTYPE instance.

The code has been tested with the following releases and editions (all 32-bit versions under Windows XP SP3):

• Oracle Database 10g Release 1 10.1.0.5.0 Personal Edition.
• Oracle Database 10g Release 2 10.2.0.4.0 Personal Edition.
• Oracle Database 10g Release 2 10.2.0.1.0 Express Edition.
• Oracle Database 11g Release 1 11.1.0.6.0 Personal Edition.
• Oracle Database 11g Release 2 11.2.0.1.0 Personal Edition.

The first version of this package had the following issues:

1. It used packaged types for the bind variable collection. This is fine if you use PL/SQL to call the function FROM_SELECT with a bind variable collection. However, it wasn't possible to call this function directly from a PL/SQL expression or from SQL, so I changed the implementation to use two object types instead.
2. FROM_REF_CURSOR didn't generate date values in the XML Schema compliant format. Instead, date values were generated according to the NLS_DATE_FORMAT active when the REF CURSOR was opened, at least on some Oracle versions on some platforms and sometimes only for the first row. This was attempted fixed by calling DBMS_XMLGEN.RESTARTQUERY after having changed the NLS settings but this also proved unstable.
3. Changing the NLS settings on the fly has the following side effect: Any call to TO_CHAR/TO_DATE/TO_TIMESTAMP etc in the incoming query/REF CURSOR (including PL/SQL functions called) without a specific date/time format will change semantics as the result will adhere to the XML Schema compliant formats (for dates, truncated to the least of the lengths of the original NLS_DATE_FORMAT and the XML compliant NLS_DATE_FORMAT), instead of the original NLS setting in place when calling XML_GEN. This may seem critical, however it's considered poor practice to call these functions without a specific format.

Anyway, in order to work around the issues with XML Schema-compliant date/time values I decided to fix this for the third updated version of the code. My idea was to implement some code that would do the following:

• Fetch the dynamic SQL/REF CURSOR into memory or a table.
• Change the NLS settings for XML Schema-compliance.
• Open a new REF CURSOR on the copy in memory/table.
• Call DBMS_XMLGEN on the new REF CURSOR.
• Change the NLS settings back to the previous session NLS.

This way we would have no side effects as the NLS would be changed after the original data had been fetched. However, this proved to be a major task to implement and in fact I've written two separate blog posts about it:

• Describing a REF CURSOR in Oracle 10g+ Using PL/SQL, Java and C
• Copying/Transforming a REF CURSOR in Oracle 10g+

In fact, had I known it would have been that difficult, I would probably have chosen to write an improved version of DBMS_XMLGEN instead. Perhaps a subject for a future blog post or product?

You control the NLS handling through the optional parameter DATE_TIME_OPTION. If using this with a value of either DATE_TIME_XML_SET_NLS_COPY_MEM or DATE_TIME_XML_SET_NLS_COPY_TAB you need the version of REF_CURSOR_COPY from this blog post: Copying/Transforming a REF CURSOR in Oracle 10g+.

Here is the code:

create or replace type bind_variable_t as object (

/**
 * Bind variables: Name and value.
 * @version     $Revision: 1 $
 */

  --- Name.
  name varchar2(30),
  --- Value.
  value varchar2(32767)
);
/

create or replace type bind_variable_c as

/**
 * Unordered collection of bind variables.
 * @version     $Revision: 1 $
 */

table of bind_variable_t;
/

create or replace package xml_gen
authid current_user as
/**
 * XML utilities for converting Oracle data and objects to XML.
 * @version   $Revision: 3 $
 * @author    Finn Ellebaek Nielsen.
 */

  /**
   * Generate date/time values according to current session NLS settings.
   */
  date_time_sess_nls             constant pls_integer := 1;

  /**
   * Generate XML Schema-compliant date/time values. This is done by internally
   * by changing the NLS settings and generating the XML. This has the side
   * effect that any implit/explicit calls to TO_CHAR without a specific format
   * mask will work on the internal NLS setting. Also, for REF CURSORs, the
   * internal NLS setting will not be in effect in all cases, because Oracle
   * pre-allocated bindings to the session NLS in effect when the REF CURSOR was
   * opened.
   */
  date_time_xml_set_nls          constant pls_integer := 2;

  /**
   * Generate XML Schema-compliant date/time values. This is done by internally
   * by changing the NLS settings, restarting the query and generating the XML.
   * For dynamic SQL, this has the side effect that any implit/explicit calls to
   * TO_CHAR without a specific format mask will work on the internal NLS
   * setting. For REF CURSORs, the internal NLS setting sometimes will be in
   * effect because the REF CURSOR was restarted. However, when is very unstable
   * across Oracle versions. On some versions DATE columns will have a truncated
   * value if the session NLS_DATE_FORMAT was shorter than the XML Schema-
   * compliant one, on some it will have a format according to the session
   * NLS_DATE_FORMAT. TIMESTAMP% columns seem to work correctly. On some
   * versions Oracle will throw an error "ORA-01801: date format is too long for
   * internal buffer" if the session NLS format mask in effect when opening the
   * REF CURSOR was shorter that what is required for XML Schema-compliance.
   * This also depends on how many rows are returned by the REF CURSOR.
   * Sometimes it doesn't work correctly for the first row, but it works
   * correctly for subsequent rows.
   * In general, you should avoid using this setting as it's very unstable.
   * Setting your session's NLS_DATE_FORMAT to at least 23 characters makes it
   * slightly more stable.
   */
  date_time_xml_set_nls_restart  constant pls_integer := 3;

  /**
   * Generate XML Schema-compliant date/time values. This is done by internally
   * by fetching the incoming dynamic SQL or REF CURSOR to memory, changing the
   * NLS settings, generating the XML on the memory copy. This has no side
   * effects on implit/explicit calls to TO_CHAR without a specific format mask
   * but the performance is affected.
   * Used with the FROM_SELECT function this is only supported on Oracle 11.1
   * and newer.
   */
  date_time_xml_set_nls_copy_mem constant pls_integer := 4;

  /**
   * Generate XML Schema-compliant date/time values. This is done by internally
   * by fetching the incoming dynamic SQL or REF CURSOR to a global temporary
   * table, changing the NLS settings, generating the XML on the table copy.
   * This has no side effects on implit/explicit calls to TO_CHAR without a
   * specific format mask but the performance is affected.
   * Used with the FROM_SELECT function on Oracle versions older than 11 (or if
   * you're using quoted bind variable names -- necessary because DBMS_SQL does
   * not support quoted bind variable names) this attempts to locate the bind
   * variables in SELECT_STATEMENT and this is not done in a fool proof manner
   * if you're using quoted bind variable names using non-identifier characters.
   */
  date_time_xml_set_nls_copy_tab constant pls_integer := 5;

  function from_select(
    select_statement in varchar2,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  function from_select(
    select_statement in varchar2,
    bind_variables in bind_variable_c,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  function from_ref_cursor(
    ref_cursor in out sys_refcursor,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  function from_anydata(
    ad in anydata,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  function from_anydataset(
    ads in anydataset,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  function from_anytype(
    "at" in anytype,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

  procedure set_xml_nls_formats;
  procedure set_session_nls_formats;
end xml_gen;
/

create or replace package body xml_gen as
  --- @version  $Revision: 3 $

  /**
   * XML Schema compliant datetime format, no fraction, no time zone
   * (Oracle DATE).
   */
  xml_nls_date_format constant varchar2(23) :=
      'yyyy-mm-dd"T"hh24:mi:ss';
  /**
   * XML Schema compliant datetime format, with fraction, no time zone (Oracle
   * TIMESTAMP).
   */
  xml_nls_timestamp_format constant varchar2(27) :=
      'yyyy-mm-dd"T"hh24:mi:ss.ff9';
  /**
   * XML Schema compliant datetime format, with fraction, with time zone (Oracle
   * TIMESTAMP WITH [LOCAL] TIME ZONE).
   */
  xml_nls_timestamp_tz_format constant varchar2(34) :=
      'yyyy-mm-dd"T"hh24:mi:ss.ff9tzh:tzm';

  /**
   * Collection of V$NLS_PARAMETERS.VALUE, indexed by
   * V$NLS_PARAMETERS.PARAMETER.
   */
  type nls_parameters_c is table of
  v$nls_parameters.value%type
  index by v$nls_parameters.parameter%type;

  --- Session NLS formats.
  session_nls_parameters nls_parameters_c;
  --- NLS Formats required for XML compliance.
  xml_nls_parameters nls_parameters_c;

  -- Empty bind variable collection.
  bind_variables_empty bind_variable_c;

  function get_xml(
    context in dbms_xmlgen.ctxtype,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype;

/**
 * Gets the result set of a given SELECT statement as XML in Oracle's
 * "canonical form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   select_statement
 *                  SELECT statement.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  Result set as XML.
 */

  function from_select(
    select_statement in varchar2,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

  begin
    return from_select(
      select_statement, bind_variables_empty, date_time_option
    );
  end from_select;

/**
 * Gets the result set of a given SELECT statement as XML in Oracle's
 * "canonical form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   select_statement
 *                  SELECT statement with bind variables.
 * @param   bind_variables
 *                  Collection of bind variables to be bound.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  Result set as XML.
 */

  function from_select(
    select_statement in varchar2,
    bind_variables in bind_variable_c,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

    context dbms_xmlgen.ctxtype;
    dbms_sql_cursor binary_integer;
    n pls_integer;
    ref_cursor sys_refcursor;
    quoted_bind_variable_names boolean := false;

  begin
    if substr(
         upper(ltrim(select_statement, chr(13) || chr(10) || ' ')),
         1,
         6
       ) != 'SELECT' then
      -- Invalid statement given, not a SELECT. Attempting to prevent injection.
      raise_application_error(
        -20000,
        'xml_gen.from_select(select_statement => ''' ||
          select_statement || '''): ' ||
          'Only SELECT statements are allowed'
      );
    end if;
    if date_time_option in (
          date_time_xml_set_nls_copy_mem,
          date_time_xml_set_nls_copy_tab
        ) then
      if bind_variables is null or bind_variables.count = 0 then
        open ref_cursor for
        select_statement;
      else
        for i in
              nvl(bind_variables.first, 0) ..
              nvl(bind_variables.last, -1)
            loop
          if substr(bind_variables(i).name, 1, 1) = '"' then
            quoted_bind_variable_names := true;
          end if;
        end loop;

        if portable.get_major_version < 11 or quoted_bind_variable_names then
          -- Older than Oracle 11.1. Dynamically build REF CURSOR OPEN
          -- statement. This is not foolproof if quoted bind variable names are
          -- used and we might not find them correctly.
          declare
            select_statement_upper varchar2(32767) := upper(select_statement);
            first_bind_positions dbms_utility.number_array;
            sorted_bind_variables bind_variable_c := bind_variable_c();
            j pls_integer;
            open_statement varchar2(32767);
            next_char varchar2(1);
            quoted_bind_variable_name boolean;
          begin
            for i in
                  nvl(bind_variables.first, 0) ..
                  nvl(bind_variables.last, -1)
                loop
              -- Locate the bind variables in the SELECT statement.
              n := 1;
              quoted_bind_variable_name :=
                substr(bind_variables(i).name, 1, 1) = '"';
              loop
                if quoted_bind_variable_name then
                  -- Case sensitive search, bind variable name quoted..
                  j := instr(
                    select_statement,
                    ':' || bind_variables(i).name,
                    1,
                    n
                  );
                else
                  -- Case insensitive search, bind variable name not quoted.
                  j := instr(
                    select_statement_upper,
                    ':' || upper(bind_variables(i).name),
                    1,
                    n
                  );
                end if;
                -- Ensure that this is the whole word and not part of another
                -- bind variable. Check that next character cannot be part of an
                -- identifier.
                next_char := substr(
                  select_statement_upper,
                  j + length(bind_variables(i).name) + 1,
                  1
                );
                exit when next_char is null or
                    ascii(next_char) <= 32 or
                    instr('+-/*(),|''', next_char) > 0;
                n := n + 1;
              end loop;
              if j = 0 then
                -- Not found, raise error.
                raise_application_error(
                  -20001,
                  'xml_gen.from_select: Bind variable ' ||
                      bind_variables(i).name || ' not found in select_statement'
                );
              end if;
              first_bind_positions(j) := i;
            end loop;

            j := first_bind_positions.first;
            while j is not null loop
              -- Sorted list of bind positions, sort the bind variables
              -- accordingly.
              sorted_bind_variables.extend(1);
              sorted_bind_variables(sorted_bind_variables.count) :=
                  bind_variables(first_bind_positions(j));
              j := first_bind_positions.next(j);
            end loop;

            open_statement :=
                'begin' || chr(10) ||
                '  open :rc for ' || chr(10) ||
                '''' || replace(select_statement, '''', '''''') || '''' || chr(10) ||
                '  using ';

            for i in
                  nvl(sorted_bind_variables.first, 0) ..
                  nvl(sorted_bind_variables.last, -1)
                loop
              open_statement := open_statement ||
                  '''' || replace(sorted_bind_variables(i).value, '''', '''''') ||
                      ''', ';
            end loop;
            open_statement :=
                rtrim(open_statement, ', ') || ';' || chr(10) ||
                'end;';

            execute immediate open_statement
            using in out ref_cursor;
          end;
        else
          -- Oracle 11.1 or newer -- open with DBMS_SQL, then convert to REF
          -- CURSOR. This does not support quoted bind variable names as
          -- DBMS_SQL will raise a "ORA-01008: not all variables bound" even
          -- through it doesn't raise a "ORA-01006: bind variable does not
          -- exist".
          dbms_sql_cursor := dbms_sql.open_cursor;
          dbms_sql.parse(dbms_sql_cursor, select_statement, dbms_sql.native);

          for i in
                nvl(bind_variables.first, 0) ..
                nvl(bind_variables.last, -1)
              loop
            -- Bind each bind variable.
            dbms_sql.bind_variable(
              dbms_sql_cursor,
              bind_variables(i).name,
              bind_variables(i).value
            );
          end loop;

          begin
            n := dbms_sql.execute(dbms_sql_cursor);
          exception
            when others then
              dbms_output.put_line('xml_gen.from_select');
              dbms_output_put_line(
                'select_statement = ''' || select_statement || ''''
              );
              if sqlcode = -1008 then
                -- ORA-01008: not all variables bound.
                for i in
                      nvl(bind_variables.first, 0) ..
                      nvl(bind_variables.last, -1)
                    loop
                  -- Bind each bind variable.
                  dbms_output_put_line(
                    bind_variables(i).name || ' => ' ||
                        '''' || bind_variables(i).value || ''''
                  );
                  if substr(bind_variables(i).name, 1, 1) = '"' then
                    dbms_output.put_line(
                      'ERROR: DBMS_SQL doesn''t support quoted bind ' ||
                          'variable names (EXECUTE IMMEDIATE does)'
                    );
                  end if;
                end loop;
              end if;
              dbms_output.put_line(sqlerrm);
              dbms_output.put_line(dbms_utility.format_error_backtrace);
              raise;
          end;

          execute immediate
          'begin :ref_cursor := dbms_sql.to_refcursor(:dbms_sql_cursor); end;'
          using out ref_cursor, in out dbms_sql_cursor;
        end if;
      end if;

      return from_ref_cursor(ref_cursor, date_time_option);
    else
      context := dbms_xmlgen.newcontext(select_statement);

      if bind_variables is not null then
        for i in
              nvl(bind_variables.first, 0) ..
              nvl(bind_variables.last, -1)
            loop
          -- Bind each bind variable.
          dbms_xmlgen.setbindvalue(
            context,
            bind_variables(i).name,
            bind_variables(i).value
          );
        end loop;
      end if;
    end if;

    return get_xml(context, date_time_option);
  end from_select;

/**
 * Gets the result set of a given REF CURSOR as XML in Oracle's "canonical
 * form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   select_statement
 *                  SELECT statement.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  Result set as XML.
 */

  function from_ref_cursor(
    ref_cursor in out sys_refcursor,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype is

    context dbms_xmlgen.ctxtype;
    result xmltype;
    rc2 sys_refcursor;
    destination char(1);
    type_name user_types.type_name%type;

  begin
    if date_time_option in (
          date_time_xml_set_nls_copy_mem,
          date_time_xml_set_nls_copy_tab
        ) then
      -- Copy to memory/table first, then change NLS, then generate off the
      -- copy.
      if date_time_option = date_time_xml_set_nls_copy_mem then
        destination := 'M';
      else
        destination := 'T';
      end if;

      rc2 := ref_cursor_copy.to_ref_cursor(
        ref_cursor,
        type_name,
        destination,
        plsql_block_before_copy_open => 'xml_gen.set_xml_nls_formats;'
      );

      ref_cursor := rc2;
    end if;

    context := dbms_xmlgen.newcontext(ref_cursor);
    result := get_xml(context, date_time_option);
    begin
      if ref_cursor%isopen then
        close ref_cursor;
      end if;
    exception
      when others then
        null;
    end;
    return result;
  end from_ref_cursor;

/**
 * Gets the result set of a given ANYDATA instance as XML in Oracle's "canonical
 * form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * This is the function to use when you need to obtain XML for object type/
 * collection instances and ANYDATA in general. You use ANYDATA.CONVERTOBJECT
 * and ANYDATA.CONVERTCOLLECTION respectively to convert from object types and
 * collections for this. XMLTYPE can also convert these directly but has several
 * flaws in terms of incurrent DATE/TIMESTAMP formats, missing elements for NULL
 * values, etc.
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   ad      ANYDATA instance.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  ANYDATA instance as XML.
 */

  function from_anydata(
    ad in anydata,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

    ref_cursor sys_refcursor;

  begin
    open ref_cursor for
    select ad
    from   dual;

    -- Get rid of surrounding XML due to the SELECT statement.
    return from_ref_cursor(ref_cursor, date_time_option).extract('/*/*/*/*');
  end from_anydata;

/**
 * Gets the result set of a given ANYDATASET instance as XML in Oracle's
 * "canonical form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   ads     ANYDATASET instance.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  ANYDATASET instance as XML.
 */

  function from_anydataset(
    ads in anydataset,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

    ref_cursor sys_refcursor;

  begin
    open ref_cursor for
    select ads
    from   dual;

    -- Get rid of surrounding XML due to the SELECT statement.
    return from_ref_cursor(ref_cursor, date_time_option).extract('/*/*/*/*');
  end from_anydataset;

/**
 * Gets the result set of a given ANYTYPE instance as XML in Oracle's "canonical
 * form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   "at"    ANYTYPE instance.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  ANYTYPE instance as XML.
 */

  function from_anytype(
    "at" in anytype,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

    ref_cursor sys_refcursor;

  begin
    open ref_cursor for
    select "at"
    from   dual;

    -- Get rid of surrounding XML due to the SELECT statement.
    return from_ref_cursor(ref_cursor, date_time_option).extract('/*/*/*/*');
  end from_anytype;

/**
 * Gets the result set of a given DBMS_XMLGEN context as XML in Oracle's
 * "canonical form", ie:
 * <ROWSET>
 *   <ROW>
 *     <COLUMN_NAME_1>column_value_1</COLUMN_NAME_1>
 *     <COLUMN_NAME_2>column_value_2</COLUMN_NAME_2>
 *     ...
 *     <COLUMN_NAME_N>column_value_n</COLUMN_NAME_N>
 *   </ROW>
 * </ROWSET>
 * Text columns will have their values converted to XML character entities, also
 * if it's XML, HTML etc. This is not done with XMLTYPE column values as we
 * assume it's valid XML.
 * @param   context DBMS_XMLGEN context created based on either SELECT statement
 *                  or REF CURSOR.
 * @param   date_time_option
 *                  Date/time options. One of the DATE_TIME_% constants from the
 *                  package specification.
 * @return  Result set as XML.
 */

  function get_xml(
    context in dbms_xmlgen.ctxtype,
    date_time_option in pls_integer := date_time_xml_set_nls
  )
  return xmltype as

    result xmltype;

  begin
    if date_time_option in (
          date_time_xml_set_nls,
          date_time_xml_set_nls_restart
        ) then
      set_xml_nls_formats;
    end if;
    if date_time_option = date_time_xml_set_nls_restart then
      dbms_xmlgen.restartquery(context);
    end if;
    dbms_xmlgen.setnullhandling(context, dbms_xmlgen.empty_tag);
    dbms_xmlgen.setconvertspecialchars(context, true);
    result := dbms_xmlgen.getxmltype(context);
    dbms_xmlgen.closecontext(context);
    if date_time_option in (
          date_time_xml_set_nls,
          date_time_xml_set_nls_restart,
          date_time_xml_set_nls_copy_mem,
          date_time_xml_set_nls_copy_tab
        ) then
      set_session_nls_formats;
    end if;

    return result;
  exception
    when others then
      if date_time_option in (
            date_time_xml_set_nls,
            date_time_xml_set_nls_restart,
            date_time_xml_set_nls_copy_mem,
            date_time_xml_set_nls_copy_tab
          ) then
        set_session_nls_formats;
      end if;
      raise;
  end get_xml;

/**
 * Sets specific NLS date/timestamp formats required for XML compliance.
 */

  procedure set_xml_nls_formats is
  pragma autonomous_transaction;

    needs_commit boolean := false;

  begin
    for nls in (
          select /*+ cursor_sharing_exact */
                 parameter,
                 value
          from   v$nls_parameters
          where  parameter like 'NLS_%FORMAT' and
                 parameter not like 'NLS\_TIME\_%' escape '\'
        ) loop
      session_nls_parameters(nls.parameter) := nls.value;
      if nls.value != xml_nls_parameters(nls.parameter) then
        -- Change NLS format for XML-compliance.
        dbms_session.set_nls(
          nls.parameter,
          '''' || xml_nls_parameters(nls.parameter) || ''''
        );

        needs_commit := true;
      end if;
    end loop;

    if needs_commit then
      commit;
    end if;
  end set_xml_nls_formats;

/**
 * Restores NLS date/timestamp formats as it was before setting up for XML
 * compliance.
 */

  procedure set_session_nls_formats is
  pragma autonomous_transaction;

    needs_commit boolean := false;
    nls_parameter v$nls_parameters.parameter%type;

  begin
    nls_parameter := xml_nls_parameters.first;
    while nls_parameter is not null loop
      if xml_nls_parameters(nls_parameter) !=
          session_nls_parameters(nls_parameter) then
        -- Restore original NLS format.
        dbms_session.set_nls(
          nls_parameter,
          '''' || session_nls_parameters(nls_parameter) || ''''
        );

        needs_commit := true;
      end if;

      nls_parameter := xml_nls_parameters.next(nls_parameter);
    end loop;

    if needs_commit then
      commit;
    end if;
  exception
    when no_data_found then
      -- Ignore that we don't have a backup. Presumably because
      -- SET_XML_NLS_FORMATS hasn't been called.
      null;
  end set_session_nls_formats;

begin
  xml_nls_parameters('NLS_DATE_FORMAT') := xml_nls_date_format;
  xml_nls_parameters('NLS_TIMESTAMP_FORMAT') := xml_nls_timestamp_format;
  xml_nls_parameters('NLS_TIMESTAMP_TZ_FORMAT') := xml_nls_timestamp_tz_format;
end xml_gen;
/

Example Usage

Here are a few examples of usage:

set long 100000
set pagesize 50000
set linesize 1000

alter session set nls_date_format = 'dd.mm.yyyy hh24:mi:ss';
alter session set nls_timestamp_format = 'dd.mm.yyyy hh24:mi:ss.ff';
alter session set nls_timestamp_tz_format = 'dd.mm.yyyy hh24:mi:ss.ff tzh:tzm';

variable xml clob

prompt INSERT

begin
  :xml := '';
  :xml := xml_gen.from_select(
    'insert into dept values (50, ''abc'', ''def'')'
  ).getclobval();
end;
/

print :xml

prompt FROM_SELECT

begin
  :xml := '';
  :xml := xml_gen.from_select('select * from dept').getclobval();
end;
/

print :xml

prompt FROM_SELECT with bind variables

begin
  :xml := xml_gen.from_select(
    'select * from emp where empno in (:empno1, :empno2)',
    bind_variable_c(
      bind_variable_t('empno1', '7499'),
      bind_variable_t('empno2', '7934')
    )
  ).getclobval();
end;
/

print :xml

prompt FROM_REF_CURSOR with bind variables

declare
  rc sys_refcursor;
begin
  open rc for
  'select * from emp where empno in (:empno1, :empno2)'
  using 7369, 7499;

  :xml := xml_gen.from_ref_cursor(rc).getclobval();
end;
/

print :xml

-- grant execute on sys.aq$_subscribers to scott;
-- grant execute on sys.aq$_agent to scott;

prompt FROM_ANYDATA object type instance

begin
  :xml := '';
  :xml := xml_gen.from_anydata(
    anydata.convertobject(sys.aq$_agent('name', 'address', 1))
  ).getclobval();
end;
/

print :xml

prompt FROM_ANYDATA collection

begin
  :xml := '';
  :xml := xml_gen.from_anydata(
    anydata.convertcollection(
      sys.aq$_subscribers(
        sys.aq$_agent('name1', 'address1', 1),
        sys.aq$_agent('name2', 'address2', 2)
      )
    )
  ).getclobval();
end;
/

print :xml

prompt FROM_ANYTYPE throws ORA-19201

declare
  a anytype;
begin
  anytype.begincreate(
    typecode => dbms_types.typecode_object,
    atype => a
  );
  a.addattr(
    typecode => dbms_types.typecode_number,
    aname => 'n',
    prec => 38,
    scale => 0,
    len => null,
    csid => null,
    csfrm => null
  );
  a.addattr(
    typecode => dbms_types.typecode_date,
    aname => 'd',
    prec => 5,
    scale => 5,
    len => null,
    csid => null,
    csfrm => null
  );
  a.endcreate;

  begin
    :xml := '';
    :xml := xml_gen.from_anytype(a).getclobval();
  exception
    when others then
      dbms_output.put_line(sqlerrm);
  end;

  begin
    select sys_xmlgen(a).getclobval()
    into   :xml
    from   dual;
  exception
    when others then
      dbms_output.put_line(sqlerrm);
  end;
end;
/

print :xml

prompt FROM_ANYDATASET throws ORA-19201

declare
  a anytype;
  ads anydataset;
begin
  anytype.begincreate(
    typecode => dbms_types.typecode_object,
    atype => a
  );
  a.addattr(
    typecode => dbms_types.typecode_number,
    aname => 'n',
    prec => 38,
    scale => 0,
    len => null,
    csid => null,
    csfrm => null
  );
  a.addattr(
    typecode => dbms_types.typecode_varchar2,
    aname => 'vc2',
    prec => null,
    scale => null,
    len => 10,
    csid => null,
    csfrm => null
  );
  a.endcreate;

  anydataset.begincreate( dbms_types.typecode_object, a, ads);
  for i in 1 .. 2 loop
    ads.addinstance;
    ads.piecewise;
    ads.setnumber(i);
    ads.setvarchar2('vc2 ' || i);
  end loop;
  ads.endcreate;

  begin
    :xml := '';
    :xml := xml_gen.from_anydataset(ads).getclobval();
  exception
    when others then
      dbms_output.put_line(sqlerrm);
  end;

  begin
    select sys_xmlgen(ads).getclobval()
    into   :xml
    from   dual;
  exception
    when others then
      dbms_output.put_line(sqlerrm);
  end;
end;
/

print :xml

Source Code

You can download the source code here.
Feel free to use the code at your own risk. I welcome your feedback and suggestions for improvements but the code as such is not supported.

Conclusion

Did I achieve my goals? Not entirely, but close. I'll describe this in detail in the following sections.

Object Type Instance

Object type instance: Unfortunately, Oracle doesn't allow us to write custom PL/SQL code that can receive or return every possible object type instance. Oracle reserves the datatypes "<ADT_1>" (ADT is an acronym for Abstract Data Type, another term used for object types) and "<COLLECTION_1>" for SYS objects so we're not allowed to use them. This is very unfortunate, as in this case we could just pass them on to other Oracle functionality that would know what to do with them.

We have a few workarounds:

• Convert the object type instance and collection to an ANYDATA instance by calling the ANYDATA.CONVERTOBJECT and ANYDATA.CONVERTCOLLECTION "constructors" respectively and then calling XML_GEN.FROM_ANYDATA.
• Add an overloaded FROM_OBJECT function in XML_GEN for each object type you need to support. This should simply convert the object type instance to an ANYDATA instance (by calling the ANYDATA.CONVERTOBJECT "constructor") and invoke XML_GEN.FROM_ANYDATA. If your object types inherit from a few base object types this would be easily done by just adding overloads for the base object types.
• Add a member function to each of the object types called TO_XML, which basically does what the overloaded FROM_OBJECT function mentioned above would do.

ANYTYPE and ANYDATASET

ANYTYPE and ANYDATASET are not supported by DBMS_XMLGEN and any of the other XML generating functionality. Oracle throws the following error if you invoke with any of those datatypes:

ORA-19201: Datatype not supported

I considered implementing a workaround that would generate the XML by recursively inspecting the structure (and values for ANYDATASET) but I'll leave this for a future version of my XML_GEN package.

Related Blog Posts

A few other related and interesting blog posts are (please notify me if you know of other interesting, relevant blog posts on this subject and I'll add them to this list):

• Kevin Meade's blog: Easy XML - Let the Database do the Work: http://www.orafaq.com/node/1025.
• Adrian Billington's blog: working with long columns: http://www.oracle-developer.net/display.php?id=430.

References

• Oracle® XML DB Developer's Guide 11g Release 2 (11.2) Chapter 18 Generating XML Data from the Database: http://download.oracle.com/docs/cd/E11882_01/appdev.112/e16659/xdb13gen.htm#g1047191.
• Oracle XSQL: Combining SQL, Oracle Text, XSLT...: Michael D. Thomas. Wiley 2003. ISBN 0471271209: http://wiley.com/compbooks/thomas/.
• Oracle® XML Developer's Kit Programmer's Guide 11g Release 2 (11.2) Chapter 15 Using the XSQL Pages Publishing Framework: Advanced Topics: http://download.oracle.com/docs/cd/E11882_01/appdev.112/e10708/adx_j_xsqladv.htm#BABCIGBB.
• Oracle® XML Developer's Kit Programmer's Guide 11g Release 2 (11.2) Chapter 30 XSQL Pages Reference: http://download.oracle.com/docs/cd/E11882_01/appdev.112/e10708/adx_ref_xsql.htm#BJEEABJJ.

The arcane LONG datatype has as you’re aware of many intrinsic limitations but unfortunately Oracle still uses quite a few LONG columns in the data dictionary. It’s beyond me why these haven’t been converted to CLOB a very long time ago (when Oracle deprecated usage of LONG). At the same time, Oracle only provides the following means of converting these LONG columns to eg CLOB:

1. TO_LOB: A function that converts a LONG column to a CLOB. Can only be used in a SELECT list of a subquery in an INSERT statement. This means that the function is only useful if you’re converting the underlying table definition.
2. ALTER TABLE <table_name> MODIFY <long_column_name> CLOB: This converts the column datatype and the data as well.
3. Use DBMS_REDEFINITION to redefine the column datatype.
4. Use Oracle Data Pump to convert the column datatype.
5. Use CAST function to cast from LONG to CLOB: Unfortunately, CAST doesn’t support LONG.
6. Write a PL/SQL function that performs a SELECT ... INTO l FROM user_views, where l is a PL/SQL variable of type LONG. However, in PL/SQL, a LONG variable can only hold up to 32,760 characters (yes, not 32,767, ie different from VARCHAR2) so this will only solve the problem for small to medium sized views.
7. Use dynamic SQL with DBMS_SQL.COLUMN_VALUE_LONG to access the LONG piecewise.

Problem

Imagine the following scenario:

• You need to access the source of a given view.
• USER_VIEWS.TEXT contains the view source but it is a column with datatype LONG so very limited in use.
• Using DBMS_METADATA to obtain the source is not an option for various reasons, eg performance, practicality, etc.

Solution

Seen that we need to access the data from the view, we cannot use TO_LOB or any of the other solutions that alter the underlying table column definition. However, the DBMS_SQL.COLUMN_VALUE_LONG function comes to the rescue as this allows us to fetch the LONG data piecewise and construct a CLOB with the same data. This obviously means that we have to use dynamic SQL for the query of the table/view we’re trying to convert the LONG column for.

We have two different methods of applying dynamic SQL to this problem:

1. Dynamic SQL for just the LONG column. We need to write a function that receives the primary key value(s) of the underlying table/view (could be ROWID if a table) as input parameter(s), builds a SELECT statement for the underlying table/view for the LONG column using bind variable(s) for the primary key value(s), does the DBMS_SQL magic and uses the DBMS_SQL.COLUMN_VALUE_LONG function. For scalability, the solution should keep a collection of parsed statements and re-use those without re-parsing.
2. Dynamic SQL for the whole underlying table/view generated and called in a pipelined table function that utilizes object types.

For this blog post I’ll use solution 2, for which I’ll demonstrate the following:

1. Create a standalone function with “LONG-to-CLOB” functionality.
2. Create an object type with attributes corresponding to USER_VIEWS. This object type will have member methods used with DBMS_SQL.
3. Create an object type collection based on the object type from 2.
4. Create a standalone function that takes an optional argument for a WHERE clause used against USER_VIEWS, using a combination of DBMS_SQL and 1., 2., and 3. above. This method is a pipelined table function.
5. Optionally create a view on top of the standalone function from 4.

You can apply the structure of this solution in use cases where you need to access a LONG value in a table/view as a CLOB, without converting the underlying persistent column: The standalone function from 1. above is generic — items 2. through 5. are specific to the underlying table/view.

We’ll go through the different parts in the following sections.

“LONG-to-CLOB” Function

This is the function that converts a LONG column to a CLOB value through a DBMS_SQL cursor that has been parsed, prepared (given column “defined” with DBMS_SQL.DEFINE_COLUMN_LONG) and executed:

create or replace function long_to_clob(
  dbms_sql_cursor in integer,
  col_id in integer
)
return clob as

/**
 * Fetches LONG column value and converts it to a CLOB.
 * @param   dbms_sql_cursor
 *                  DBMS_SQL cursor parsed, prepared (given column "defined"
 *                  with DBMS_SQL.DEFINE_COLUMN_LONG) and executed.
 * @param   col_id  Column ID.
 * @return  LONG column value as a CLOB.
 */

  long_val long;
  long_len integer;
  buf_len  integer := 32760;
  cur_pos  number := 0;

  result   clob;

begin
  -- Create CLOB.
  dbms_lob.createtemporary(result, false, dbms_lob.call);

  -- Piecewise fetching of the LONG column, appending to the CLOB.
  loop
    dbms_sql.column_value_long(
      dbms_sql_cursor,
      col_id,
      buf_len,
      cur_pos,
      long_val,
      long_len
    );
    exit when long_len = 0;
    dbms_lob.append(result, long_val);
    cur_pos := cur_pos + long_len;
  end loop;

  return result;
end long_to_clob;
/

Object Type

In Oracle 11.2.0.1.0, USER_VIEWS has the following columns:

SQL> desc user_views
 Name             Null?    Type
 ---------------- -------- --------------
 VIEW_NAME        NOT NULL VARCHAR2(30)
 TEXT_LENGTH               NUMBER
 TEXT                      LONG
 TYPE_TEXT_LENGTH          NUMBER
 TYPE_TEXT                 VARCHAR2(4000)
 OID_TEXT_LENGTH           NUMBER
 OID_TEXT                  VARCHAR2(4000)
 VIEW_TYPE_OWNER           VARCHAR2(30)
 VIEW_TYPE                 VARCHAR2(30)
 SUPERVIEW_NAME            VARCHAR2(30)
 EDITIONING_VIEW           VARCHAR2(1)
 READ_ONLY                 VARCHAR2(1)

which means that we could create our object type like the following (notice how TEXT is represented by a CLOB instead of the original LONG):

create or replace type user_views_t as object (
/**
 * Object type representing columns in data dictionary view USER_VIEWS, with the
 * TEXT column represented by a CLOB instead of a LONG.
 */

  view_name        varchar2(30),
  text_length      number,
  -- CLOB instead of LONG.
  text             clob,
  type_text_length number,
  type_text        varchar2(4000),
  oid_text_length  number,
  oid_text         varchar2(4000),
  view_type_owner  varchar2(30),
  view_type        varchar2(30),
  superview_name   varchar2(30),
  editioning_view  varchar2(1),
  read_only        varchar2(1),

  constructor function user_views_t
  return self as result,
  constructor function user_views_t(dbms_sql_cursor in integer)
  return self as result,

  member procedure define_columns(dbms_sql_cursor in integer)
);
/

The object type has two constructors and one member function. The member function is used to define the columns for DBMS_SQL and in order to be able to use the attributes, this has to be a member function (working on an object type instance as opposed to a static function) and the first constructor is used to create such a dummy instance, with all the attributes set to NULL. The second constructor sets all attributes to corresponding column values in a given fetched DBMS_SQL cursor.

This is the implementation of the object type body:

create or replace type body user_views_t as
/**
 * Constructor. Sets all attributes to NULL.
 * @return  New object type instance.
 */

  constructor function user_views_t
  return self as result as

  begin
    return;
  end user_views_t;

/**
 * Constructor. Sets all attributes to corresponding column values in fetched
 * DBMS_SQL cursor.
 * @param   dbms_sql_cursor
 *                  Executed and fetched DBMS_SQL cursor on a query from
 *                  USER_VIEWS.
 * @return  New object type instance.
 */

  constructor function user_views_t(dbms_sql_cursor in integer)
  return self as result as

  begin
    dbms_sql.column_value(dbms_sql_cursor, 01, view_name);
    dbms_sql.column_value(dbms_sql_cursor, 02, text_length);
    -- Convert LONG to CLOB.
    text := long_to_clob(dbms_sql_cursor,  03);
    dbms_sql.column_value(dbms_sql_cursor, 04, type_text_length);
    dbms_sql.column_value(dbms_sql_cursor, 05, type_text);

    dbms_sql.column_value(dbms_sql_cursor, 06, oid_text_length);
    dbms_sql.column_value(dbms_sql_cursor, 07, oid_text);
    dbms_sql.column_value(dbms_sql_cursor, 08, view_type_owner);
    dbms_sql.column_value(dbms_sql_cursor, 09, view_type);
    dbms_sql.column_value(dbms_sql_cursor, 10, superview_name);

    dbms_sql.column_value(dbms_sql_cursor, 11, editioning_view);
    dbms_sql.column_value(dbms_sql_cursor, 12, read_only);

    return;
  end user_views_t;

/**
 * Defines all columns in DBMS_SQL cursor.
 * @param   Parsed DBMS_SQL cursor on a query from USER_VIEWS.
 */

  member procedure define_columns(dbms_sql_cursor in integer) as

  begin
    dbms_sql.define_column(dbms_sql_cursor, 01, view_name, 30);
    dbms_sql.define_column(dbms_sql_cursor, 02, text_length);
    -- LONG column.
    dbms_sql.define_column_long(dbms_sql_cursor, 03);
    dbms_sql.define_column(dbms_sql_cursor, 04, type_text_length);
    dbms_sql.define_column(dbms_sql_cursor, 05, type_text, 4000);

    dbms_sql.define_column(dbms_sql_cursor, 06, oid_text_length);
    dbms_sql.define_column(dbms_sql_cursor, 07, oid_text, 4000);
    dbms_sql.define_column(dbms_sql_cursor, 08, view_type_owner, 30);
    dbms_sql.define_column(dbms_sql_cursor, 09, view_type, 30);
    dbms_sql.define_column(dbms_sql_cursor, 10, superview_name, 30);

    dbms_sql.define_column(dbms_sql_cursor, 11, editioning_view, 1);
    dbms_sql.define_column(dbms_sql_cursor, 12, read_only, 1);
  end define_columns;
end;
/

Notice how the second constructor uses our LONG_TO_CLOB function.

The process of matching the original table/view columns into attributes and handling of these in one of the constructors and the DEFINE_COLUMNS method is tedious and error prone. If you need to do this often you should consider writing a generator that generates the object type specification and body based on the definition of a given table/view.

Object Type Collection

This is the implementation of the object type collection, using a nested table:

create or replace type user_views_c as
table of user_views_t;
/

We need this object type collection for the pipelined table function as this pipes back a collection of object type instances back to the SQL engine — namely, one object type instance for each row in USER_VIEWS this function finds.

Pipelined Table Function

This is the implementation of the standalone pipelined table function that takes an optional parameter to be matched against the VIEW_NAME column in a LIKE expression and uses DBMS_SQL to parse the query from USER_VIEWS, uses the USER_VIEWS_T and USER_VIEWS_C object types (which in turn calls LONG_TO_CLOB) and pipes the rows back to the SQL engine (type PTF suffix refers to “Pipelined Table Function”):

create or replace function user_views_ptf(
  view_name_like in varchar2 := '%'
)
return user_views_c pipelined as

/**
 * Gets collection of user views representing rows in USER_VIEWS. Resolved
 * through a dynamic SQL query against USER_VIEWS and the TEXT column is
 * converted from a LONG to a CLOB.
 * @param   view_name_like
 *                  LIKE expression used in a WHERE clause predicate against
 *                  USER_VIEWS.VIEW_NAME. Default '%', ie all.
 * @return  Collection that can be used in a FROM clause with a TABLE() cast.
 */

  query           varchar2(200);
  dbms_sql_cursor binary_integer;
  n               pls_integer;
  each            user_views_t;

begin
  query :=
     'select * ' ||
     'from   user_views uv ' ||
     'where  uv.view_name like :view_name_like';

  -- Create cursor, parse and bind.
  dbms_sql_cursor := dbms_sql.open_cursor;
  dbms_sql.parse(dbms_sql_cursor, query, dbms_sql.native);
  dbms_sql.bind_variable(dbms_sql_cursor, 'view_name_like', view_name_like);

  -- Define columns through dummy object type instance.
  each := user_views_t();
  each.define_columns(dbms_sql_cursor);

  -- Execute.
  n := dbms_sql.execute(dbms_sql_cursor);

  -- Fetch all rows, pipe each back.
  while dbms_sql.fetch_rows(dbms_sql_cursor) > 0 loop
    each := user_views_t(dbms_sql_cursor);

    pipe row(each);
  end loop;

  dbms_sql.close_cursor(dbms_sql_cursor);
exception
  when others then
    dbms_output.put_line('long_to_clob: ' || sqlerrm);
    dbms_output.put_line(dbms_utility.format_error_backtrace);
    if dbms_sql.is_open(dbms_sql_cursor) then
      dbms_sql.close_cursor(dbms_sql_cursor);
    end if;

    raise;
end user_views_ptf;
/

It’s inconvenient that we can’t create this function as a member function on USER_VIEWS_T where it really belongs. However this is not possible because that would introduce a cyclical dependency between USER_VIEWS_T and USER_VIEWS_C, which is not allowed (even using forward object type declarations).

SELECT

You can select from the pipelined table function with a TABLE expression, such as:

select *
from   table(user_views_ptf('V%'));

Installation

You need to install the solution objects in the following order:

1. LONG_TO_CLOB function.
2. USER_VIEWS_T object type specification.
3. USER_VIEWS_T object type body.
4. USER_VIEWS_C object type collection.
5. USER_VIEWS_PTF function.

Test

A small test case:

create or replace view v1 as
select 1 n,
       'abc' vc2,
       sysdate d,
       systimestamp(6) t
from   dual;

SQL> select view_name,
  2         text
  3  from   table(user_views_ptf('V%'));

VIEW_NAME                      TEXT
------------------------------ ------------------------
V1                             select 1 n,
                                      'abc' vc2,
                                      sysdate d,
                                      systimestamp(6) t
                               from   dual

View

You can optionally create a view on top of the pipelined table function. In this case, you cannot push the LIKE expression into the argument to the table function so the optimizer has no alternative to perform a full table scan on USER_VIEWS and then a match on the returned rows on whatever predicate the view is used with.

Maintenance

Every time you need to convert a LONG column in a table/view to a CLOB you need to do the following:

1. Create the object type with attributes corresponding to the columns of the table/view you need to obtain data for. The object type needs two constructors and the member function DEFINE_COLUMNS as for the USER_VIEWS_T object type.
2. Create the object type collection for 1.
3. Create the standalone pipelined table function.
4. Optionally create a view on top of 3.

If you prefer to use packages over standalone functions, you could bundle them up in a package called LONG_TO_CLOB and rename the function LONG_TO_CLOB to TO_CLOB.