How to output response of type atom/xml feed into Jquery DataTable?

Question

How to output response of type atom/xml feed (from arxiv call) into Jquery DataTable?

I have the datatable working for a simple json from Ajax call to flask server example.

When i try to do it with the xml from an arxiv api response, i cant seem to get it to display in the datatable (though i can just print the raw xml using <pre lang="xml" > or json).

I also tried to convert to json first via python dictionary, but still couldnt get it formatted into datatable as im unsure how to access the properties properly in the Ajax call when theyre deeper than the first level as in the basic example linked.

The HTML in template:

<table id="arxivtable" class="display" style="width:100%">
  <thead>
      <tr>
          <th>title</th>
          <th>id</th>
          <th>link</th>
          <th>author</th>
          <th>published</th>
      </tr>
  </thead>
</table>

I tried via xml :

  $('#arxivtable').DataTable({
            "ajax": {
                // "url": "static/objects2.txt", // This works for the static file
                "url": "/get_arxivpapers", // This now works too thanks to @kthorngren
                "dataType": "xml",
                "type":"GET",
                "dataSrc": "{{name}}",
                "contentType":"application/atom+xml"
            },
            "columns": [
                {"data": "title"},
                {
                        "data": "link",
                        "render": function(data, type, row, meta){
                           if(type === 'display'){
                               data = '<a href="' + data + '">' + data + '</a>';
                           }

                           return data;
                        }
                      },
                      { "data": "id" },
                      { "data": "link" },
                       { "data": "author" },
                       { "data": "journal" },
                       { "data": "published" },
                       { "data": "summary" }

            ]
            });

JSON from AJAX call:

    {
  "feed": {
    "@xmlns": "http://www.w3.org/2005/Atom",
    "link": {
      "@href": "http://arxiv.org/api/query?search_query%3Dall%3Aeinstein%26id_list%3D%26start%3D0%26max_results%3D2",
      "@rel": "self",
      "@type": "application/atom+xml"
    },
    "title": {
      "@type": "html",
      "#text": "ArXiv Query: search_query=all:einstein&id_list=&start=0&max_results=2"
    },
    "id": "http://arxiv.org/api/vehKAQR+bheXtHwJw3qx/OG/XXw",
    "updated": "2022-06-14T00:00:00-04:00",
    "opensearch:totalResults": {
      "@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
      "#text": "36970"
    },
    "opensearch:startIndex": {
      "@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
      "#text": "0"
    },
    "opensearch:itemsPerPage": {
      "@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
      "#text": "2"
    },
    "entry": [
      {
        "id": "http://arxiv.org/abs/1801.05533v2",
        "updated": "2018-11-22T14:04:43Z",
        "published": "2018-01-17T03:05:51Z",
        "title": "Einstein-Weyl structures on almost cosymplectic manifolds",
        "summary": "",
        "author": {
          "name": "Xiaomin Chen"
        },
        "arxiv:comment": {
          "@xmlns:arxiv": "http://arxiv.org/schemas/atom",
          "#text": "accepted by Periodica Mathematica Hungarica, 14 pages, no figures"
        },
        "link": [
          {
            "@href": "http://arxiv.org/abs/1801.05533v2",
            "@rel": "alternate",
            "@type": "text/html"
          },
          {
            "@title": "pdf",
            "@href": "http://arxiv.org/pdf/1801.05533v2",
            "@rel": "related",
            "@type": "application/pdf"
          }
        ],
        "arxiv:primary_category": {
          "@xmlns:arxiv": "http://arxiv.org/schemas/atom",
          "@term": "math.DG",
          "@scheme": "http://arxiv.org/schemas/atom"
        },
        "category": [
          {
            "@term": "math.DG",
            "@scheme": "http://arxiv.org/schemas/atom"
          },
          {
            "@term": "53D10, 53D15",
            "@scheme": "http://arxiv.org/schemas/atom"
          }
        ]
      },
      {
        "id": "http://arxiv.org/abs/0802.2137v3",
        "updated": "2008-04-01T04:36:21Z",
        "published": "2008-02-15T04:40:56Z",
        "title": "",
        "summary": ".",
        "author": {
          "name": ""
        },
        "arxiv:comment": {
          "@xmlns:arxiv": "http://arxiv.org/schemas/atom",
          "#text": "18 pages, added Theorem 5"
        },
        "link": [
          {
            "@href": "http://arxiv.org/abs/0802.2137v3",
            "@rel": "alternate",
            "@type": "text/html"
          },
          {
            "@title": "pdf",
            "@href": "http://arxiv.org/pdf/0802.2137v3",
            "@rel": "related",
            "@type": "application/pdf"
          }
        ],
        "arxiv:primary_category": {
          "@xmlns:arxiv": "http://arxiv.org/schemas/atom",
          "@term": "math.DG",
          "@scheme": "http://arxiv.org/schemas/atom"
        },
        "category": [
          {
            "@term": "math.DG",
            "@scheme": "http://arxiv.org/schemas/atom"
          },
          {
            "@term": "53C30; 53C25",
            "@scheme": "http://arxiv.org/schemas/atom"
          }
        ]
      }
    ]
  }
}

Or the original atom/xml:

<feed xmlns="http://www.w3.org/2005/Atom">
  <link href="http://arxiv.org/api/query?search_query%3Dall%3Aeinstein%26id_list%3D%26start%3D0%26max_results%3D2" rel="self" type="application/atom+xml">
  <title type="html">ArXiv Query: search_query=all:einstein&amp;id_list=&amp;start=0&amp;max_results=2</title>
  <id>http://arxiv.org/api/vehKAQR+bheXtHwJw3qx/OG/XXw</id>
  <updated>2022-06-14T00:00:00-04:00</updated>
  <opensearch:totalresults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">36970</opensearch:totalresults>
  <opensearch:startindex xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">0</opensearch:startindex>
  <opensearch:itemsperpage xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">2</opensearch:itemsperpage>
  <entry>
    <id>http://arxiv.org/abs/1801.05533v2</id>
    <updated>2018-11-22T14:04:43Z</updated>
    <published>2018-01-17T03:05:51Z</published>
    <title></title>
    <summary> 
</summary>
    <author>
      <name></name>
    </author>
    <arxiv:comment xmlns:arxiv="http://arxiv.org/schemas/atom">accepted by Periodica Mathematica Hungarica, 14 pages, no figures</arxiv:comment>
    <link href="http://arxiv.org/abs/1801.05533v2" rel="alternate" type="text/html">
    <link title="pdf" href="http://arxiv.org/pdf/1801.05533v2" rel="related" type="application/pdf">
    <arxiv:primary_category xmlns:arxiv="http://arxiv.org/schemas/atom" term="math.DG" scheme="http://arxiv.org/schemas/atom">
    <category term="math.DG" scheme="http://arxiv.org/schemas/atom">
    <category term="53D10, 53D15" scheme="http://arxiv.org/schemas/atom">
  </category></category></arxiv:primary_category></entry>
  <entry>
    <id>http://arxiv.org/abs/0802.2137v3</id>
    <updated>2008-04-01T04:36:21Z</updated>
    <published>2008-02-15T04:40:56Z</published>
    <title></title>
    <summary>
</summary>
    <author>
      <name></name>
    </author>
    <arxiv:comment xmlns:arxiv="http://arxiv.org/schemas/atom"></arxiv:comment>
    <link href="http://arxiv.org/abs/0802.2137v3" rel="alternate" type="text/html">
    <link title="pdf" href="http://arxiv.org/pdf/0802.2137v3" rel="related" type="application/pdf">
    <arxiv:primary_category xmlns:arxiv="http://arxiv.org/schemas/atom" term="math.DG" scheme="http://arxiv.org/schemas/atom">
    <category term="math.DG" scheme="http://arxiv.org/schemas/atom">
    <category term="53C30; 53C25" scheme="http://arxiv.org/schemas/atom">
  </category></category></arxiv:primary_category></entry>
</feed>

The End Point:

@app.route('/get_arxivpapers')
def getArxivPapers(name="einstein"):

    max_results = 2
    searchterm = name.replace("_", "&#32")
    url = 'http://export.arxiv.org/api/query?search_query=all:' + searchterm + '&start=0&' + 'max_results='+ str(max_results)
    data = urllib.request.urlopen(url)

    # data_dict = xmltodict.parse(data)
    # json_data = json.dumps(data_dict)
    # print(json_data)
    # return jsonify(json_data)

    return data.read().decode('utf-8')

i also tried the properties in the Ajax columns call like: data: feed.entry.id but then i get invalid json response. — jsky
– jsky, Commented Jun 14, 2022 at 5:27

andrewJames · Accepted Answer · 2022-06-14 14:58:26Z

1

I will use your JSON source data instead of the XML, since that is easier to handle in DataTables.

Here is a basic demo, to start with, followed by some explanatory notes:

<!doctype html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Demo</title>

  <script src="https://code.jquery.com/jquery-3.5.1.min.js"></script>
  <script src="https://cdn.datatables.net/1.10.21/js/jquery.dataTables.min.js"></script>
  <link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.21/css/jquery.dataTables.min.css">
</head>

<body>

<div style="margin: 20px;">

    <table id="arxivtable" class="display" style="width:100%">
  <thead>
      <tr>
          <th>title</th>
          <th>id</th>
          <th>link</th>
          <th>author</th>
          <th>published</th>
          <th>summary</th>
      </tr>
  </thead>
</table>


</div>

<script type="text/javascript">
    
  $(document).ready(function(){

    $('#arxivtable').DataTable({
      "ajax": {
        url: "YOUR_URL_GOES_HERE",
        dataSrc: "feed.entry"
      },
      "columns": [
        {"data": "title"},
        { "data": "id" },
        { "data": "link[].@href" },
        { "data": "author.name" },
        { "data": "published" },
        { "data": "summary" }
      ]
    });
    
  });

</script>


</body>
</html>

Notes

1 - Because you have provided hard-coded HTML column headers, you need to make sure the number of those headers matches the number of columns defined in the DataTable. Alternatively, you can remove the HTML <thead> section and use the DataTables columns.title option.

2 - Your Ajax JSON source data contains an array [ ... ]. DataTables needs to know where this array is located in your JSON response, as part of the Ajax handling option, so that it can iterate over that array. Each element in the array will be used to create a row of HTML table data. The ajax.dataSrc option therefore needs to be set accordingly:

dataSrc: "feed.entry"

Once you have set the above Ajax JSON starting point correctly, then you can use field names for each separate column data value - as shown below.

3 - The author JSON value is actually an object:

"author": {
  "name": "Xiaomin Chen"
},

Therefore you need to drill down into that to get the field you want to show in the DataTable:

{ "data": "author.name" },

4 - I removed your column renderer function to keep my initial demo simple, but it can be used to access fields and sub-fields - and concatenate strings and other values as needed (as in your example in the question).

5 - The link JSON value is actually an array of objects. For my basic demo, I just accessed the final entry in that array, and then took the href field:

{ "data": "link[].@href" },

This may not be what you want. You may want to only choose links of a certain type, or choose all links, or something different.

This is where DataTables is limited in what it can handle. It cannot display arbitrary nested JSON values of this type (not surprisingly).

In such cases, you would need to re-structure the JSON, prior to sending it to DataTables - or restructure it in a dataSrc function inside DataTables itself:

"dataSrc": function ( json ) { ...transform and return your JSON here... }

6 - I was not sure what you wanted to display for { "data": "journal" }. I did not see anything called journal in the JSON.

7 - Note that all the source JSON data outside of the feed.entry array is also not available to DataTables. DataTables can only iterate over that outer array. Anything you may also need which is not in that outer array would need to be added to the array, to be accessible to DataTables.

See also Nested object data (arrays) and Nested object data (objects) for more related notes.

answered Jun 14, 2022 at 14:58

andrewJames

22.6k11 gold badges32 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

jsky Over a year ago

awesome, i actually thought i tried this or something similar. your solution here wasnt loading the datatable when i was returning jsonifyed data at the endpoint. (which is what i did in the simple example where i constructed the json. perhaps i didnt need to. when i just return the dumped unjsonifyed data here it loads properly using your example. not sure why the discrepancy exactly but this was much easier than what i was just trying, reconstructing entry elements in an e tree and stripping namespaces. thanks.

jsky Over a year ago

i was previously using dataSrc: {{name}} to pass the name parameter into the endpoint function . getArxivPapers(name="einstein"). einstein is only meant to be the default param but the results here are only showing that search. how do i pass the name param into the ajax call? dataSrc: "feed.entry", data: "name: {{name}}" doesnt seem to work.

andrewJames Over a year ago

ajax.dataSrc is for receiving Ajax data from the server, so that would not work. Also {{...}} is not a JavaScript variable - looks more like a templating feature. Anyway, yes, you use ajax.data to send data to the server - but, again it expects a JavaScript variable, or a hard-coded string. And also, if you want the value to be dynamic, then you have to use the function syntax: data: function ( d ) { ... }.

andrewJames Over a year ago

There are lots of questions covering this (with and without HTML forms). Just one example: Submit data from jQuery DataTables. If that does not help, there are probably better/more relevant examples, or you can ask a new question showing the specific problem in more detail. Trying to ask and answer in comments like this may be difficult - but maybe this has helped.

jsky Over a year ago

i didnt need to pass by url, i just wasnt passing and accessing the param from the req properly. thanks for your answer.

|

Collectives™ on Stack Overflow

How to output response of type atom/xml feed into Jquery DataTable?

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related