Category: DCC News

All the fun of the FAIR: FAIR WG inaugural meeting held

Re-posted from the EOSCSecretariat website
 
Thursday, 4 July was the inaugural meeting of the EOSC FAIR Working Group. We have 26 members representing 20 different Member States and Associated Countries and a remit which covers FAIR practice, a …

DMPonline subscription model – revisited

Last November we launched a subscription model for DMPonline. Thank to all who have signed up in the first 6 months. We currently have over 40 institutional subscribers in the UK, Netherlands and Sweden. Their stories are being featured in the Knowled…

DMPonline release notes: April 2019

DMPonline release notes: April 2019
We do appreciate your feedback, as it is very valuable for us to know what works well for you, or anything else you wish to be fixed, improved or deployed! If you are interested to see what we are currently working o…

Sam Rust explains benefits of API for DMPonline

Sam Rust – explains benefits of DMPonline API

 

DMPonline has a REST API to enable you to query some statistics about your organisation, download your guidance, check the templates in the tool, and even create plans for your users.  This blog post will detail some clients to use APIs in general, the data you can pull from our api, and run through a more detailed example.  We will also talk about some of the planned extensions to the API to enable you to run your own queries on your organisation’s data.

What is an API?

An API, or “Application Programming Interface”, is simply a defined format for multiple systems to communicate over.  It defines the language for information interchange, both that of the requests from the client, and that of the response from the server.  A REST, or “Representational State Transfer”, API means that the API responds to a predefined set of stateless operations.

How do I use an API?

A variety of clients can be used to communicate with an API.  At a basic level, any system which can make a HTTP request can use an API.  Commonly, this would be any programming language, for example Python and the requests package it uses to make web requests.  For a more user-friendly approach, A GUI application like Postman can be used.  For the purposes of this tutorial, we will use Postman to request data from the API, so please install that now if you want to follow-along.

Who can use the DMPonline API?

Currently, the DMPonline API usage is available to all org-admins within the tool.  To check if you have permission to use the API, go to your ‘edit profile’ page within DMPonline by clicking your name in the top-right corner.  Scrolling down, under your Institutional Credentials, if you have API access, you will see two sections `API token` and `API Information`.  If you are an org-admin and do not see these sections, please get in touch and we can extend this access to your account.

What can the DMPonline API do?

The DMPonline API has a variety of endpoints, all of which are detailed in full on our wiki.

In this guide, we will focus on the statistics endpoints, but the others are listed at the above link.  Currently, API users on DMPonline can pull out the number of users which have joined their organisation; the number of plans by any user based off of your organisation’s non-customized templates; the number of plans created by your organisation’s users grouped by template; and some generic metadata about all of the plans created by your organisation’s users.  Additionally, all of these endpoints can be passed a start and end date to scope the results to those created within the specified time-period.  All DMPonline endpoints currently respond only in JSON, but if there is a demand for other outputs, these could be developed.  If you have need for a different format, please let us know.

An example API use case

Here, we will walk through the process of querying the users-joined so far in 2019 for our organisation.  In this example, we will detail the specifics for using Postman, but the steps are transferable to the application or language of your choice.  Assuming you’ve installed Postman, create a new Request object using the orange ‘New’ button in the top-left.

Screenshot 1: New Request Button

To do so, we’ll want to give the request a name and an optional description. 

Since we are querying the users who have joined our system, I’ll call the request ‘Users Joined’. Here we must also choose to create a collection (essentially a folder) to keep our request grouped with like requests.

Screenshot 2: New Request Dialogue

After creating the request, we next need to add a request URL.  We want to use our base URL for DMPonline, ‘https://dmponline.dcc.ac.uk/’, along with the URL-partial given in the api wiki page in the statistics section, for the command users_joined, ‘/api/v0/statistics/users_joined’. 

Screenshot 3: Request URL

Now that we have told Postman the request we want to make, we next need to turn towards authentication and authorization.  The wiki section on shared headers details the convention used here.  We do Authorization via the ‘API token’ which you can find, as described above, on your ‘edit profile’ page.  The second header, `Content-Type`, lets the API know that your want a JSON response.  To put these into Postman, we will need to go to the ‘Headers’ tab, immediately below where we input the request URL.

These headers will be key/value pairs.  The first key “Authorization” will have a value of “Token token=<API_TOKEN>”, where <API_TOKEN> is the token from your edit profile page, as described above.  This token is used to both authenticate us as the user, and to authorize us to view this API endpoint.  The latter key, “Content-Type” should have a value of “application/json”.  No matter which API endpoint we use, we will want to keep these two headers the same.

If we now hit send, the above is sufficient to query the API for the number of users in your organisation, and you should get a response in JSON.  Congratulations, you’ve just made your first successful API request.  

Don’t get too happy just yet, because we said that for this example, we are only interested in users joining throughout 2019, so we need to scope this query to only include users created between those dates.

 

The API Wiki Section on Statistics details how this is done.  Essentially, we are allowed to specify two additional arguments as a query string, ‘start_date’ and ‘end_date’, and specify these values as YYYY-MM-DD.  In Postman, we will need to navigate to the ‘Params’ tab where we had previously selected ‘Headers’.  Here we will add a key, ‘start_date’, and a value, ‘2019-01-01’.  We do not need to specify an end_date, as 2019 has not yet ended so an open-ended query is sufficient for our purpose.

Re-running the query, we now see that a smaller total number of users has joined throughout 2019 rather than over the whole year.

Planned Extensions

We have a planned extensions to the API which should be released in the near future.  We are going to be creating an export of the full-text of all your organisation’s plans in the system as a (large) JSON object.  This should enable organisations to run their own, more interesting, analyses on their data, whether statistical, semantic analysis, or otherwise.  It is also much more flexible than us defining the set of questions which you are allowed to ask about your data.

To keep up with DMPonline news, you can subscribe to the RSS feed to receive our blogs and tweets, and watch GitHub for code updates. You are also welcome to discuss our new features on the user group mailing list.

DMPonline user group: where next?

We’ve run a couple of user group sessions over the last few weeks. Many thanks to those who attended in Amsterdam and Manchester. It was particularly helpful to hear how you are using the tool and what changes you would like. The DCC team met on Monday to process your feedback and have set the following priorities:

  1. Full text API (#2086)
    You wanted an extension of the plans API to allow authorised admin users to pull out the full text of DMPs for their organisation. Sam has already developed this but it takes a long time to run in real-time so we are going to harvest the data overnight. This requires some changes to our infrastructure but we should be able to release the API later this month.
  2. Reviewer admin permission (#2087)
    It seems that many of you have multiple reviewers of plans and would like this to be a separate role that can be assigned by the org admin. We’re in the process of adding this, and will adjust our contract so you’re not paying more for multiple plan reviewers.
  3. Adding a field for School / Division / Department (#2088)
    We plan to add one field to the edit profile page for which you can name and define a controlled value list. This will allow you to identify the sub-unit affiliation of your users. In time we could extend the functionality to allow you to pull stats on this (e.g. plans by school) and to allow customisations to be done by sub-unit level. Currently it is only the guidance that works in their vein.
  4. Conditional questions (#1772)
    This is a bigger piece of feature development, which we have scheduled for the summer months. It was top priority for you so we will add it first.
  5. Custom section on funder templates (#2072)
    Many of you have the same custom section on funder templates and want to create it once and apply across all / a selection. This will be provided as a new feature in Summer 2019.
  6. Plan versioning 
    Again, this is a bigger piece of feature development, which we have scheduled for the summer months.

There were a number of smaller items raised during discussion too. These have been added as tickets (see the ‘user group’) label and will be addressed during the coming few sprints.

Other areas of interest which represent larger feature development and are to be placed into development plans have been captured on the ‘Future enhancements’ section of the wiki for now.
 
Manchester’s case study on using DMPonline was really inspiring and we wanted to ask how we can support others to do some similar things. We realise not everyone has development effort in-house so we wondered whether it would help if we wrote a script to send you weekly/monthly email notifications of new users and plans so you can monitor usage and start to interact with and assist users more? Let us know what you think.
 
We are due to push out a new release immediately after Easter. Amongst other things, this includes:
  • bug fixes for guidance / comment display when text runs outside the box
  • adjustments to comment notification so users don’t receive multiple emails when you provide feedback
  • display of themed guidance in ‘customise template’ preview
  • table-styling fixes
  • plans by template statistics

You can see these on the test site now and in live deployment soon.

All feedback welcome!
 
Magdalena, Diana, Sam & Sarah
 

More new jobs at the DCC – closing date 10th May

New projects in which we’re advancing the FAIR agenda and continued growth of our consultancy work mean that we are recruiting again at the DCC. If that’s enough to pique your interest, follow the link now and ensure that your application reaches us by the closing date of 10th May 2019. If not, read on and let me try to persuade you why this could be the opportunity for you.

We’re recruiting at least two new staff to join our distributed team at the DCC. The successful applicants will spend much of their time on one or other of the FAIRs FAIR or RDA Europe 4 projects, but will also have the opportunity to engage with other areas of DCC work such as training and consultancy. You will already have some knowledge of issues around data reuse, which you may have acquired as someone who generates data, who uses it or who supports those who do either of these things, in either the public or private sector. We’re not looking for experts – just for people who have some relevant knowledge and have the capacity to expand that through working with our friendly and knowledgeable team. There is a significant amount of travel associated with most of our work and you will have the opportunity to take advantage of this, along with the opportunity to work with leading organisations around the world.

Depending on your skills and interests you’ll spend time on other areas of DCC activity as well. In recent months, these have included training  for universities and researchers in the UK, Japan, Malaysia, Norway and South Africa, and work for funders such as the ERC and Wellcome Trust and for UK national research facilities. We run services such as DMPonline for customers around the world, and there’s also our annual conference, IDCC, and much more. We offer flexible working practices and a supportive working environment with a diverse and friendly team in Edinburgh and Glasgow. We encourage applications from all communities.

You’ll need to use the University of Edinburgh’s online recruitment system to apply. This requires you to register; please don’t leave it until the last minute to enter your application. (You can revise an application that’s already been submitted as long as you do so before the closing date.) If you would like to discuss these posts informally, please contact me (Kevin Ashley – director@dcc.ac.uk), Sarah Jones (sarah.jones@glasgow.ac.uk) or the DCC helpdesk (info@dcc.ac.uk) Applications will close at 1700 UTC on May 10th. For the right candidates, this is an ideal opportunity to broaden your knowledge and experience. That candidate is very possibly you – we look forward to hearing from you.

 

DMPonline – information about phasing out for non subscribers and comparison of subscription models

Dear DMPonline Administrators,
In October 2018, we announced that DMPonline is going to charge its administrative users. We have now formalised the phasing out DMPonline for non-subscribers, which will begin in April 2019. For organisations that d…

Let the ideas flow…

Colleagues ask me what an unconference is. I say it’s playing it by ear. You come along without an agenda and see what sparks everyone’s interest on the day. I’m really excited that we’re trialling an unconference at IDCC, particularly on the last day. People are always so buzzing with ideas and inspired by Cliff Lynch’s insightful synopsis, that it’s often hard to get them out the door! This year you can bring all that enthusiasm back and focus on the key ideas to emerge.

There are lots of different types of interactive sessions that can take place:

  • Group discussions
  • Learn about / how to
  • Fishbowl dialogues
  • Show and tell
  • Knowledge café

Whether you have a task you want to work on with others, a fresh idea or solution you want to demo, or topics you want to discuss to learn from others’ experience, the unconference provides a space for you to achieve all of this. Just pitch an idea, gather like-minded souls and get cracking.

Three of us have thrown our hat in the ring to act as compares for the day: me, Peter Neish of Melbourne Uni and Adam Bell of AARNet. We don’t have a grand plan and certainly don’t have all the ideas and answers. We’re just there to help steer things along. All of you fresh recruits are what will make this happen.

As it stands there is very little structure for the day. We’ll begin standing in the plenary room – don’t get comfortable and start checking email… You need to listen up, divide into groups and start interacting! We’ll invite some pitches from people who volunteer ideas and then all vote with our feet. We’ve scheduled three parallel slots but can have more or less as the mood takes us. There are no rules.

The day will be very free-form. Tea/coffee and light bites will be available all day long. There are nominal slots for breaks and lunch but you can pause when suits. We’ll periodically pitch new ideas in the plenary room but these will also be recorded online so you can join any group at any point.

Ideas for the unconference can be put forward at any time. You can respond to this blog with a comment, add details on the inspiration boards that will be available during IDCC, or pitch ideas in the etherpad

I’m interested in comparing findings from all the new FAIR reports. There have been several studies in recent years that either investigate practice in different disciplines, make recommendations on what needs to change, or release statement of commitment to change the status quo.

  • How similar are the findings and recommendations across these studies?
  • Are there significant differences in practice/needs across disciplines & countries?
  • What can we learn from the advancements that different groups have made?
  • What are the biggest challenges that need international collaboration?

The Australasian digital preservation community are keen to explore how communities of practice can support the sharing of digital preservation knowledge and skills. Lessons could perhaps be learned from the excellent Data Curation Network in the USA or the Community of Practice which is emerging from a set of European projects and the Research Data Alliance skills group.

Undoubtedly you will also have pet projects that you want to collaborate on, cool tools to show, or things you want to teach others. Get your thinking caps on and bring what you need to the unconference. We look forward to seeing you there!

Image CC-BY: Unconference by JD Lasica