Category: DCC News

DMPonline release notes: April 2019

DMPonline release notes: April 2019
We do appreciate your feedback, as it is very valuable for us to know what works well for you, or anything else you wish to be fixed, improved or deployed! If you are interested to see what we are currently working o…

Sam Rust explains benefits of API for DMPonline

Sam Rust – explains benefits of DMPonline API

 

DMPonline has a REST API to enable you to query some statistics about your organisation, download your guidance, check the templates in the tool, and even create plans for your users.  This blog post will detail some clients to use APIs in general, the data you can pull from our api, and run through a more detailed example.  We will also talk about some of the planned extensions to the API to enable you to run your own queries on your organisation’s data.

What is an API?

An API, or “Application Programming Interface”, is simply a defined format for multiple systems to communicate over.  It defines the language for information interchange, both that of the requests from the client, and that of the response from the server.  A REST, or “Representational State Transfer”, API means that the API responds to a predefined set of stateless operations.

How do I use an API?

A variety of clients can be used to communicate with an API.  At a basic level, any system which can make a HTTP request can use an API.  Commonly, this would be any programming language, for example Python and the requests package it uses to make web requests.  For a more user-friendly approach, A GUI application like Postman can be used.  For the purposes of this tutorial, we will use Postman to request data from the API, so please install that now if you want to follow-along.

Who can use the DMPonline API?

Currently, the DMPonline API usage is available to all org-admins within the tool.  To check if you have permission to use the API, go to your ‘edit profile’ page within DMPonline by clicking your name in the top-right corner.  Scrolling down, under your Institutional Credentials, if you have API access, you will see two sections `API token` and `API Information`.  If you are an org-admin and do not see these sections, please get in touch and we can extend this access to your account.

What can the DMPonline API do?

The DMPonline API has a variety of endpoints, all of which are detailed in full on our wiki.

In this guide, we will focus on the statistics endpoints, but the others are listed at the above link.  Currently, API users on DMPonline can pull out the number of users which have joined their organisation; the number of plans by any user based off of your organisation’s non-customized templates; the number of plans created by your organisation’s users grouped by template; and some generic metadata about all of the plans created by your organisation’s users.  Additionally, all of these endpoints can be passed a start and end date to scope the results to those created within the specified time-period.  All DMPonline endpoints currently respond only in JSON, but if there is a demand for other outputs, these could be developed.  If you have need for a different format, please let us know.

An example API use case

Here, we will walk through the process of querying the users-joined so far in 2019 for our organisation.  In this example, we will detail the specifics for using Postman, but the steps are transferable to the application or language of your choice.  Assuming you’ve installed Postman, create a new Request object using the orange ‘New’ button in the top-left.

Screenshot 1: New Request Button

To do so, we’ll want to give the request a name and an optional description. 

Since we are querying the users who have joined our system, I’ll call the request ‘Users Joined’. Here we must also choose to create a collection (essentially a folder) to keep our request grouped with like requests.

Screenshot 2: New Request Dialogue

After creating the request, we next need to add a request URL.  We want to use our base URL for DMPonline, ‘https://dmponline.dcc.ac.uk/’, along with the URL-partial given in the api wiki page in the statistics section, for the command users_joined, ‘/api/v0/statistics/users_joined’. 

Screenshot 3: Request URL

Now that we have told Postman the request we want to make, we next need to turn towards authentication and authorization.  The wiki section on shared headers details the convention used here.  We do Authorization via the ‘API token’ which you can find, as described above, on your ‘edit profile’ page.  The second header, `Content-Type`, lets the API know that your want a JSON response.  To put these into Postman, we will need to go to the ‘Headers’ tab, immediately below where we input the request URL.

These headers will be key/value pairs.  The first key “Authorization” will have a value of “Token token=<API_TOKEN>”, where <API_TOKEN> is the token from your edit profile page, as described above.  This token is used to both authenticate us as the user, and to authorize us to view this API endpoint.  The latter key, “Content-Type” should have a value of “application/json”.  No matter which API endpoint we use, we will want to keep these two headers the same.

If we now hit send, the above is sufficient to query the API for the number of users in your organisation, and you should get a response in JSON.  Congratulations, you’ve just made your first successful API request.  

Don’t get too happy just yet, because we said that for this example, we are only interested in users joining throughout 2019, so we need to scope this query to only include users created between those dates.

 

The API Wiki Section on Statistics details how this is done.  Essentially, we are allowed to specify two additional arguments as a query string, ‘start_date’ and ‘end_date’, and specify these values as YYYY-MM-DD.  In Postman, we will need to navigate to the ‘Params’ tab where we had previously selected ‘Headers’.  Here we will add a key, ‘start_date’, and a value, ‘2019-01-01’.  We do not need to specify an end_date, as 2019 has not yet ended so an open-ended query is sufficient for our purpose.

Re-running the query, we now see that a smaller total number of users has joined throughout 2019 rather than over the whole year.

Planned Extensions

We have a planned extensions to the API which should be released in the near future.  We are going to be creating an export of the full-text of all your organisation’s plans in the system as a (large) JSON object.  This should enable organisations to run their own, more interesting, analyses on their data, whether statistical, semantic analysis, or otherwise.  It is also much more flexible than us defining the set of questions which you are allowed to ask about your data.

To keep up with DMPonline news, you can subscribe to the RSS feed to receive our blogs and tweets, and watch GitHub for code updates. You are also welcome to discuss our new features on the user group mailing list.

DMPonline user group: where next?

We’ve run a couple of user group sessions over the last few weeks. Many thanks to those who attended in Amsterdam and Manchester. It was particularly helpful to hear how you are using the tool and what changes you would like. The DCC team met on Monday to process your feedback and have set the following priorities:

  1. Full text API (#2086)
    You wanted an extension of the plans API to allow authorised admin users to pull out the full text of DMPs for their organisation. Sam has already developed this but it takes a long time to run in real-time so we are going to harvest the data overnight. This requires some changes to our infrastructure but we should be able to release the API later this month.
  2. Reviewer admin permission (#2087)
    It seems that many of you have multiple reviewers of plans and would like this to be a separate role that can be assigned by the org admin. We’re in the process of adding this, and will adjust our contract so you’re not paying more for multiple plan reviewers.
  3. Adding a field for School / Division / Department (#2088)
    We plan to add one field to the edit profile page for which you can name and define a controlled value list. This will allow you to identify the sub-unit affiliation of your users. In time we could extend the functionality to allow you to pull stats on this (e.g. plans by school) and to allow customisations to be done by sub-unit level. Currently it is only the guidance that works in their vein.
  4. Conditional questions (#1772)
    This is a bigger piece of feature development, which we have scheduled for the summer months. It was top priority for you so we will add it first.
  5. Custom section on funder templates (#2072)
    Many of you have the same custom section on funder templates and want to create it once and apply across all / a selection. This will be provided as a new feature in Summer 2019.
  6. Plan versioning 
    Again, this is a bigger piece of feature development, which we have scheduled for the summer months.

There were a number of smaller items raised during discussion too. These have been added as tickets (see the ‘user group’) label and will be addressed during the coming few sprints.

Other areas of interest which represent larger feature development and are to be placed into development plans have been captured on the ‘Future enhancements’ section of the wiki for now.
 
Manchester’s case study on using DMPonline was really inspiring and we wanted to ask how we can support others to do some similar things. We realise not everyone has development effort in-house so we wondered whether it would help if we wrote a script to send you weekly/monthly email notifications of new users and plans so you can monitor usage and start to interact with and assist users more? Let us know what you think.
 
We are due to push out a new release immediately after Easter. Amongst other things, this includes:
  • bug fixes for guidance / comment display when text runs outside the box
  • adjustments to comment notification so users don’t receive multiple emails when you provide feedback
  • display of themed guidance in ‘customise template’ preview
  • table-styling fixes
  • plans by template statistics

You can see these on the test site now and in live deployment soon.

All feedback welcome!
 
Magdalena, Diana, Sam & Sarah
 

More new jobs at the DCC – closing date 10th May

New projects in which we’re advancing the FAIR agenda and continued growth of our consultancy work mean that we are recruiting again at the DCC. If that’s enough to pique your interest, follow the link now and ensure that your application reaches us by the closing date of 10th May 2019. If not, read on and let me try to persuade you why this could be the opportunity for you.

We’re recruiting at least two new staff to join our distributed team at the DCC. The successful applicants will spend much of their time on one or other of the FAIRs FAIR or RDA Europe 4 projects, but will also have the opportunity to engage with other areas of DCC work such as training and consultancy. You will already have some knowledge of issues around data reuse, which you may have acquired as someone who generates data, who uses it or who supports those who do either of these things, in either the public or private sector. We’re not looking for experts – just for people who have some relevant knowledge and have the capacity to expand that through working with our friendly and knowledgeable team. There is a significant amount of travel associated with most of our work and you will have the opportunity to take advantage of this, along with the opportunity to work with leading organisations around the world.

Depending on your skills and interests you’ll spend time on other areas of DCC activity as well. In recent months, these have included training  for universities and researchers in the UK, Japan, Malaysia, Norway and South Africa, and work for funders such as the ERC and Wellcome Trust and for UK national research facilities. We run services such as DMPonline for customers around the world, and there’s also our annual conference, IDCC, and much more. We offer flexible working practices and a supportive working environment with a diverse and friendly team in Edinburgh and Glasgow. We encourage applications from all communities.

You’ll need to use the University of Edinburgh’s online recruitment system to apply. This requires you to register; please don’t leave it until the last minute to enter your application. (You can revise an application that’s already been submitted as long as you do so before the closing date.) If you would like to discuss these posts informally, please contact me (Kevin Ashley – director@dcc.ac.uk), Sarah Jones (sarah.jones@glasgow.ac.uk) or the DCC helpdesk (info@dcc.ac.uk) Applications will close at 1700 UTC on May 10th. For the right candidates, this is an ideal opportunity to broaden your knowledge and experience. That candidate is very possibly you – we look forward to hearing from you.

 

DMPonline – information about phasing out for non subscribers and comparison of subscription models

Dear DMPonline Administrators,
In October 2018, we announced that DMPonline is going to charge its administrative users. We have now formalised the phasing out DMPonline for non-subscribers, which will begin in April 2019. For organisations that d…

Let the ideas flow…

Colleagues ask me what an unconference is. I say it’s playing it by ear. You come along without an agenda and see what sparks everyone’s interest on the day. I’m really excited that we’re trialling an unconference at IDCC, particularly on the last day. People are always so buzzing with ideas and inspired by Cliff Lynch’s insightful synopsis, that it’s often hard to get them out the door! This year you can bring all that enthusiasm back and focus on the key ideas to emerge.

There are lots of different types of interactive sessions that can take place:

  • Group discussions
  • Learn about / how to
  • Fishbowl dialogues
  • Show and tell
  • Knowledge café

Whether you have a task you want to work on with others, a fresh idea or solution you want to demo, or topics you want to discuss to learn from others’ experience, the unconference provides a space for you to achieve all of this. Just pitch an idea, gather like-minded souls and get cracking.

Three of us have thrown our hat in the ring to act as compares for the day: me, Peter Neish of Melbourne Uni and Adam Bell of AARNet. We don’t have a grand plan and certainly don’t have all the ideas and answers. We’re just there to help steer things along. All of you fresh recruits are what will make this happen.

As it stands there is very little structure for the day. We’ll begin standing in the plenary room – don’t get comfortable and start checking email… You need to listen up, divide into groups and start interacting! We’ll invite some pitches from people who volunteer ideas and then all vote with our feet. We’ve scheduled three parallel slots but can have more or less as the mood takes us. There are no rules.

The day will be very free-form. Tea/coffee and light bites will be available all day long. There are nominal slots for breaks and lunch but you can pause when suits. We’ll periodically pitch new ideas in the plenary room but these will also be recorded online so you can join any group at any point.

Ideas for the unconference can be put forward at any time. You can respond to this blog with a comment, add details on the inspiration boards that will be available during IDCC, or pitch ideas in the etherpad

I’m interested in comparing findings from all the new FAIR reports. There have been several studies in recent years that either investigate practice in different disciplines, make recommendations on what needs to change, or release statement of commitment to change the status quo.

  • How similar are the findings and recommendations across these studies?
  • Are there significant differences in practice/needs across disciplines & countries?
  • What can we learn from the advancements that different groups have made?
  • What are the biggest challenges that need international collaboration?

The Australasian digital preservation community are keen to explore how communities of practice can support the sharing of digital preservation knowledge and skills. Lessons could perhaps be learned from the excellent Data Curation Network in the USA or the Community of Practice which is emerging from a set of European projects and the Research Data Alliance skills group.

Undoubtedly you will also have pet projects that you want to collaborate on, cool tools to show, or things you want to teach others. Get your thinking caps on and bring what you need to the unconference. We look forward to seeing you there!

Image CC-BY: Unconference by JD Lasica

Work together, not alone: doing things the Batswana way

Botswana University Campus Appeal statue. CC-BY by Heritage Adventures

International Data Week in Botswana was unlike any conference I’ve been to before. It brought together over 820 participants from 66 countries. Botswana was represented by over 250 participants and the engagement from the wider African continent was marked and added a different dimension and richness of conversation that isn’t always present at RDA plenaries.

The conference was opened by the President of Botswana, His Excellency Dr Mokgweetsi Masisi, who gave an impassioned and informed address which showed his genuine understanding and concern about data issues. Dr Masisi spoke on the importance of a knowledge-based economy and the potential for Open Data and Open Science platforms to foster regional and continental integration through collaboration. Speaking on the data marketplace, he demonstrated a commitment to actively engage in this agenda in a way that benefits the community and improves the lives of the people he serves. During the opening session we were also treated to a first-rate marimba band, an operatic rendition of the national anthem, and an infectious 20+ strong traditional dance troupe that had many of us dancing in our seats (you know who you are!). This was the first indication that we were going to be taught how to do things by the Africans.

A small group of us were lucky enough to do some sightseeing before the conference. During a city tour with Andy of Heritage Adventures we learned about the history of Botswana’s independence and the importance of peacefulness and collaboration which shone through at the conference. Botswana gained independence in June 1966, since when it has been one of the world’s fastest growing economies, averaging about 5% per annum over the past decade. There is a strong sense of community and collective action. The development of the University of Botswana campus in Gaborone is a case in point. Andy explained about the Botswana University Campus Appeal (BUCA) or “One Man, One Beast” campaign in which the people of Botswana made donations of cattle, chicken, eggs and grain to raise funds to establish a University. Launched by the President in March 1976 with an ambitious target of one million Rand, by mid-October the same year, over R800,000 had been raised and 2,700 cattle had been donated or pledged. This is commemorated on campus with a statue of a farmer driving an Ox. Limited resources need not limit achievement.

Attending the CODATA General Assembly as the UK delegate was also an eye-opening experience. I learned more about the CODATA membership model there and that it is evolving these categories and planning a recruitment drive to grow membership and ensure sustainability. Having spent the last six months considering similar issues for the Research Data Alliance to help formulate its RDA Regions proposal, I think closer collaboration is necessary. Barend Mons was elected as the new CODATA President in Gaborone. Although he will now step down from GO FAIR activities to focus on the new role, his position will inevitably raise the profile of GO FAIR in such discussions. Government investment in the Netherlands, Germany and France has led to the establishment of the GO FAIR International Support & Coordination Office (GFISCO), with each of the founding countries having its own international office to guide new and existing Implementation Networks. National budgets and investment in data is always tight and countries will undoubtedly question what value they derive from each membership. Closer cooperation and collaboration between RDA, CODATA and GO FAIR is needed to avoid competing for the same funds and pursuing overlapping agendas. It is important to define the role of each initiative and how they complement one another. We should learn from the collaborative approach of the Botswanans here.

The Botswanan flag is representative of its national values. The blue represents water, and the white-black-white bands depict the racial harmony of the people. Botswana is the continent’s longest continuous multi-party democracy, has a good human rights record and is ranked as the least corrupt country in Africa. On the journey to the CODATA General Assembly during my last day in Botswana, Joseph Wafula from Kenya spoke about how he admired the decision-making approach in the country. Leaders cannot just pass new legislation. Proposals need to be taken to the people and debated in ordinary households. This may slow down the process, but it leads to more robust and implementable policy, and as he observed, politicians shouldn’t be scared of having their ideas debated – if they are strong ideas they will be passed. It made me ashamed to think of the diversionary news tactics that are often implemented in more ‘developed’ countries to pass new legislation by the back door when people are preoccupied with other issues.

What continues to impress me about RDA is the ground-up approach to its work. This does lead to some duplication and certain groups using the forum as a publicity platform, but the equity of voice that allows any member to bring forward ideas and convene groups is an important ideal to protect. Developing solutions in an international forum not only strengthens the outputs as they are enriched by multiple perspectives, but the ideas gain validity and have increased adoption. It also helps to challenge the ‘not-invented-here’ mentality which can often lead to reinvention of wheels and misuse of public funds. Genuine collaboration, inclusivity and diversity is hard work and can be intensely frustrating, but in most cases those cross-culture and cross-domain activities are a lot of fun, enrich your own understanding, and are ultimately worth it in the end.

“If you want to go fast, go alone. If you want to go far, go together.”   African proverb

 

 

Roadmap back to school edition

Image: CC-BY-NC-ND ‘Pencils‘ by Vanessa Lynn
 
Summer activities and latest (major 2.0.0) release
The DMPRoadmap team is checking in with an overdue update after rotating holidays and work travels over the past few months. We also experienced some core team staff transitions and began juggling some parallel projects. As a result we haven’t been following a regular development schedule, but we have been busy tidying up the codebase and documentation. 
 
This post summarizes the contents of the major release and provides instructions for those with existing installations who will need to make some configuration changes in order to upgrade to the latest and greatest DMPRoadmap code. In addition to infrastructure improvements, we fixed some bugs and completed some feature enhancements. We appreciate the feedback and encourage you to keep it coming since this helps us set priorities (listed on the development roadmap) and meet the data management planning needs of our increasingly international user community. On that note, we welcome Japan (National Institute for Informatics) and South Africa (NeDICC) as additional voices in the DMP conversation!
 
Read on for more details about all the great things packed into the latest release, as well as some general updates about our services and of course machine-actionable DMPs. The DCC has already pushed the release out to its services and the DMPTool will be upgrading soon – separate communications to follow. Those who run their own instances should check out the full release notes and a video tutorial on the validations and data clean-up (thanks Gavin!) to complete the upgrade.
 
DMPRoadmap housekeeping work (full release notes, highlights below)
  • Instructions for existing installations to upgrade to the latest release. Please read and follow these carefully to prevent any issues arising from invalid data. We highly recommend that you backup your existing database before running through these steps to prepare your system for Roadmap 2.0.0!
  • Added a full suite of automated unit tests to make it easier to incorporate external contributions and improve overall reliability.
  • Added data validations for improved data integrity.
  • Created new and revised existing documentation for coding conventions, tests, translations, etc (Github wiki). We can now update existing translations and add new ones more efficiently.
DMPRoadmap new features and bug fixes
  • Comments are now visible by default without having to click ‘Show.’ Stay tuned for additional improvements to the plan comments functionality in upcoming sprints. 
  • Renamed/standardized text labels for ‘Save’ buttons for clarity.
  • Added a button to download a list of org users as a csv file (Admin > ‘Users’ page)
  • Added a global usage report for total users and plans for all orgs (Admin > ‘Usage’ page)
  • Admins can create customized template sections and place them at the beginning or end of funder templates via drag-and-drop 
  • Removed multi-select box as an answer format and replaced with multiple choice
DCC/DMPonline subscriptions [Please note: this does not apply to DMPTool users]
Another recent change is in the DMPonline service delivery model. The DCC has been running DMP services for overseas clients for several years and is now transitioning the core DMPonline tool to a subscription model based on administrator access to the tool. The core functionality (developing, sharing and publishing DMPs) remains freely accessible to all, as well as the templates, guidance and user manuals we offer. We also remain committed to the Open Source DMPRoadmap codebase. The charges cover the support infrastructure necessary to run a production-level international service. More information is available for our users in a recent announcement. We’re also growing the support team to keep up with the requests we’re receiving. If you are interested in being at the cutting edge of DMP services and engaging with the international community to define future directions, apply to join us!
 
Machine-actionable DMPs
Increasing the opportunities for machine-actionability of DMPs was one of the spurs behind the DMPRoadmap collaboration. Facilities already exist via use of a number of standard identifiers and we’re moving on both the standards development tracks and code development and testing.
 
The CDL has been prototyping for the NSF EAGER grant and started a blog series focused on this work (#1, #2, next installation forthcoming), with an eye to seeding conversations and sharing experiences as many of us begin to experiment in multiple directions. CDL prototyping efforts are separate from the DMPRoadmap project currently but will inform future enhancements.
 

We’re also attempting to inventory global activities and projects on https://activedmps.org Some updates for this page are in the works to highlight new requirements and tools. Please add any other updates you’re aware of! Sarah ran a workshop in South Africa in August on behalf of NeDICC to gather requirements for machine-actionable DMPs there and the DCC will be hosting a visit from DIRISA in December. All the content from the workshop is on Zenodo and you can see how engaged the audience got in mapping our solutions. The DCC is also presenting on recent trends in DMPs as part of the OpenAIRE and FOSTER webinar series for Open Access week 2018. The talk maps out the current and emerging tools from a European perspective. Check out the slides and video

Image: CC-BY ‘Active DMPs in South Africa‘ by Sarah Jones
 
You can also check out the preprint and/or stop by the poster for ‘Ten Principles for Machine-Actionable DMPs’ at Force2018 in Montreal and the RDA plenary in Botswana. This work presents 10 community-generated principles to put machine-actionable DMPs into practice and realize their benefits. The principles describe specific actions that various stakeholders are already undertaking or should take.
 
We encourage everyone to contribute to the session for the DMP Common Standards working group at the next RDA plenary (Nov 5-8 in Botswana). There is community
consensus that interoperability and delivery of DMP information across systems requires a common data model; this group aims to deliver a framework for this essential first step in actualizing machine-actionable DMPs.