There were a lot of really good things about doing this project the way it was done, that were specific to the situation we were in (pandemic, no access to a particular commercial tool, and the short time frame requiring a certain kind of risk management – i.e. don’t hide data in a non-human format or store it in a fragile place, so we don’t put student or staff work at risk if the tool breaks unexpectedly). But it did take a dedicated team of three of us to process the exams – often I was coding and testing features until 3:00, 4:00am or later, then they were going straight into production with one of our team using them from 7:00am or so. That was fun, but hard work.

Looking back, I still can’t quite believe the quantity of features, or level of detail that things got to, with this project … perhaps with the lack of sleep I wasn’t forming memories too well … but if a thing needs fixing, it needs fixing properly, so you can sleep at night (even if that is just for a an hour or two before you are bug-fixing your latest release), and that means handling detail.

One of the least rewarding aspects of this project was that the risk management strategy forced us to upload and download a lot of redundant data (all the images in the PDF exam copies had to go with the document everywhere, even if the new data added by a marker was just a few hundreds of text characters), and to add to our pain, we had decided to use a typical corporate data store that was at times, how shall we say, vexing and tiresome. There were also the inevitable human behaviour challenges – ideally, users shouldn’t need to read instructions (it’s a nice ideal, at least), nor should they be able to mess things up when they are trying to be a good actor. But relying on an eco-system like PDF, across a lot of devices, with a lot of patchy implementations of the standard was probably the thing that the biggest issue for me. With PDF there is no way you can script in the level of interactive guidance that you can with javascript on the web, and you can’t control what software a user wants to use. And that can be a major issue ….

For example, when a user misses instructions not to do so (not really their fault, in my view), and uses the inbuilt software from a major premium brand of device supplier, which does not implement the spec correctly, and breaks your PDF parsing ability to the point it cannot actually understand the file (messing with the PDF catalogue is problematic for obvious reasons), then you have to get someone else to re-key the marking. At this point, you know you are at the limits of your ability to improve the user experience without moving away from PDF. When you are at the limit of the ecosystem like that, it’s not like more coding from you will help. Nor will tech testing, really, because how do you communicate all that to the users? It’s not like you can check the user agent and throw up some guidance on a better browser with the features you need, as you can do with the web.

So for this project, the issues of the ecosystem reliability, lack of control of user experience, and data overhead, mean it is now time to find a way to put on a web frontend and an online backend. That’s definitely possible but … the question is – would you really want to develop a whole new tool to do that when you can just buy one? Which is where we started – we had something we wanted to buy but couldn’t get it.

Well, more than half a year later, we’ve finally managed to get our mitts on the commercial tool we originally wanted, and others are now taking on the process of integrating it into our procedures. I’ve stepped back from that process because I want to get back to my first love, remote labs. Which I owe a whole shed load of long-overdue development time. Thanks to a great team of people at my current place, I now have a lab again for the first time in about two years to underpin that development work (There is more information on this project coming soon, to a different blog site which I’ll link here when it is up).

What does that mean for gradeX®? Well, astute Readers will have noticed the name is now registered. If you buy me a coffee post pandemic I’ll explain one of the other motivations for doing this, but mainly, it was a symbolic move to show the people that worked with me on this that what we did has laid the groundwork for some future goodness, which will remain true to the open-source, development-in-public approach of the project so far. For now, I think that future goodness needs to be something that adds to the tableau of marking codes out there, rather than duplicates (e.g. trying to compete with existing tools to do the same job). So, novel marking practices that are not adequately supported by existing tools are all potential candidates, with a possible aim being to support the piloting of a bunch of things related to authentic assessment and scale them up quickly if they are promising. It might be something to support some twist on evaluative judgment development, peer interaction, reflection or some other idea. I’m expecting those ideas to emerge from ongoing curriculum development efforts, and we’ll know when it is time to work on the software to support them. Meanwhile, if you find the unique combination of features in pdf.gradex to be useful, then remember, it is open-source, and remains available to use in the form that got us through >39,000 pages, although I won’t be available to hold your hand (unless you can make me an offer I can’t refuse, and chances are, you could get a commercial product for that sort of money). See the repo here https://github.com/timdrysdale/gradex-cli. Enjoy! Until the next update ….

I’ve learned plenty about our behind-the-scenes exam processes and the limitations of PDF – this was definitely the right solution for us, for the emergency situation we were in, despite the immense amount of work to create it. It de-risked the process by keeping everyone’s work safe from edit conflicts between students and staff, and between staff members assessing it, as well as being backwards-compatible with the old Mk1 Human so we could revert to the pain of a fully manual process at any time if we hit insurmountable problems (which we didn’t in the end).

Working with the PDF ecosystem reminded me why I prefer browser-based solutions – you can interact with the user and keep them on track.

Me

An online electronic marking system is the way to go for us in the future. That’s possible for us now that we’ve taken that first step. Using a digital process that mapped exactly onto our existing workflow avoided a possible barrier to adoption of a new approach. Now our staff have seen the benefit of a digital workflow, we can have a chat about adopting a fully online system – especially since colleagues ran limited trials with other systems (interestingly, using this system to pre-process files before feeding them into the competition). The issues I ran into with the PDF ecosystem itself mean I don’t think a better user experience overall is possible no matter how good the underlying library that you use to work with PDF – you are still at the mercy of the interpretation of the specification for PDF by the viewing tools used by the staffs. We topped out on the quality of that experience, so time to move back to my favourite domain – the browser.

So, as you might guess, eyeing up the browser takes us smack bang into the territory already served by some existing tools. To compete or not to compete? This was not an easy decision because there is a really strong ethical benefit to operating assessment software from an open-source academic team. It provides an opportunity to fund the development from teaching income, and avoid the allure of monetising the student data. By the way, the reason that is even an issue for the edtech sector is that if you have shareholders, you have a legal obligation to do right by them, and that makes it rather difficult to leave money on the table (money you might need to stay in business, to grow, or to make a suitable exit for the founders). We can get around that by using a different funding model.

On the other hand, is it worth the effort to develop a tool which doesn’t monetise student data, but still (only) achieves the same (limited) academic outcomes? And how do you feel about that when you step back and realise you’d really like to see a bunch of change in the way things are done in that educational sector? Is that a reason to aim for a seat at the table now, just so you can try and make change later? Or would that be so much effort you’d never get to the transformative stuff? And is the context in which you’d be operating sufficiently flexible that you can slot the transformative stuff into whatever you are doing, or might it not fit?

There’s a really excellent discussion of the difference between migration online, and transformation online, in assessment systems here – an article suggested to me by Jen Ross, during a conversation we were having with Tim Fawns, who also added great stuff to my thinking (which I am not re-iterating here simply because it needs a proper writing out!)

There is no bald statement of the right path in that paper, but there is certainly a strong indication of the risk of getting bogged down in migration instead of doing transformation. What sort of development journey will evaluative judgement, contract grading, accept-revise, student-in-control grading, and various other practices need in order to flourish – and what role will digital assessment play – if any? Can you make a logical sequence of technical development moves to get from online “commodity” grading to the transformative stuff? Or are they paths that do not cross? In which case, what happens if you try to start with migration? What if you think there might be a way to do all of this without any traditional grading at all? (Health warning: people of a traditional disposition, look away now. Too late? Did I already say that? Sorry not sorry). Ok that’s clearly an extreme provocation. But it is a thought process I went through to offset the dangers of sunk-cost fallacy and train-track thinking (i.e. trying to avoid the pitfall of “I’m doing this, so I shall do more of this”).

How to resolve this in the case of future tools in the gradex ecosystem? Since academics who care about teaching, care about improving the experience for students, it seems a greater good can be served by focusing immediately on new practices that lead to better experiences for students and putting up with short term ethical concerns around any usage of existing commercial tools that fulfill a need. So gradeX development work is going to go on hold, while I bury myself in my keyboard again to deliver an open-source remote laboratory infrastructure over the coming months. I’ve a few other things up my sleeve as well … but more about those in good time!

Feel free to drop me a line for a copy of the doc itself!

admin@gradex.io

https://www.youtube.com/watch?v=jIs0a4yU2tU

This post is actually about two different types of comments …

• feedback from a colleague
• how we can use/handle pop-up comments

The post title reflects me realising that I wanted to support pop-up comments in the pdf-handling bit of the gradex tool…

1. Feedback on the marking process

Overall, I think it’s cool. As long as we can get students to upload stuff that is readable I think it should be possible to mark at pretty much the same speed as normal.

An Anonymous Colleague

Awesome, I am pleased with that. Avoiding time wasted with the tool is one of the main goals – use effort for marking, not marking up.

And here is one that should be suitable to follow through as a worked example for the web page.
Page 1 – normal marked page, one comment and addition should be correct
Page 2 – marked page but should have an adding up error (to show how this will be moderated)
Page 3 – I’ve checked the box at the top to indicate that I can’t read it (assuming that is what the box is for)

.. The Same Colleague

Here’s a look at their marked-up file (images-only )

This is a great example of how marking will get done – careful consideration, some half marks, a simulated adding mistake (it happens in real life from time to time, so we have processes to catch it) and a duff page getting flagged (that page wasn’t duff, but we’re using our imagination).

2. Handle those pesky virtual post-it comments

Seeing the use of pop-up comments in this marking demo, made me realise I wanted to handle them in the gradex tool. They reduce your cognitive load (you can park some parts of your thoughts on paper to focus on other bits). It’s unlikely that students will use comments on their scanned exam scripts, but they might. Markers definitely will. For marking in PDF, comments do produce some difficulties though:

• PDF readers don’t respect read-only settings
• Ghostscript doesn’t flatten them to the page image like the other annotations (see images above!)

That means I need to handle them myself, and flatten them in some way, so that they cannot be edited in a rogue non-compliant editor.

It just so happens that both comments in this marked document popped open to cover over parts of the marking area where there could be important information. So I can’t just flatten them where the document says to put them, because they would permanently obscure important information. I thought through a bunch of options

• stashing the comment text within the document, then duplicating the comment in a new pop-up
• putting comments in special part of the sidebar*
• putting comments in a special header or footer
• finding out which editors respect the readOnly flag and putting the rest on a “do not use” list.

These approaches all have wrinkles or gremlins. The least worst option is to stash comments in a reserved area so that nothing is overwritten. So I have made a space to investigate putting comments in the sidebar, once I have prefill text fields implemented in a general way. For now – I’m writing them to the bottom of the page. I’m also putting in a yellow numbered marker where the comment was located, because position is important context.

These two pages (the two with comments) don’t look that different with the comments on. There is a chance of conflicting with student work at some point …. hence the plan to shift it into the sidebar if possible. Meanwhile, even if it is not as pretty as the grids, at least we’ve captured and preserved the comments. Phew.

Postscript (sorry not sorry)

The more you dig into PDF, the more you start to have thoughts about its clunkiness compared to web markup – these are well articulated by others (like this). The TLDR is that we’d do things differently now we are more worried about security more than we are about performance – and avoid structures that permit infinite cycles.

You know it’s a serious when the CEO of a software house is dropping comments like this in their code (presumably from a code review):

GH: Are we fully protected against circular references? (Add tests).

But it’s not bad for a library that’s older than the average age of the students sitting the exams. And the structure diagrams are pretty – if you aren’t thinking about parsing it!

The design is flexible so it can be changed to suit feedback or different procedures in different unis.

I learned that a lot of PDF viewers have patchy implementations, so you get a nicer experience if you go with

• Adobe Reader
• Drawboard PDF (for pen display)
• (your suggestions here)

Things that DON’T work, or have quirks

• Microsoft Teams (you have to download to your desktop and use something else)
• Chrome (can edit but not save, known chrome bug)
• Edge (can edit, but does uses small letters so not as satisfying)

Here’s an original submission, with no pdf.gradex treatment. You can delete the annotations. Oops!

If you fancy seeing what happens to your work next, send me your edited form (and let me know if I can post here ….)

That also provided the impetus to put up a quick demo release. I do my dev on linux, but it seems pretty standard across the people I’ve sent demos to (Canada, Scotland) that teaching offices are using Windows 10. You can get the demo code from github. If you want to experiment with your own layouts, then there is information on the editing process in the demo README.

Even though I know it is a good thing in projects like this, the curse of YAGNI is always in your mind when you are sinking time into automation processes with their usual little wrinkles that (hopefully) no-one else need ever be burdened with. It started to pay off today, already. I had some user feedback on the grid design, and within 10min of the call ending, I had the design revised and working. I’d split the question boxes to nudge freehand markers to spread their characters out so I could do OCR bounding box estimation better. Of course, I don’t use half marks, and had assumed no one else did either. Oops. I like it better this way, and we can handle the OCR by either (a) begging and pleading with markers to put a space between their handwritten characters, (b) not doing it, (c) giving in an using an external image service (some issues there). I estimate it would have been about an hour’s work to do the geometry change by writing coordinates into golang structs, and it would not have been done with such good alignment either (or the with the bordeline relief-smugness hybrid of knowing it wasn’t YAGNI). So thanks self-from-last-week.

Here’s a couple of the steps shown on their own, showing the moderator has only to check the page if all is ok, but can handle minor issues and communicate them back to the office. If there needed to be additional stages, we’ve got some other coloured “query” and “resolve” sidebars waiting in the wings.

Performance

Since we’re going for robustness and safety, we’re rendering all the annotations at each step, exporting to an image file, reimporting that into a new page that has the marking forms added, then exporting that again (phew!) – the stuff in blue all happens in a single command.

The process in averages around 1.4sec/page/stage at our current quality setting of 90 JPEG quality at 175dpi.

mark             1.3s/page
moderate-active  1.3s/page
check            1.4s/page

The round trip to collect scripts from our exam halls easily comes in at over 45 minutes, or enough time to process about 2,000 pages.

You can tell where the name “ladder” came from when you look at the earliest version shown to colleagues for their feedback (red for marker, green for moderator):

Then I started branching out a bit with the design “flair,” but still handling the box geometry calculations by hand. It was clear that novelty would be limited by how long it took to wield the ruler on screen to line things up and encode the coordinates, but things did start to look a little smarter:

Then I realised that these could be made a bit more self documenting in their design, if I could just put boxes where ever I wanted. It would also mean I could take feedback from colleagues without wincing about how long it would take to recalculate where the boxes go. It was like going back to my first interactive educational computer program (on Venn diagrams, written for a primary school science fair on an IBM XT). Some ~30 years later we can do better than that!

I set myself the challenge of being able to parse the SVG output from Inkscape and autogenerate the acroforms for this sample page with three different size boxes that don’t line up with each other. There’s a video of that in action here, and a picture below:

Then I got to work on some more expressive side bars, with an option to include moderation or not. Good user interfaces (even if asynchronous like this) should be self-documenting, ideally. So the vertical lines show that you might indicate the scan was not good, and not do any further marking. Or you might fill in some sub-totals so you can keep track, then tot them up, put them in the Q box, and then tick the ‘page marked’ box. That’s something we can track automagically to make sure every page gets seen.

After marking, our office might select this paper for moderation:

… or you might not .So we can make that more obvious by putting in a visually different sidebar, so we’re not wondering if the moderator left some green boxes blank by accident or not:

Either way it will be checked afterwards in the office. At this point, it is more likely the entries will be keyed in. Then we can extract the marks and make reports. This sort of approach is likely to be faster than the paper process, and can give you more confidence (e.g. run reports that make sure every page got checked).

The paper wasn’t moderated, we don’t want the blue bar hanging off in space, so we move it over to the left – that dynamic X-spacing is another reason for the X in pdf.gradex.

You can imagine further stages for resolving issues, and re-checking. What we do after that are topic(s) for a further post.

You can download the actual example here . You can edit and save using Adobe Reader, reload and keep going later. Other editors have variable support. For example, you edit and save in Edge, but not Chrome (it lets you edit but not save). If you’re using chrome, like me, feel free to try out the editing side in the browser, just download the form and use a better editor before doing any work you want to keep!

To support additional steps in the process – e.g. for moderating and checking – additional sidebars can be added with different content, designs, colours, names – examples to follow soon!

With multiple people involved in the marking process, they may have different preferred ways of working. So we can mix and match by flattening any annotations that are made on the script.

The script “freezing” is using ghostscript to render the original image in the PDF so that it bakes in any annotations that the student has made (e.g. erasing mistakes or adding extra work). This way, the marker cannot accidentally undo the electronic editing that the student did to their own work.

It’s even conceivable to mix in printing, marking and then scanning, although it will require some additional steps in the workflow to match up returning scripts with the metadata in the electronic version (QR codes are one possibility here – topic for a future post).