Advice from beyond the PhD

1. 🧭 Don’t stress but plan ahead

Questions I often received towards the end of the PhD were along the lines of "How do I ensure that my thesis has a cohesive storyline?" and "Did you plan for this story from the start?". Answering them is non-trivial: in a fast-paced field like NLP, planning three to six years ahead for a top-down approach to PhD research is nearly impossible, and many a thesis is constructed in a bottom-up manner, where the candidate bundles their papers towards the end, determining the storyline somewhat post-hoc. This works well for many, so do not worry too much if you are currently in a situation in which you can’t completely see the dissertation’s forest through your papers' trees yet. However, I personally very much enjoy the fact that my dissertation has clear, connected themes.

I believe that crucial reasons for why this is the case are (a) that I entered the PhD with a strong interest in (non-)compositionality and generalisation and didn’t stray too far from that through the years, while having a flexibility regarding projects to conduct along the way, and (b) that I planned my projects ahead of time, supported by my university’s administrative structures.

During three intermediate PhD evaluations, I planned ahead. During evaluation 1, I sketched out research directions for the next year, during evaluation 2, I discussed how the thesis’s storyline was starting to emerge, and what was missing, and during evaluation 3, I discussed a concrete thesis outline. This does not mean that I stuck exactly to what was planned (see lesson 2, below), but it was a massive help. If you’re a student, and your university/supervisor does not enforce such evaluation and planning checkpoints, be proactive and schedule them yourself, at least once a year.

For me, the right approach was not working bottom-up, not working top-down, but adopting a hybrid approach, where most meetings are low-level, bottom-up project meetings, but some take the helicopter view of the PhD as a marathon, to study whether I was on track, and was headed in the right direction.

2. 📖 Curate the thesis’s storyline

I can’t say with certainty that I did it well, but I attempted to curate a very specific storyline in the dissertation. Three things I did to achieve that, are the following:

Selecting what to include. Not everything one does during a PhD has to be included in the thesis, and even if you include all first-author papers, you can often write multiple stories using the same papers. My thesis contains two primary parts, circled around "memorisation vs generalisation" and "compositionality vs non-compositionality". Planning for the thesis writing, I originally thought I’d include a Part III — based on Dankers et al., 2022 and Dankers & Lucas, 2023 — but I followed the advice of my UoE annual review board to omit it. More isn’t always better.

Deciding where to include what. In my case (but perhaps not yours) the papers didn’t have to be included in a chronological order, and I opted for putting the most recent two first to curate a narrative in which the second two papers (on (non-)compositionality) are considered a case study of the more general memorisation–generalisation paradox.

Going beyond the original papers. Every experimental chapter of my thesis contains experiments that were not in the original paper, either because I felt something was missing that could have made the conclusions stronger, or because seeing the individual papers in the light of the overall dissertation made me realise that certain experiments could provide inter-chapter connections.

3. 🔁 Plan for reproducibility

By the time the thesis comes around you might be getting back into 4-year-old codebases, to see whether you can recreate some graphs with old data — either just because the thesis would look nicer with updated graphs, or because you actually want to extend the experiments, like I did. At that point, you might realise that not everything is reproducible.

Even with the greatest README.md out there, packages will change, clusters will change, and checkpoints you thought you had stored will no longer be there because the cluster admin decided a clean-up was in order. When you’re now only one year into the PhD, I know you’re likely not thinking of the dissertation, and you may want to move on from a project as soon as it is submitted, but please invest time into the reproducibility aspect. First and foremost because of others that might want to build upon your work, and second because you yourself may want to rerun those exact experiments. Think ahead: "Which model checkpoints would be crucial if I wanted to test a few more hypotheses for the thesis?", or "What data would I need to store if I wanted to regenerate the figures, or run an additional statistical significance test on the results?", or "Should I ask my internship manager for approval for exporting these models, since I won’t be an intern at the time of graduation?".

I wish I had curated a PhD-thesis-ready version of all of my papers on a dedicated hard drive, with the most interesting models, and the data required to regenerate tables and graphs. Although it was there for most of the chapters, some of it needed to be regenerated.

4. 🏷️ Give your dataset a name

A very silly mistake I realised I made was that in Dankers et al., 2022 I produced a compositional generalisation evaluation dataset that never received a name. Therefore, various papers referred to it under names they had come up with. The consensus was OPUS En-Nl, but we clearly should have assigned the evaluation set a name. Lesson learnt!

5. 🪞 Write a retrospective

Inspired by Voita (2024)'s sections entitled "Implications: View from the future", I wrote my own "Retrospective and outlook" sections, which were also highly recommended by my phenomenal supervisor Prof Titov. Taking my own work, and reflecting on (a) what impact this specific paper has had and (b) how the field has changed since this paper saw the light of day, was very informative for myself, and according to my thesis examiners. I can safely say they are my favourite thesis sections.