The Braindump Blog

Exciting news about the next version of R’s dplyr library! v1.1 will introduce non-equi joins, as well as rolling and overlap joins. The lack of these was a big frustration to me when I first started with it.

So in the future we’ll be able to do things like:

customers |>
left_join(sales_data, join_by(id, sale_date >= promotion_start))

This is a great reference, giving guidance and citations around common statistical myths. For example, what actually is a p value? What’s wrong with analysing “change from baseline”? etc.

An awesome resource when pushing back against pressure to do subtopimal statistical things

The 2021 England and Wales census results are out now. Most headlines are focussing on the fact that this is the first time that less than 50% of our country now identifies as Christian. It’s dropped 13 percentage points vs the 2011 census and now sits at 46%.

Some are finding this particularly noteworthy in a country where the head of state - King Charles - is also the head of the Church of England, officially the “Defender of the Faith”. 26 Church of England bishops hold power in the House of Lords - the so-called “Lords Spiritual”. In theory state schools are required to hold daily acts of Christian worship, although many in reality do not and suffer no consequences.

The vast majority of the difference is explained by an increasing number of people declaring that they do not follow any religion. That’s increased by 12 percentage points compared to 10 years ago and currently sits at 37% of the country; around 22 million people.

In terms of percentage growth rates, some of the smaller “religions” top the leaderboard - although of course there’s a lot more room to grow by incredible sounding rates when you start from a very small number. Devotees of Shamanism have increased from 650 people ten years ago to 8,000 now. Pagans increased from 57,000 to 74,000, and there are now 13,000 folk identifying as Wiccan.

In the excoriating misery that is the “Elon Musk bought Twitter” discourse - an institution which has disproven any sense we had between 2016 and 2021 that it couldn’t really get much worse - there have been a couple of brighter spots I’ve appreciated.

Firstly I often ponder about the power of defaults. I’d wonder about what would happen if society ever decided to reverse some of the more ingrained ones. For example, what if if each month the norm was that you actively opted in to your job if you were happy to keep it, rather than the more conventional awkward and emotional opt out scenario.

Another big one is my eternal curiosity about what would happen if the threat of immediate poverty wasn’t there. That might reduce the incentive to stay with your current employer, however unsatisfactory they are, purely on the basis that they’re slightly less dreadful than the prospective life of penury might be.

Happily Elon managed to address both at the same time. He emailed all Twitter employees to let them know that they had to click on a link if they wanted to keep their job; very much an opt-in workflow. Let’s hope that by some magical force everyone read and understood the email by the deadline. Also this is likely illegal in many countries Twitter has employees in, but, you know, Elon.

He also seemed to offer 3 months pay, no questions asked, for those that didn’t want to opt in. Additionally, this is a company where many folk were likely well-paid enough to have some savings (citation needed I accept). Those who held any Twitter stock might in fact be richer than they’d ever been before given how much above market value Elon was forced to buy the company for.

I should note that it’s not all upside; I dare say many of the staff weren’t in a particularly privileged position - the company did not survive on elite coders alone. I also felt especially sorry for people on the kind of foreign visas that require you to keep working or be deported, not least because he’d fired half the company even before he got to this point.

The consequence? Well, it seems like thousands of employees - more than half of Twitter’s entire staff - decided to take the offer and quit. Whole departments in some cases.

Of course this particular situation is very confounded by having the knowledge that if you choose to stay then you do so in the knowledge that your new boss will be something of an egomaniac madman who not only goes to sleep next to an unsettling display of mostly Americana each night but made it very clear that you will have to sacrifice all other aspects of your life so you can do ‘extremely hardcore’ work and be subjected to his fire-at-will tantrums. So I suppose to answer my underlying questions I’d really need to wait for a more natural natural experiment.

The other entertaining thing was when Elon chose to make one of the most consequential and controversial social media moderation decisions of the recent era - whether to allow Donald Trump back in - via, of course, an unconstrained Twitter poll. It gave him the result I imagine he wanted, albeit only by the same kind of margin as the Brexit referendum ended up having - many of us Britons cannot see the numbers 52 and 48 next to each other without a slight flash of anxiety.

He then reinstated the Trump account together with a declaration in Latin - “Vox Populi, Vox Dei” that in its original context means the virtually the opposite of what I imagine he was aiming to project. Translating to something like “The voice of the people is the voice of God”, the earliest known usage of that fragment was around the year 800 in a sentence which in fact reads as “…those people should not be listened to who keep saying the voice of the people is the voice of God, since the riotousness of the crowd is always close to insanity”.

On the subject of “close to insanity”, in any case Trump was all ‘no thanks, not interested’. I first assumed this was because he was either in a huff that he was subject to the whims of another dangerous weird billionaire, or too busy coming up with some semi-illegal scheme to keep on reducing the voteshare that the US Republicans get by virtue of being their presidential nominee.

But now I understand that he’s just literally not allowed to tweet in his usual style as part of his contract with his current maelstrom of a social network - possibly the most incorrectly named site on the internet - Truth Social. Not to say he won’t break the contract; he has of course plenty of form conducting unethical and likely illegal business practices.

Ok, that’s more than enough on that subject. Honestly I have wasted a lot more time reading about Musk and Twitter in recent times than I should have. But then again I have heard this article is good…must resist, must resist.

Nerd accomplishment: made my first R package today. Super rough around the edges but it works. So far it contains a whopping 1 function I hope will be useful to my team.

It was much easier than I imagined. Hilary Parker’s extremely readable guide got me half the way there.

Finished playing The Legend of Zelda: Breath of the Wild 🎮.

This game created something of a sensation when it came out 6 years ago. I’d no doubt it was good but I’d understood it to be a time consuming open-world adventure so figured it wouldn’t work well for my typical adult life 20 minute-ish gaming sessions. However it turns out that I did find it extremely worthy of all the game of the year awards it won so I’m very glad I gave it a go. It is in fact really great, and you can definitely play it in shortish bursts if you have to (and have the willpower to turn it off).

It tells the story of Link, who awakens with no memory of the past but is nonetheless thrust on a mission to save the land of Hyrule from the evil forces of Ganon.

My fear of its open-world nature meaning that typically I’d find myself wandering around lost for ages not knowing what to do was entirely alleviated by the facts that both 1) the game makes it pretty obvious what your potential next steps are, and 2) the world is so delightful, in how it looks and how it works, that for the first time I was very happy to wander around kinda lost, dealing with the strange little side quests that in most games I’d feel compelled to ignore just to get the main mission done.

The so-called physics and chemistry engine is such that there’s a real sense of freedom. You can do almost anything you think you might be able to do within the scope of the game. Metal swords are good for stabbing monsters, sure. But they can also create a spark which, if you do it with some flint near a pile of wood, will start a fire and let you cook some food. The updraft from said heat source lets you float upwards should you have the right equipment. You could also use the fire to light an arrow and fire it to ignite any dry grass around where monsters are having a nap consume them in a fiery death. There’s probably 100 other things you could do like that I never discovered.

Even watching someone else play it for a while gave me a pleasant sense of the kind of feeling one gets when travelling to a new place and exploring in real life. This was most welcome at the time, as it was the heart of the Covid-19 lockdown when actually travelling anywhere wasn’t allowed.

And as you can see from the screenshot, it’s a mostly pretty, nature-based place, which made me wonder if there’s any research out there about using computer games to provide a mild version of the benefits humans tend to get with actual exploration in the natural world. Whilst this might sound a bit implausible, and it certainly won’t provide the substantial benefits of the physical activity that’s typically associated with exploring the outdoors, there are supposedly health benefits that can be realised via some video games. That’s a topic I’d like to explore sometimes.

One tip that I wished I’d have known before playing (for compulsive in-game hoarders such as myself) is that you really don’t need to keep most of the things you find. You don’t need to collect and upgrade everything to win. If you follow the main storyline then the end battle isn’t actually super hard, and I’m really not that good at videogames - I never really mastered the combat in this one.

But by the end I was constantly running out of inventory space. So feel free to sell most of the “consumable” things that you discover, even some of the valuable ore, to make enough money to buy the stuff in the shops that will make life less frustrating, some of which is basically essential. You can almost always find more of whatever consumables you run out of. Of course if you’re a pro trying to collect and upgrade 100% of everything then this may be terrible advice.

Listening to Hold The Girl by Rina Sawayama 🎶.

Whilst the album’s approach was purportedly inspired by Taylor Swift’s Folklore, perhaps performing some kind of therapeutic function for Rina, perhaps what most stands out about it is the sheer variety of genres that the songs encompass.

Wikipedia currently lists it as being some combination of:

1990s alternative rock, pop rock, soft rock, Europop, trance, industrial, country pop, hi-NRG, pop punk, Eurotrance, stadium rock, Britpop, disco, R&B, hyperpop, J-pop, house, Eurodance, electronic, UK garage, techno, folk and psychedelic music.

In any case, there’s got to be something for most people on this album. Here’s one I liked.

The rule of thumb that, in exchange for having expensive seeming prices and usage restrictions on its devices, Apple was less likely to creepily surveil and store your personal behavioural data on its servers for undisclosed and unknowable reasons seems to be invalid now.

Just a few years ago we saw Apple’s “What happens on your iPhone, stays on your iPhone” marketing slogan. More recently though researchers have shown that, even when you disable the sharing of device analytics, the App Store and many of the other built-in iPhone apps send tracking info back to apple.

…including what you tapped on, which apps you search for, what ads you saw, and how long you looked at a given app and how you found it. The app sent details about you and your device as well, including ID numbers, what kind of phone you’re using, your screen resolution, your keyboard languages, how you’re connected to the internet…

…your list of watched stocks, the names stocks you viewed or searched for and time stamps for when you did it, as well as a record of any news articles you see in the app…

I guess if the future of Apple does end up involving showing scammy looking adverts promoting gambling wherever they can slot them in then this is the kind of thing we would expect.

Anyone who occasionally has to reference journal papers etc. may find the DOI Citation Formatter a very useful tool. Give it the DOI of the paper in question, pick your formatting style from a truly vast array of institutional options, and out pops a nicely formatted reference.

I learned a lot about eating poo this week. Apparently it’s well-known that rabbits eat their own faeces - or at least the caecotroph variant of droppings. The nutrients it still contains are good for their health. But it wasn’t until recently that we found a species that eats a good amount of any other species' excrement.

A genetic analysis shows that foxes eat dog poo, particularly in environments where there’s not much else out there to eat. Unsurprisingly poop is much easier to hunt than wild prey. Perhaps more surprisingly it has a similar calorific content.

It does potentially add to the risk of disease, including of new pathogens appearing - so I doubt “I’m feeding the foxes” can be considered a valid excuse for not being bothered to pick up any dog faeces you may be responsible for.

There’s a common idea that you can translate the age of a dog to the equivalent age of a human by multiplying by 7. It’s easily disprovable at a basic level insomuch as different breeds of dogs tend to live for different numbers of years . I chanced upon a preprint that also suggests the idea of a linear “human years multiplied by X” isn’t right either.

Something more akin to reality turns out to involve slightly more complicated math. The researchers based their results on Labrador retrievers as the reference dog, and determined via comparing methylomes between species that the relationship between humans and dog lives is better expressed as:

human age = 16ln(dog age) + 31

The ln there represents taking the natural log of a number, which you can calculate here.

One advantage of this non-linear view is that it allows certain physiological milestones line up much better - for example lining up the typical time puppies and humans develop teeth, as well as correspondingly similar average overall lifespans.

Finished reading All Our Relations by Tanya Talaga 📚.

Starting out by focussing on a suicide epidemic amongst young Canadian Indigenous people, this book chronicles the historic, and unfortunately oftentimes ongoing, injustices that have been meted out on Indigenous people ever since the usually white European colonists ‘discovered’ the ‘New World’.

From a past of overtly eliminating a people they regarded as inferior, through to more recent decades which included a ‘residential schools’ policy forcibly assimilating any remaining Indigenous children, the author argues that the now-dominant society has caused physical, mental, cultural and spiritual crises amongst entire communities of Indigenous people.

A few success stories are included showing that there may be hope when these communities are able to develop or collaborate on culturally and practically relevant solutions to situations so bad that poor health, poverty, addiction, isolation, despair and even suicide is commonplace.

The focus seems mainly on the Canadian experience, but several examples from elsewhere are provided. Upsettingly, they lead towards the conclusion that the offences committed against, and resulting challenges faced by, Indigenous people around the world are quite similar.

Unflatpacked some garden furniture today, just in time for it to be far too cold outside to actually use.

I’m somewhat shocked to feel this way given the source, but I don’t hate all the policies from the Conservative’s autumn statement announced yesterday. At least not in the short term.

It’s amazing how much changes within a few weeks. Whilst some of the policies under the Truss administration weren’t nearly as tax-cutting as it seemed on the surface, they were extremely regressive. Just last month we were hearing that the top rate of income tax was going to be entirely abolished - free money for the rich! And now under the Sunak administration it’s actually going to be applied to more people; anyone with an income of over around £125k as opposed to the current £150k.

Benefits are going up inline with inflation, we’re seeing an increase in capital gains tax and a fiscal drag on inheritance tax. And a windfall tax on the incredible profits of energy producers. All very un-Conservative-stereotype.

Overall it’s still grim news in the economy - basically everyone is going to be worse off in the near feature. Living standards are set to fall by an unprecedented 7%. Plenty of this is due to the abject mismanagement of the economy by the Conservative party in recent times. Unemployment, inflation and energy bills are all set to rise. If you’re already struggling that’s obviously a much huger burden than if you’re doing well enough right now.

The statement’s policies are not nearly enough to make life particularly bearable for anyone already in or near crisis. But, I guess, credit where credit’s due - the tax changes announced yesterday are actually somewhat progressive.

Here’s a reproduction of the impact calculated by the Resolution Foundation provided by the Guardian.

Where the statement is rather more disingenuous is with regards to cuts in public funding. Cuts to public services tend to impact the poorest in society more than the rich. Cuts are certainly in the statement, and they’re big (and dangerous). But the bulk of them are scheduled not to hit until 2024-2025. Of course by then there’s a good chance we’ll have had a general election. A cynic might suggest it’s a cursed gift from a party that doesn’t really believe it’ll be in power at that point.

Watched season 12 - the final season - of The Big Bang Theory 📺.

It’s mostly more of the same, so if you found the (post-introduction of female scientists) stereotypes problematic then you still will. If you found the geek references, characters or general style annoying or stale then not all that much changed. But if you took pleasure from the fairly predictable series and its nerdy references then it’s worth finishing it off.

Much of the ending is comfortably heart-warming, although I do feel sympathy for Kathryn VanArendonk’s frustration with one of the storylines (warning: the link containers spoilers, albeit for a 3 year old episode).

Listening to the If Books Could Kill podcast 🎙️.

Peter Shamshiri (of 5-4 cohost fame) and Michael Hobbes (from Maintenance Phase) discuss and debunk ‘airport bestseller’ books that they believe have ruined our minds.

So far they’ve covered Freakonomics and Outliers, two books that in all honesty I remember being extremely into in my youth, in a pop-sci way. The former felt even somewhat formative in terms of my future interests.

This was at a point in time I was blissfully unaware of study design shenanigans, the replication crisis and the various other practices that can infect the scientific record. Here’s hoping not too much of my worldview is built on lies.

The British Museum, London, inside and out.

The wellbeing agenda, with its focus on milder problems, can lead to great statistics in a way that doesn’t work for severe mental illness

This is a sentiment I feel I’ve seen from quite a few articles recently.

Not just in terms of statistics, but a more general idea that we may finally be becoming more understanding and supportive of people with more common, typically less severe, mental health issues such as mild anxiety or depression - something that is undoubtedly good and necessary. But that there’s a way in which it’s been somewhat at the expense of folk with extremely serious and debilitating conditions.

I don’t yet have a sense of to what extent this is a real phenomenon or not. But it does seem like it’s an idea that’s in the ether at present.

Some context as Hunt prepares to unleash horrendous spending cuts and tax increases on us: Liz Truss' catastrophic failure of a 7 week premiership is estimated to have cost the UK £30 billion.

£20bn was blown…on unfunded cuts to national insurance and stamp duty, with a further £10bn added by higher interest rates and government borrowing costs as the markets reacted with dismay.

A recent paper quantifies the economic inequities exacerbated by the way that credit cards that offer rewards to their users work.

Of course the cost of these rewards has to come from somewhere.

This includes the various fees and interest charged by the credit card companies to its users. Unsurprisingly, people with low credit scores are more likely to end up paying these fees.

Now whilst credit rating doesn’t correlate 1:1 with e.g. income - so as the paper says the net effect isn’t a pure “take money from the poor to give to the rich” money grab - it does predictably correlate with various other dimensions of potential privilege. It’s obviously a lot easier to pay off your credit card balance and not incur interest charges if you have a big pile of existing money at your disposal.

Credit card companies also make money by charging shops every time a customer uses their credit card. To the extent that the need to fund rewards results in the credit card companies increasing the merchant fees to retailers, who correspondingly increase their products' prices to compensate, that’s also another way that poorer folk may be penalised - a small % increase in cost may not be noticed by the wealthy, but for people in poverty it might be that every penny counts.

Retailers don’t usually charge different prices for customers based on which card they use to purchase something, and so those people who don’t choose to use or do not have access to rewards credit cards may end up paying for other people to get rewards.

We estimate an aggregate annual redistribution of $15 billion from less to more educated, poorer to richer, and high to low minority areas, widening existing spatial disparities.

Listening to Midnights, by Taylor Swift 🎶.

In possibly the coldest take available on the internet today, Taylor Swift’s new album is really good. It’s broken a few records in terms of sales and popularity.

When I first listened to it I felt like all the bassy synths gave it a kind of ominous feel that I didn’t need in 2022, but a couple of listens later and I fully appreciate it. The general conceit is that each song reflects a sleepless night of her life, and sure enough some of the included angst may be familiar to those of us who are not multi-millionaire musical megastars too.

Apparently the album is filled with secret messages for the internet to discover. I came to Swift far too late and probably lackadaisically to get any of them but they’re there for those more informed.

Here’s one of the highlights, with a fun spooky (albeit originally controversial) video.

For anyone thinking to join the relative masses in signing up to Mastodon in protest to Elon Musk’s shenanigans now he owns Twitter, Danielle Navarro’s post looks to be a great intro guide. Particularly for anyone who’s in the “data community”, but most of it is applicable to the platform as a whole.

Whilst I’ve not yet explored it myself, Mastodon is actually one way you can subscribe to this blog - follow @adam@braindump.amedcalf.com if that’s something you’d like to do!

Things like this are exactly what cause me angst about joining the Labour party.

To be fair the headline “Keir Starmer: Too many people from overseas recruited to NHS” is a little selective, perhaps misleadingly so. Within the same interview Starmer also said he’d encourage immigration for different types of jobs. He also acknowledged that we have a desperate need for more NHS staff. But his proposed solution was that the UK needs to do way more in terms of increasing training opportunities and making jobs in the NHS bearable enough that any sane person isn’t so deterred by the conditions that they wouldn’t want the training in the first place.

I’m fully on board with that as a plank of the approach. It’s absolutely necessary. But it’s not an either/or situation. We have a NHS staffing crisis right now. Earlier this year it was calculated that around 110,000 NHS vacancies were unfilled. A British politician with the wellbeing of their constituents in mind should use whatever reasonable means we have to solve it.

Starmer is not known to be a naïve or stupid politician so when he said “I think we’re recruiting too many people from overseas into, for example, the health service” I’m sure he’d have understood how that might sound.

What’s particularly gruelling about this example is that reducing the immigration of medical workers isn’t even a popular policy. In the latest poll I can find, most people are actively in favour of immigration in terms of increasing the supply of doctors and nurses in the UK.

From page 15 of the British Future report on attitudes to immigration this year:

55% of respondents to this survey commissioned from IPSOS, designed to be representative of the British pouplation, would prefer recruitment of migrants as nurses to increase, with only 13% against. The figures for doctors are very similar.

Even most Conservative party supporters are in favour! Compared to Labour supporters they’re more likely to say the numbers should be reduced, but only by a few percentage points.

Of course this might not be apparent in the public discourse. Inevitably there is a certain percentage of the population who would rather see the immigration numbers go down than sustain the NHS. They’re loudly represented on social media, likely encouraged by the recent Conservative tendency to seemingly try and recreate a US-style culture war to our shores.

This is not to say that what job you do necessarily should be the basis of your ability to immigrate to the UK - I believe it shouldn’t be. But the point is that there is likely little need to “strategically” appear to be against immigration to fill NHS positions if that’s what’s going on, because most of the population actually wants to see more.

There certainly is a moral case to be made against encouraging immigration to fulfil Britain’s unmet medical needs, but it’s a very different one. As Kollar and Buyx write in their review of this phenomena:

Health-worker migration, commonly called “medical brain drain”, refers to the mass migration of trained and skilled health professionals (doctors, nurses, midwives) from low-income to high-income countries. This is currently leaving a significant number of poor countries, particularly in sub-Saharan Africa, with critical staff shortages in the healthcare sector. A broad consensus exists that, where medical brain drain exacerbates such shortages, it is unethical…

Simplistically, one foreign nurse moving here is potentially one less nurse serving the country they were originally in. If done en masse, probably some recompense, ideally of a structural nature, should be given. However, this debate doesn’t seem to be the one that interests the politicians, media or general public at present.

Witnessing the Lincoln Chorale, accompanied by the Lindum Baroque Orchestra, honouring the king with their voices.

Featuring such classics as Zadok the Priest. I had no idea that was the title of this evergreen number.

Recently learned that I’m just one degree of separation from one of those accidental Bitcoin millionaires.

A acquaintance of a friend mined BTC to a wallet stored on a SD card years ago. After recently re-finding the card, he’s now got a new house with a few million $ left over.