Adventuring with Crypto-Code and LLMs

tl;dr: personal thoughts on LLMs with examples.

After a long, long, long radio-silence, let me update you all with the latest news! As usual, life got busy with work, hobbies and projects! I wouldn’t say there is nothing interesting to share… …maybe the problem is that I do not have the time to write them down! 😅

But I have been playing with LLM lately, both for work and hobby, by following the (fear-mongering) news and taking a look at “what is going on”. For this reason, I think it is time for me to share some thoughts and my own experience with the tool.

This is not a “LLMs are magic” nor a “LLMs suck” rant¹. I hope it develops as a reflection “on the tool”, how it helps or breaks the normal (code-)flow.

Note: I like to use LLM as a generic term for whatever LLM I use, free or under membership. This reflects my vision that LLM is a tool.

Working with LLM : from Ideas, through Math and into Code

Working in cryptography can involve a lot of different tasks: thinking, doodling symbols, formal writing in LaTeX and sometimes implementing math into code. Often, only some of these steps are fun and will push you do give your best, while others… oh my… you only wait for them to be done!

Coming up with new ideas is really fun, IMHO. It involves reading new papers, searching for applications, looking at the wall in front of pen and paper. Connecting concepts is hard and sometimes being creative steers the research into elegant and novel solutions which, especially now, must be implementable².

Let me be concise: LLM are nowhere near in helping my idea development at work/research. Finding new solutions is less of a “solve the problem” and more a “find a clever idea” which is not what all the LLM I tried are able to do.

They power is in searching and connecting known points, i.e. translating and pointing into structured content. These are, in my experience, good skills that only appears when implementing the math. When asking LLM to write the content, the results’ quality is always a gamble and (too) often a losing one.

Looking at what other people are doing, I gave LLM a go and I treat the LLM as less than a junior developer. I didn’t ask LLM “implement X” because the result will definitely be slop. I started providing really clear tasks, split into points, provided one at a time so to guide the LLM into small, well-defined problems with minimal manoeuvring. This is/was only possible because I know exactly what I want, so my workflow looks like this:

write down the mathematical objects and operations in precise notation (for a documentation, paper or just from my notes)
design and describe the expected API: input, output, constraints and testing assessments
plan how to solve the problem, as if I will do it
ask the LLM to solve a small-problem at a time with minimal view on the next steps
review the result³, point out mistakes, ask for alternatives, refine
when good, follow up with the next point

Looks simple: LLM turns into a tireless pair-programmer that never complains and spits code at a crazy speed! But this is an intense way of working since I am the one steering the development by checking correctness, design directions and general judgement.

My lightbulb discovery was that LLMs are translators, not engineers. That means that I if I can provide clear, correct mathematical description of the algorithms, then the LLM will turn them into code a computer can actually run.

I’m impressed by how good the translation is: LLM can identify small code-representation problems relating to design-choices (e.g. endianness, serialisation, etc.) but will follow the provided math highlighting if something is incorrect, maybe in a type-check fashion.

The implementation velocity gains are bonkers: as a rule-of-thumb, I would say that I easily get a “5x” speed-up, what would take 5 working days for me to implement, turns easily into a single intense day of pair-programming.

I highlight “intense” (and avoid the footnote) because the mental load in LLM pair-programming is really sustained and it is quite difficult to have natural recovering breaks when LLM doesn’t give you the time to “digest” your directions.

Imagine having to provide instruction for a job/project like some sort of supervision to a (junior) colleague new in the office. This takes time and energy because a lot of context must be provided to understand the project, explanations are required to clarify expectations and you must provide how to validate solutions. Good, you got these instructions and your human-brain expects these tasks to take at least a couple of hours, maybe a full day to properly get a result. Your brain thinks has time to recover from this effort.

Then, the LLM takes five minutes, executes everything and you didn’t even had the time to think about the next steps. Your supervising-brain feels paralysed, in between “it must be wrong” and “oh, what should I do now” where another 5-10 minutes can be taken to validate the answer but then… it’s time to create the new instructions.

This is extremely unnatural and I can definitely feel the harshness⁴ of this intense workload.

Hobbies, Beers and LLM

Fair to say that using LLM at work is “part of the job” for many, even LLM-sceptical friends. But what about outside work, during daily life, for hobbies and similar? Well, here I’m half turning into a sceptical too, maybe open to the responsible use but quite against the hype or the forced LLM-usage.

Hard opinion: I do not trust LLM’s output, especially for search-queries⁵.

So, what else can LLM be useful for? Custom tooling, a.k.a. personalised mini-app designed for my personal workflow, for my own data, fully local and designed for my own need.

Sound fun, but… what are these mini-apps?

wallie is an expenses-tracker CLI/TUI with some fixed category-labels, specific accounting and structure. It has a CLI for adding expenses with fuzzy-search and a complete TUI with minimal ASCII plots to quickly check budget.

Why not use one of the million apps already out there? Because many push data online, advertisement everywhere, too complicated interfaces, too many features. All I want was to transform a local spreadsheet into a slightly more general tool, that uses text-based databases, can be easily portable and never leave my laptop.

Reason 1: keep data local.

Something weird I vibe-coded is fmtex, an extremely opinionated LaTeX formatter that only formats tex files by following an extremely small amount of rules. fmtex is designed to be executed only once: when you receive LaTeX file from other people that do not follow my formatting style, you run fmtex that does 90% of the job, then I will finish the job myself.

Why not use a public formatter? Well, they have too many options, too many features and too many configurations and, despite all of these, I’m still unable to obtain the format I want⁶. 🙃

Reason 2: simpler applications, filtered features.

Another tool is wer, a weekly report viewer. I kid you not, this is a viewer for some text-report I use for work. The only objective for the tool is to quickly move between different reports and highlight some sections to read the whole content I wrote. A hidden automatism is that it allows me to generate the “next week” report in an instant, with the correct template, the previous week todos. All in an instant.

Without wer, I spend 5-10 minutes a week to create the new report and, once a month, another 10 minutes to read multiple reports to figure out what I have done in the last month.

It might not be a lot of time, but it’s a boring task I don’t have to do.

Reason 3: accelerating ad-hoc workflow.

I have other small vibe-coded projects I’m still working on but these all follow the principle of “highly custom application designed for my personal use” and I believe that this is how LLM can be useful.

Opinions : Good, Bad and Ugly

I doubt LLM will solve all humankind problems, in fact I bet no LLM will solve any Millennium Prize before the end of 2027⁷. But, something good is there: LLM are amazing to develop ad-hoc tools or translate math into code as my previous examples points out.

This is possible only for harness-LLM, e.g. opencode or similar. I really suggest anyone to avoid chat-LLM whenever using LLM for development or code-discovery. Chats are fine for reasoning, as if you would have in a human-discussion where the important aspect is having a clear sequential interaction of writing and reading. But, for fixing code, writing text, checking content, well… allowing LLM to read/write files makes the whole experience more natural and easier to handle (no copy-paste of code from the chat, no “I forgot this line” or “I cannot copy-paste 10 files easily”).

A weird observation I have is that with LLM, you must, as they say in the aviation industry, always “stay ahead of the plane”. LLM are algorithms that quickly outputs content that we read and find interesting. Our brain is incredibly good in finding answers but it requires time and energy, especially when the raw-answers are of bad quality. But, our brain has limits and LLM are statistical computing machine that will output whatever they regurgitated during training.

You cannot validate LLM-output at the same rate as you obtain it. You cannot provide useful instructions to obtain better answers without consuming energy. You cannot outspeed a computing machine in a computation race. Take a break and give your brain the time/energy to catch up. You will be surprised how much better it can be in finding answers or alternative solutions.

However, I’m still sceptical in using LLM for real private data (both at work and outside). Open-weight model are amazing and are, IMHO, pushing LLM research into the public domain, creating natural competition and real advancement of these tools. Having the model available pushes the industry to price-fight for the inference providing cheaper and higher-quality to the customers (us!). That is all good but still, someone is still technically getting the input/output data.

The only solution is going full-local and run a model locally. Local models, i.e. small open-weight models, are getting really good but I wouldn’t say that investing €3-5k in hardware (plus electricity) for running a local inference machine on private data is reasonable. While, the really-really small models are good for very simple tasks, not helping me navigate private data and develop around them.

What is Next

LLM are here to stay, these tools will improve and (hopefully) we will be able to cheaply use them locally. The question is probably when we will get cheap, computationally efficient⁸.

So, what is next?

Well, I will for sure create new ad-hoc tools for my own usage, just for fun and to simplify my work or hobbies. Currently, I’m looking into finding a clever way to sync laptop and smartphone without a native application, maybe via progressive web-app and all the communication limitation it has.

For sure I will revisit long research idea I have that required scouting-code for the next steps. With LLM the code is arguable but having a prototype is the first step because only after getting a draft-code, it is possible to push the code to the natural limit (by hand, most of the time).

Maxing M1Air usage with math-search algorithm, a.k.a. checking if I can use a M1Air as a cooking pan!

However, for LLM to be useful, there is much to learn and be “in front of the tool”. It’s not easy, sure, but by continuing learning will keep the tool in their place.

And if the tools gets too crazy (or make me crazy), I plan to take a break and do thing at my own pace, slowly. Kinda like this blog!

Thanks for reading, have a nice day!

Footnotes