About Studying in the LLM realm

There was a MIT experiment¹ where three groups of students were asked to write a software in Fortran (which neither of then was proficient in). First group used ChatGPT featured by the Ctrl-C/V trick and finished first. Second group used coding assistance (Llama) and finished second. Third group used old school Google/manuals/Stackoverflow approach, decomposed the task and eventually came up with the solution; for obvious reasons they finished last.

It won’t be that interesting if the story ended this way 😁
These groups were asked to recap their solutions — and the results were quite opposite: the 1^st group recalled nothing, the half of the 2^nd group remembered the solution and the 3^rd group passed this tasks in flying colors.

We discussed this with some friends and there were interesting thoughts I’d like to share.

Missing perspective

Nobody doubts the LLMs are fast and can solve a problem right here right now. Hence, the value can be gained fast. But would it work in long run? The strategy of “being able to X without understanding the X” does not look reliable 😕

Whoever needs a real knowledge and understanding, usually gets it hard way via denial-anger-bargain studying, failing and exercising.

The effort of obtaining is reflected in the value. It holds for the knowledge as well and the MIT experiment confirmed it. We don’t praise what comes easy, so the copy-pasted ChatGPT solution is never remembered. Why should it be indeed?

Therefore, only study, only hardcore. Gosh, I envy the Matrix way of uploading the knowledge via the head plug!

Short-term win strikes back

On the other hand, if it’s an one-off task, is it really worth studying? Let the robots do the work and forget it!

Again, the value of getting the task done is gained! Millions of people drive their cars without understanding how the internal combustion (or electric) engines work. They gain the value in completely different way; the car engine is just a tool.

Catch 22

Let’s borrow the best from both worlds:

• get the trivial or one-off tasks done by the robots,
• be truly proficient in more complex (relevant, needed – ?) stuff.

The question is, how to distinguish which tasks should be outsourced to the LLM and which ones deserve to be understood. This wisdom requires broader vision, abstract thinking and systematic analysis of incoming challenges, which emerge —  — during gaining the fundamental knowledge through studying and exercising.

Even though some people successfully develop the gut feeling for making the “LLM-or-learn” decisions, the old-school study is so far the only scalable way of grasping the nature of the tasks.

I see the analogy between the situation around the LLMs nowadays and the Internet becoming globally available in 90-00s. At the first glance it was “hooray, no need to remember stuff because the Google AltaVista² and Yahoo! exist”. Later we mastered how to use the World-Wide Web for synthesizing new ideas, collaborating and eventually turning the 21^st century completely digital.

How did it happen? — Not through ditching the remembering but through seeing broader perspectives that the new tool opens.