One day I was working on a TYPO3 14 upgrade. I have my own skills for AI that help to perform upgrades faster. Each site is different, so the skill can help a lot, but it is not a universal solution. In many cases, I need to give more information to AI and also ask it to stop at certain points because I need to verify what it did. I have experience communicating with AI and I know how to prompt it for effective work. It worked quite well upgrading from version 11 to version 14. It worked fine from 11 to 13, and then it started on 14. I looked at what it did and suddenly I saw that something is not right there. It was writing code which was not compatible with TYPO3 14 anymore. Most of the changes were compatible because they were very similar to version 13, but some of them were not. It tried to use non-existing APIs. I raised my eyebrow, but I waited.
AI finished and told me that everything is done, it tested frontend and frontend worked. So the upgrade was done successfully, said the AI.
I went to backend and of course I saw exceptions. I asked AI to check the logs and see if there are any exceptions. It went checking logs and of course found the exception from the backend. It started to investigate. It took a long time because it started to look in the code and it said it needs to reverse engineer the code. This is where I stopped it. I didn't want to spend more tokens and more time on this. I opened TYPO3 core documentation, found the necessary page and gave it a link. I also said that this is a change in TYPO314, so check the documentation and do according to the documentation.
AI loaded documentation, read everything and went to change the code as necessary. It didn't have to reverse engineer TYPO3 core. The error was fixed and the site worked.
Why did the problem happen?
AI has a term called “knowledge cutoff”.
When a model is trained by its creators, it is given a lot of data, which it learns. However, when training ends, this is all the model knows. So if the model knows about TYPO3 11, 12 and 13, then that's it. It doesn't know about version 14. It doesn't know that some APIs are removed or changed. It can write you the code for 11, 12 or 13 without consulting the documentation. However, if you ask it to write for 14, in most cases it will silently do so but use APIs from 13. If those APIs were already deprecated in version 13 and they are gone in 14, then the model will still use them. It doesn't know that they are gone.
What do you have to do in such cases? You have to give it pointers, and this was my mistake in this case: I did not provide pointers there. After this case, I updated my upgrade skills for AI and I added pointers to the information. I asked AI to review it and see if this would be helpful if those pointers were there before. It said yes, this would help and there would be no mistake.
Practical conclusions
If you work with skills, take knowledge cutoff into account. Make sure that you provide pointers to updated versions of your library or new AI model names if you use them in your project, because AI will not even know that new models were released after it was trained. You can tell it “Use model X1000,” but it can silently replace it with “X900” because it will think that the user made a mistake and “X1000” does not exist. Give it a URL to the model page.
If you use skills, you can put those pointers directly in the skill. This way you will not have to repeat yourself each time when you use that skill.