I've been using Github Copilot daily: an AI-powered code autocompletion tool that seems to have a life of its own.
A real case searching for a currency conversion API.
GitHub Copilot is available as an extension for Visual Studio Code, Neovim, and JetBrains. It sends the code and context (programming language, entire project code, file names) to OpenAI's servers, where it's completed and sent back as a suggestion. Pressing Tab adds the suggestion to the code, while Esc discards it.
It's based on an AI called Codex, a descendant of GPT-3 tailored for programming languages and trained on all public domain repositories on Github. This includes code, comments, and documentation in every imaginable programming language.
Using it for the first time is exciting. Writing a function name and having it generate not only the code but also a valid link to a currency conversion API feels like something out of a sci-fi novel. However, after the initial hype, it's easy to see that the tool isn't perfect. It needs guidance to achieve the desired result, and you need to understand what it's writing to maintain control.
For example, at second 0:30 in the previous video, it tries to use a paid service, so I define a URL for the European Central Bank to give it a clue about which tools to use. Even then, the generated code isn't perfect, and I have to manually correct the regex to parse the data.
The death pyramids are one of its most typical errors.
Still, I decided to keep the plugin active for the last 5 months, and having tried it with many different languages: Python, JS, CSS, C++, PHP, SQL, Arduino, VBS, OpenScad, GLSL… it's been a radical change in my usual way of programming. One of those leaps that happen few times in life.
When I was young, my father bought Computer Hoy every month, and I read it from cover to cover. In one edition, there was a guide that said something like "learn to optimize your tasks with Visual Basic," and for some reason, I followed it. When I saw the magic of thinking up an idea, describing it to a computer, and seeing it come to life, I felt the first of those leaps.
Since then, I started programming in VBS with nothing more than that guide and some photocopies someone got me. My way of solving my needs was to look at those 20 photocopied pages over and over until I found an answer. When the sheets were worn and yellowed, the internet came home, and the next leap happened.
Programming became very different. Now my knowledge was not limited to a few sheets, but to the complete reference of Visual Basic or any other language I wanted to learn. And between online references, forums, and email threads, I kept programming until I started University, which was when I learned what an IDE with debugger was.
That leap was also significant. Now I could write in a text editor that colored my code, autocompleted some methods, allowed me to navigate the language reference with just Ctrl+Space, and see line by line how my application's state changed. And on top of that, the forum was Stack Overflow.
Today I still program like that, but I have another ace up my sleeve: the ability not to look at an API if I don't remember something, to let the IDE finish my code following the same style as my codebase, and in general to focus on the fun part of programming instead of struggling with the APIs of Matplotlib, Puppeteer, or PyTorch.
Programming image recognition with a webcam on a Raspberry with just kindly asking for it in a comment.
When Copilot came out, all opinions after a week of use were that it would replace programmers and that it was a magical tool, so I wanted to wait until I had tested it for a good time to understand its place in the programmer's toolbox. And my conclusion is that it's very, very useful if used with some philosophy.
It's easy to get carried away by the solutions Copilot proposes (after all there's no Oscar for best code). But as it's not perfect, without knowing what it's doing, it's easy to write dysfunctional code. Or worse: functional but with hard-to-detect vulnerabilities. Something similar to indiscriminate copy-pasting from Stack Overflow.
It's clear that there's a trend towards increasingly assisted programming. And if the future lies in tools like Copilot, we'll have to tackle challenges like avoiding the distraction of recommendations or promoting critical thinking in the face of suggestions that imply bad practices and outdated approaches; something similar to what happens with AIs trained on real datasets and their acquired biases.
For now, we can try it out to better understand its implications. If you've read this far and are interested, you'll be glad to know that telemetry is only used to see which solutions are accepted, but the code that is written does not feed the model. Being a Windows product, they're probably fishing to catch us later with a subscription service, so you can sign up for the beta before it's too late.
Every video in this post was converted to webm like this, avoiding looking 100 times to the FFMPEG reference.
This weekend I was at the Global Game Jam (GGJ) creating a game in 48 hours with @pabletos and @j_coronel. The result turned out pretty cool, so I'm sharing it here.
A game jam is an event where participants develop a game around a theme within a time limit. GGJ is an annual global event with physical venues all over the world, like the one hosted by espacio_RES this time. The time limit was 48 hours, and the theme was Duality.
We had participated in other jams before, such as the Familiar Game Jam, Ludum Dare, or GM48; however, this is one of the most polished games we've made in such a short time. The mechanics involve overcoming levels of a roguelike (a turn-based exploration game) by adding pieces to a board.
The board's behavior is quite simple: it reads instructions from top to bottom (like a Turing machine) and applies them to the player. The mechanics are quite emergent, and with very few elements, many different strategies can be found, so we'll keep refining it to upload it to the App Store.
Despite having participated in many jams, this was the first time we didn't opt for the remote mode. And although being in a house for 48 hours in 'cave mode' making a game is not bad at all, meeting so many talented developers, designers, and artists is incomparable, so we will definitely repeat the experience.
The following conversations have been fully generated by an AI that I just supplied with the first three bold sentences:
-/-<>
The magic behind this supercharged autocomplete is GPT-3, an algorithm trained to convincingly continue texts. In some cases, it manages to capture such subtle and human traits as humor while maintaining complex conversations around a topic.
GPT-3 belongs to the family of algorithms known as transformers, which have proven to be very effective in recognizing complex patterns in sequences of elements. By treating language as a sequence of words, they can be trained so that, given an input text (context), they complete it in the most credible way.
Thanks to the Information Sciences Institute in Los Angeles, I have received an invitation to test this tool, one of the most advanced currently available, the Optimus Prime of natural language. It has already achieved many successes in solving text-based challenges such as maintaining conversations indistinguishable from those of a human, helping write novels, or even programming software on demand.
Such a deep understanding of natural language can greatly improve our way of communicating with conversational interfaces like Google Assistant or Alexa. And as great power entails great responsibility, the company behind this tool is gradually and cautiously opening it to the public. Currently, it is only possible to use it by invitation or by paying a subscription with a rate per number of words generated.
But what has already been invented cannot be uninvented, and an infinity of projects have emerged that try to replicate the recent successes of GPT-3 openly. In fact, anyone without any knowledge of machine learning can now install the open-source version, GPT-Neo, and try it with a single click (and a good graphics card).
These alternatives are quite recent - they are less than 6 months old - so the tsunami of bots indistinguishable from a human is still to come, although there have already been some cases of misuse, mainly aimed at manipulating public opinion to make a crowd in the followers of a political party or alter the valuation of cryptocurrencies.
Recent studies say that 15% of Twitter accounts could be bots, and our mechanisms to detect them are still quite ineffective. And like fake news, they are here to stay, so we will have to start thinking about scalable solutions to distinguish between humans and bots. At least until we have a Voight-Kampff test.
In this project, I've analyzed color evolution in video games. I've downloaded longplays—videos of someone completing an entire game—and converted the dominant color of each frame into a vertical line, creating a timeline.
Analyzing video games from this perspective, I've found many recurring patterns. Here I discuss the most interesting ones, but you can see them all in the link to the project:
Some games use the color spectrum to give a sense of progress to the player. In Journey, we start in a desert and as we solve certain puzzles, we discover other parts of the world and the color changes.
Other games also use color to give a sense of progress, but more directly: they divide the game into worlds, each with a distinct palette. In games with little changing dynamics like Bit Trip Runner, this resource gives more sense of variety. It's interesting how they often start with bright green tones and change to dark red tones.
There are also games that don't limit color evolution to the game itself, but let it change over an entire series. This is the case with Diablo, Elder Scrolls, or BioShock. In all of them, the color becomes more vivid from one game to the next. The case of Diablo III was quite notorious, and everyone complained about how colorful it was.
Most games also use color to give identity to the game, and many can be easily identified just by looking at the color palette. Some games like GRIS and Celeste don't honor their names, but they have a palette that anyone who has played will be able to identify.
In addition to the games in this post, I've uploaded others that are quite curious, like the entire Zelda, GTA, or Final Fantasy series. But I didn't want to miss out on including other games whose palette is simply spectacular and I couldn't think of any other excuse.
I started this post as a continuation of the one I wrote about the name of colors, and initially only intended to get a color palette, not a timeline. But when I saw what the artist Dillon Baker did with movies, I thought it would be more interesting to emulate it with video games.
To obtain the longplays, I used the online video download library youtube-dl, which Github recently removed following a complaint from the American SGAE. The complaint has catapulted the popularity of this library, producing a Streisand effect and it has already been restored.
Perhaps the most challenging part has been identifying the predominant color of each frame, which remains an active field of research. I tried using the average of all colors on the screen, but it resulted in a grey and brown mass, similar to mixing all the playdough. So, I ended up doing clustering of the image's colors and keeping the color that represents the largest cluster.
Papers, Please is grey anyway.
I've considered uploading high-resolution versions of these timelines as prints on RedBubble, but creating a product for each game is a massive task, so if you're interested in any particular one, you can leave a comment and I'll upload it individually.
I recently created the library xkcdcolornamer.js, which allows you to get the name of any RGB color. You can see an example of its use by clicking on the color below to modify it and see its name.
It seems that humans are not very good at naming colors. This is not an easy task, as it involves assigning discrete categories to a continuous spectrum, and intermediate tones can vary according to our perception.
In the image above, you can see the dress that became so famous because no one could agree whether it was white-gold or black-blue. To the left is the original version, and in the middle is a version of the image using only the 256 web colors standardized by SVG 1.0.
For the right version, I took each pixel of the dress photo and replaced them with the closest color out of the 256 most used in English. The top ten are: green, blue, purple, pink, brown, red, light blue, turquoise, grey, and orange.
This list does not include colors like white or yellow. This is because I generated it using the xkcd color survey, in which 222,500 users named randomly chosen colors. As can be observed, it's not the best palette for compressing an image.
While this method has some statistical issues (the color range of each screen varies, men are overrepresented, and the average of a set of colors is not usually the most representative color), it gives a good idea of our limited ability in naming colors.
The 50 most named colors.
One reason might be genetic influence. Beyond colorblindness and similar alterations, not everyone perceives colors in exactly the same way; in fact, we don't even see colors exactly the same with our left and right eyes.
Another important factor is cultural influence: our language greatly affects how we understand color. It's surprising that in the Iliad, Homer described the sea as having the color of wine, and he made no mention of the colors blue and orange in his poems.
In general, language limits—or expands—our view of the world. For example, the Australian Thaayorre tribe has no relative direction terms such as left and right, and speak just in absolute coordinates like north and south. This in turn made them have a great orientation.
All these factors make our way of seeing colors not too apropriate to represent images with the smallest palettes possible, but it still the way we speak. Which is why it was strange taht there's nothing else available to name colors using our peculiar palette.
By the way, that dress is gray and no one is gonna convince me otherwise.
