Generative AI – How does it work?

Explore generative AI models’ capabilities, limitations, and implications. Generative AI is revolutionizing tech. But what is it? Why is it getting so much attention? In this comprehensive introduction, learn how generative AI models function, what they can and cannot accomplish, and their ramifications.

Generative AI?

Generative AI (genAI) can generate text, images, music, and videos. AI/ML used to mean supervised, unsupervised, and reinforcement learning. Each provides clustering output insights.

Non-generative AI models work with input (like image classification or sentence translation). Instead, generative models write essays, compose music, develop images, and even create realistic human faces that don’t exist.

Generative AI implications

The rise of generative AI matters. Entertainment, design, and journalism are changing because of content creation.

News organizations can utilize AI to write reports, and designers can get graphic suggestions. Although AI can generate hundreds of ad slogans in seconds, their quality is debatable.

Generative AI can create personalized content. A music app that composes a tune based on your mood or a news app that writes articles on your interests are examples.

AI’s growing role in content creation raises problems about authenticity, copyright, and human creativity.

How does Generative AI work?

Generative AI predicts the next step in a sequence, whether it’s a sentence word or an image pixel. Break down how this happens.

Statistical models

Statistical models underpin most AI systems. They represent changeable relationships with mathematical formulas. In generative AI, models learn to identify patterns in data and generate comparable data.

A model trained on English sentences learns the statistical likelihood of word order, allowing it to construct intelligible phrases.

Data gathering

Data quality and quantity matter. Generative models learn patterns from large datasets. A language model may consume billions of words from books, websites, and other publications.

It could entail evaluating millions of photos for an image model. More diverse and complete training data improves model output diversity.

Transformer and attention operation

Transformers are a neural network design introduced in Vaswani et al.’s 2017 publication, “Attention Is All You Need”. They now underpin most modern language models. Without transformers, ChatGPT fails.

As humans focus on various words in a phrase, the “attention” mechanism lets the model focus on distinct areas of the input data.

This approach helps the model choose which input portions are relevant for a job, making it flexible and powerful.

This code breaks down transformer mechanisms in plain English. A Transformer class and a TransformerLayer class may exist in principle. This is like a floor plan vs a building plan.

This TransformerLayer code demonstrates multi-head attention and customized layouts. One of the most superficial artificial neural networks is feed-forward. It has input, concealed, and output layers.

One-way data flows from input to concealed to output layers. This network has no loops or cycles.

The transformer architecture uses a feed-forward neural network after each layer’s attention mechanism—a simple two-layered linear transformation with ReLU activation.

A Simple Explanation of Generative AI

Generative AI is like weighted dice. Probabilities are based on training data. A training data word often following the current term will be weighted more if the dice reflect the next word in a sentence. Thus, “sky” may follow “blue” more than “banana.” After training, the AI is more likely to choose statistically more probable sequences when “rolling the dice” to generate content.

How can LLMs create “original” content?

Consider a listicle on the “best Eid al-Fitr gifts for content marketers” and how an LLM can create it by analyzing textual clues related to gifts, Eid, and content marketers.

Tokens, which can be one character or one word, are used to break up the text before processing.

For example, “Eid al-Fitr is a celebration” becomes “Eid”, “al-Fitr”, “is”, “a”, “celebration”.

This lets the model handle small amounts of text and grasp sentence structure.

Embeddings turn each token into a vector of numbers. Word meaning and context are captured by these vectors.

Positional encoding adds sentence location information to each word vector to preserve order information for the model.

An attention mechanism focuses on specific areas of the input text for output generation. This was what excited Googlers about BERT.

If our model perceives texts about “gifts” and “Eid al-Fitr” as important events, it will pay “attention” to these linkages.

If it encounters texts about content marketers requiring specific tools or resources, it may associate the concept of “gifts” with them.

As the model processes incoming text through many Transformer layers, it mixes learned contexts.

Even though the original texts never specified “Eid al-Fitr gifts for content marketers,” the model may generate this material by combining “gifts,” “content marketers,” and “Eid al-Fitr”. Because it knows these terms’ broader contexts.

After processing the input through the attention mechanism and feed-forward networks in each Transformer layer, the model generates a vocabulary probability distribution for the following word.

After “best” and “Eid al-Fitr,” it may imagine “gifts” will follow. It may also link “gifts” to “content marketers.”

Construction of substantial language models

Upgrading a transformer model to a complex large language model (LLM) like GPT-3 or BERT requires scaling and refining components.

The steps are as follows:

LLMs learn from massive text data. Data is so enormous it’s hard to explain.

A typical starting point for LLMs is the 750 GB C4 dataset of text data. 805,306,368,000 bytes—a lot of data. These sources may include books, articles, websites, forums, comment sections, and others.

Data variety and depth improve the model’s knowledge and generalization.

LLMs have more parameters than transformers, but the transformer design remains. For instance, GPT-3 has 175 billion parameters. Parameters are the neural network weights and biases learned during training.

Adjusting these parameters to lessen the disparity between predictions and outcomes trains a deep learning network to predict.

Optimization employs algorithms like gradient descent to alter these parameters.

The neural network’s weights modify input data within its layers. They’re tweaked throughout training to optimize model output. Neurons in adjacent layers have weights for their connections.

The neural network adds biases to the output of a layer’s transformation. They provide the model with more latitude to fit training data. Every layer of neurons has a bias.

This scaling lets the model store and handle increasingly complex data patterns and relationships.

The model takes a lot of computer power and memory for training and inference due to its many parameters. This makes training such models resource-intensive and requires GPUs or TPUs.

Strong computational resources teach the model to predict the following word in a series. With each inaccuracy, it modifies its internal settings to improve its predictions.

Attention mechanisms like those we outlined are crucial for LLMs. They let the model focus on distinct input elements when generating output.

Attention mechanisms help the model generate coherent, contextually appropriate content by weighting word importance. This huge size allows LLMs to work.

Transformers predict text. How?

Transformers use numerous layers with attention mechanisms and feed-forward networks to anticipate text.

The model generates a vocabulary probability distribution for the following word after processing. The most probable term is usually predicted.

How are large language models developed and trained?

Building an LLM requires data collection, cleansing, model training, fine-tuning, and rigorous, continual testing.

The model predicts the next word in a sequence after training on a large corpus. This step allows the model to develop word connections that pick up grammar patterns, represent world facts, and seem like logical reasoning. These linkages also reveal training data biases.

Human reviewers follow the criteria to refine the model on a limited dataset after pre-training.

Building LLMs requires fine-tuning. It involves training the pre-trained model on a specified dataset or task. Consider ChatGPT.

Playing GPT models shows that prompting is less “write this thing” and more like

Prompt: Once upon a time

There was an evil wizard on top of a tower.

Why did the chicken join a band?

The drumsticks were the reason for the continuation.

From then on, ChatGPT requires a lot of low-paying work. They develop massive corpora to measure GPT responses and expected behaviors. These workers write several prompt/continuation texts like:

The prompt is to complete the story “Once Upon a Time.”

“Continuation: Yes!” A little village was situated between two gorgeous mountains in a distant area.

Prompt: Share a chicken joke.

Continued: Why did the chicken join a band? Since it had drumsticks!

The fine-tuning process is crucial for various reasons:

· Fine-tuning narrows the model’s language understanding and behavior to specific tasks or domains, whereas pre-training provides a wide understanding. A model adjusted to medical data will answer medical inquiries better.

· Control: Fine-tuning allows developers to manage model outputs. Developers can utilize curated datasets to lead models to desired responses and avoid undesirable behaviors.

By decreasing dangerous or biased outputs it improves safety. Human reviewers can verify the model doesn’t produce incorrect content by employing guidelines during fine-tuning.

· Performance: Fine-tuning enhances model performance for certain workloads. A customer support-optimized approach is better than a generic one.

ChatGPT is fine-tuned.

LLMs struggle with “logical reasoning”. GPT-4, ChatGPT’s strongest logical reasoning model, has been extensively trained to recognize number patterns.

Instead of this:

What is 2+2?

Process: Math textbooks for children often use 2+2 = 4. Sometimes, “2+2=5” is mentioned in the context of George Orwell or Star Trek. In such a case, 2+2=5 would be more likely. Since that context doesn’t exist, the following token is expected 4.

Two plus two is equal to four.

Training works like this.

• Training: 2+2=4

Training: 4/2=2

In training, half of 4 equals 2.

• Training: 2/2 = 4

etc.

The training method for “logical” models is more rigorous and focused on ensuring the model understands and applies logical and mathematical principles.

The model is exposed to mathematical issues and their answers in order to generalize and apply these ideas to new problems.

This fine-tuning is crucial for logical reasoning. Without it, the model may answer simple logical or mathematical questions incorrectly or nonsensically.

Language vs. image models

Image and language models analyze distinct data despite using transformer-like architectures.

Imaging models

Pixel-based models analyze little patterns (like edges) first, then combine them to detect larger structures (like forms), and so on until they grasp the full image.

Models of language

These models process word or character sequences. To write coherently, they must comprehend context, syntax, and semantics.

How do generative AI interfaces function?

Dall-E + Midjourney

Image-generating model Dall-E is based on GPT-3. Text-image pairs form its massive training dataset. Another proprietary model-based image generator is Midjourney.

Enter a textual description, such as “a two-headed flamingo.”

· Processing: Models convert text into numbers, decode vectors, and locate pixel associations to create an image. Training data taught the model how written descriptions and visual representations relate.

· Output: A corresponding image to the description.

Problems, fingers, patterns

Why don’t these gadgets always make for normal-looking hands? Looking at adjacent pixels is how these technologies function.

Compare previous or more basic-generated images to recent ones: Early models are blurry. However, newer models are more precise.

These models generate graphics by anticipating the next pixel from previous pixels. This technique is performed millions of times to complete an image.

The complex characteristics of hands, especially fingers, must be precisely portrayed.

The location, length, and orientation of each finger can change substantially between photographs.

When creating an image from a textual description, the model must make several assumptions about the hand’s attitude and structure, which can cause abnormalities.

ChatGPT

ChatGPT uses the transformer-based GPT-3.5 architecture for natural language processing.

• Input: A prompt or set of messages to imitate a conversation.

• Processing: ChatGPT generates responses using extensive knowledge from various internet texts. It uses conversation context to offer the most relevant and understandable response.

The output is a text response that continues or answers the discourse.

Specialty

ChatGPT is great for chatbots and virtual assistants since it can handle multiple topics and replicate human discussions.

Bard + SGE

Bard uses transformer AI like other cutting-edge language models, although details are secret.SGE combines comparable models with additional ML algorithms from Google.

SGE presumably uses a transformer-based generative model to generate content and fuzzy-extract search results from ranking sites. This may not be true, but playing with it suggests it works. Please don’t sue me!

The input is a prompt, command, or search.

Bard processes input similarly to other LLMs. SGE has a similar design but adds a layer where it examines its internal knowledge (from training data) for a suitable response. Relevant material is generated by considering the prompt’s structure, context, and intent.

The output can be a tale, answer, or other text.

Applications and controversies of generative AI

Design, art

Generative AI can design products, music, and art. This has expanded creativity and innovation.

Controversy

AI in art has raised concerns about creative job losses.

Additionally, considerations include:

Labor breaches, mainly when AI-generated content is exploited without proper credit or compensation,

One reason for the authors’ strike was executive threats to replace them with AI.

Natural language processing

Modern chatbots, language translation, and NLP tasks use AI models.

LLMs are closest to a “generalist” NLP model, making this their best use outside of AGI.

Promoting and advertising

AI can evaluate consumer behavior and create targeted ads and promotions, improving marketing strategies.

LLMs can generate user stories or more sophisticated programmed concepts since they have context from other writing. LLMs can suggest accessories to new TV buyers instead of TVs.

Controversy

Marketing with AI poses privacy problems. The ethics of utilizing AI to influence consumer behavior are also debated.

Explore scaling huge language models in marketing.

LLMS concerns persist

Human speech contextualization and comprehension

AI models, including GPT, may struggle to recognize subtle human behaviors like sarcasm, comedy, or lying.

When a character lies to others, the AI may not understand the duplicity and may interpret remarks at face value.

Pattern matching

A limitation of AI models like GPT is that they are pattern matches. They excel in recognizing and generating content from training data patterns. However, new conditions or variations from patterns can diminish their performance.

If a new slang phrase or cultural reference appears after the model’s last training update, it may not recognize or comprehend it.

Misunderstanding common sense

AI models may hold large volumes of data but lack “common sense” knowledge of the world, resulting in technically correct yet misguided results.

Potential bias reinforcement

Ethical consideration: AI algorithms may duplicate and exacerbate biases in data after learning from it. This can produce sexist, racist, or discriminatory outcomes.

Unique concept generation difficulties

Limitation: AI models rely on observed patterns to generate content. They can integrate patterns in new ways, but they don’t “invent” like humans. Their “creativity” combines ideas.

Concerns about data privacy, IP, and quality control

Using AI models in sensitive data applications raises ethical questions about data privacy. AI-generated material raises IP ownership problems. AI-generated content quality and accuracy are also tricky.

Bad code

AI models may produce acceptable syntax but poor functionality or security when utilized for coding jobs. LLM-generated sites have required me to correct the code. It looked right but wasn’t. Although it works, LLMs use outdated coding standards like “document.write” that are no longer recommended.

MLOps engineer and technical SEO hot takes

LLM and generative AI hot takes are in this section. You can fight me.

Generational text-interface prompt engineering doesn’t exist.

Large language models (LLMs) like GPT-3 and its descendants are touted for generating coherent and contextually relevant text from prompts.

These models have become the new “gold rush,” and therefore people are monetizing “prompt engineering” as a skill with $1,400 courses or positions.

There are a few important factors:

LLMs evolve quickly

GPT-3 may respond differently to prompts than GPT-4 or a subsequent version of GPT-3.

Prompt engineering is constantly changing, making it difficult to maintain consistency. January prompts may not work in March.

Events beyond control

LLMs don’t know numbers; thus, asking them to write a 500-word essay may result in various lengths.

You can ask for accurate information, but the model may produce inaccuracies because it cannot distinguish between accurate and inaccurate information.

Using LLMs in non-language apps is terrible

LLMs can be used for various activities, although they are designed for language tasks.

Problems with new ideas

Because they’re taught on existing data, LLMs regurgitate and combine what they’ve seen previously, not “invent” in the genuine meaning.

LLMs are not for innovative or out-of-the-box tasks

This is problematic when employing GPT models for news content because LLMs struggle to handle new content.

For instance, a site using LLMs produced a potentially defamatory piece about Megan Crosby, who was caught elbowing opponents in real life.

Without context, the LLM made up a “controversial comment” scenario without evidence.

Text-focused

LLMs are created for writing, but they can be altered for image generation and music composition. However, they may not be as effective.

LLMs are unaware of the truth

They can’t check facts or distinguish truth from fiction because they produce patterns from training data. They may perpetuate mistakes in their outputs if they were trained on misleading or biased data or lack context.

This is especially problematic in news generation and academic research, where accuracy and veracity are crucial.

Consider this: if an LLM has never heard of “Jimmy Scrambles” but knows it’s a name, cues to write about it will only yield related vectors. Design is always better than AI-generated art.

AI has made great progress in art, from painting to music, yet there’s a key difference:

Feeling, vibe, intention

Art is about intent and emotion as much as product. Human artists provide depth and nuance to their work with their experiences, emotions, and views, which AI can’t match.

Personal “bad” work is more profound than prompt art.