Post

More from @rohanpaul_ai

Rohan Paul

@rohanpaul_ai

Jun 30

Another 'WOW' paper - Upto 20x improvement in inference throughput with Block Transformer compared to vanilla transformers with equivalent perplexity.🤯

How ❓ by MASSIVELY reducing KV cache IO overhead from quadratic to linear with respect to context length, solving a key challenge in scaling to very long contexts and also novel application of global-to-local modeling. 🤯

Paper - "Block Transformer: Global-to-Local Language Modeling for Fast Inference":

📌 Block Transformers can also be uptrained from pretrained vanilla models, closely approaching the performance of those pretrained from scratch, using just 10% of the training budget.

📌 adopts a hierarchical global-to-local modeling approach. It isolates the expensive bottlenecks of global modeling to lower layers and applies fast local modeling in upper layers. This is achieved through three components:

1. Embedder: aggregates each block of L_B input tokens into an input block embedding. i.e. L_B represents the block length, which is the number of tokens aggregated into a single block.

2. Block decoder: an autoregressive transformer that applies self-attention between blocks to decode a context block embedding for predicting the next block.

3. Token decoder: autoregressively decodes the token contents of the next block, applying local self-attention between only the L_B tokens within the block.

📌 The block decoder reduces overall costs through its coarse granularity. It mitigates the quadratic costs of self-attention by using coarse-grained block inputs instead of individual tokens, reducing context length by L_B. This reduces FLOPs for positionwise computations by L_B and attention score computation by L_B^2. KV cache usage and KV cache IO are also reduced by L_B and L_B^2 respectively.
📌 The token decoder nearly eliminates the costs of attention as there is no need to compute, store, and retrieve KV-cache of past tokens beyond the small local context of L_B tokens. It eliminates prefill (necessary only in the block decoder) and reduces KV cache IO from quadratic to linear with respect to context length. This allows for significantly higher compute unit utilization.
📌 To incorporate the context embedding and leverage the low-cost compute in the token decoder, the context block embedding is projected into prefix tokens. This enables further refinement of the global context and allows increasing computational width of the token decoder by extending the prefix length.

Hierarchical global-to-local architectures have shown significant potential to effectively model large-scale data by addressing global dependencies in coarse detail and capturing fine details within local regions.

Read 7 tweets

Rohan Paul

@rohanpaul_ai

Jun 30

LLMs are highly sensitive to prompt variations, leading to inconsistent performance across different prompts for the same task. 👨‍🔧

Intent-based Prompt Calibration (IPC) iteratively refines prompts to match user intent using synthetic boundary cases, addressing prompt sensitivity and optimizing with limited data.

📌 IPC generates challenging synthetic samples at each iteration, focusing on boundary cases that expose prompt ambiguities.

📌 The system employs three meta-prompts: Sample Generator, Analyzer, and Prompt Generator. The Sample Generator creates diverse, adversarial samples with balanced class distribution. The Analyzer evaluates prompt performance and identifies failure cases. The Prompt Generator suggests improved prompts based on historical performance and analysis.

📌 For generative tasks, IPC first calibrates a ranking prompt, then uses it to optimize the generative prompt. This approach allows optimization with minimal annotation effort.

📌 The system architecture consists of four components: Dataset (manages data operations), Estimator (handles predictions and annotations), Evaluator (assesses records and performs error analysis), and Optimizer (manages the optimization process flow).

📌 IPC outperforms existing methods like OPRO and PE on classification tasks (spoiler detection, sentiment analysis, PG detection) and generative tasks (enthusiastic/reliable and sarcastic/positive movie reviews).

📌 The method demonstrates superior performance with limited data, achieving higher accuracy and lower variance compared to baseline approaches.

📌 Ablation studies reveal the importance of synthetic data generation, iterative refinement, and error analysis in improving model performance.

📌 IPC effectively handles imbalanced data distributions by generating balanced synthetic samples, particularly beneficial for real-world moderation tasks.

Read 5 tweets

Rohan Paul

@rohanpaul_ai

Jun 30

A very intriguing recent paper "Nested Jailbreak Prompts can Fool LLMs Easily" - reveals the inadequacy of current defense methods in safeguarding LLMs.

Generalizes jailbreak prompt attacks into two aspects:

(1) Prompt Rewriting and
(2) Scenario Nesting.

📌 Propose ReNeLLM, an automatic framework that leverages LLMs themselves to generate effective jailbreak prompts. Sgnificantly improves the attack success rate while greatly reducing the time cost compared to existing baselines. Our study also reveals the inadequacy of current defense methods in safeguarding LLMs.

📌 ReNeLLM framework introduced, generalizing jailbreak prompt attacks into prompt rewriting and scenario nesting. Prompt rewriting involves operations like paraphrasing, altering sentence structure, misspelling sensitive words, inserting meaningless characters, partial translation, and changing expression style. These operations preserve semantic meaning while disguising harmful intent.

📌 Scenario nesting embeds rewritten prompts into task scenarios like code completion, text continuation, and table filling. This leverages LLMs' instruction-following capabilities to bypass safety alignments. Scenarios chosen align with training data, shift attention, and leave blanks for completion.

📌 Automated process uses LLMs to generate and evaluate jailbreak prompts. GPT-3.5 performs rewriting and harmfulness evaluation. Nested prompts fed to target LLM (e.g. Claude-2) for response. Success determined by harmful output generation.

📌 ReNeLLM achieves state-of-the-art attack success rates (ASR) across open and closed-source LLMs. For Claude-2, ReNeLLM attains 69.6% GPT-ASR compared to 0% for baselines. Time cost reduced by 76.61% vs GCG and 86.19% vs AutoDAN.

📌 Attention visualization reveals LLMs' priority shift from balancing external/internal instructions to favoring external ones after rewriting/nesting. This explains jailbreak success and informs potential defenses.

📌 Defense strategies explored: incorporating priority prompts (e.g. "prioritize safety"), enhancing safety through supervised fine-tuning, and using harmfulness classifiers. Results show trade-offs between safety and performance, highlighting challenges in developing robust defenses.

Read 6 tweets

Rohan Paul

@rohanpaul_ai

Jun 29

This 76-page paper on Prompting Techniques has become quite popular. A nice read for your weekend.

- "The Prompt Report: A Systematic Survey of Prompting Techniques": ✨

Explores structured understanding and taxonomy of 58 text-only prompting techniques, and 40 techniques for other modalities.

📌 The paper focuses on discrete prefix prompts rather than cloze prompts, because prefix prompts are widely used with modern LLM architectures like decoder-only models. It excludes soft prompts and techniques using gradient-based updates.

📌 The paper identifies 58 text-based prompting techniques broken into 6 major categories:

1) In-Context Learning (ICL) - learning from exemplars/instructions in the prompt
2) Zero-Shot - prompting without exemplars
3) Thought Generation - prompting the LLM to articulate reasoning
4) Decomposition - breaking down complex problems
5) Ensembling - using multiple prompts and aggregating outputs
6) Self-Criticism - having the LLM critique its own outputs

📌 For ICL, it discusses key design decisions like exemplar quantity, ordering, label quality, format, and similarity that critically influence output quality. It also covers ICL techniques like K-Nearest Neighbor exemplar selection.

📌 Extends the taxonomy to multilingual prompts, discussing techniques like translate-first prompting and cross-lingual ICL. It also covers multimodal prompts spanning image, audio, video, segmentation, and 3D modalities.

📌 More complex techniques like agents that access external tools, code generation, and retrieval augmented generation are also taxonomized. Evaluation techniques using LLMs are discussed.

📌 Prompting issues like security (prompt hacking), overconfidence, biases, and ambiguity are highlighted. Two case studies - benchmarking techniques on MMLU and an entrapment detection prompt engineering exercise - are presented.

Read 9 tweets

Rohan Paul

@rohanpaul_ai

Jun 28

Activation Beacon is such a classic paper from Jan-2024

"Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon"

Can be a revolutionary paper if implementable for all cases - for massively increasing the context window of LLMs

Authors trained LLaMA-2 for 10K-steps with 4K context window and then it generalized to 400K context window 🔥

📌Key technique is to condenses LLM's raw activations into more compact forms such that it can perceive a much longer context with a limited context window. Activation Beacon is introduced as a plug-and-play module for the LLM.

📌 "It fully preserves the LLM's original capability on short contexts while extending the new capability on processing longer contexts. Besides, it works with short sliding windows to process the long context, which achieves a competitive memory and time efficiency in both training and inference. "

📌 "Activation Beacon is learned by the auto-regression task conditioned on a mixture of beacons with diversified condensing ratios. Thanks to such a treatment, it can be efficiently trained purely with short-sequence data in just 10K steps, which consumes less than 9 hours on a single 8xA800 GPU machine."

📌 "The experimental studies show that Activation Beacon is able to extend Llama-2-7B's context length by ×100 times (from 4K to 400K), meanwhile achieving a superior result on both long-context generation and understanding tasks. Our model and code will be available at the BGE repository."

A relevant question here, that automatically comes to mind - 'How does the quality of retrieval change over context length ❓

And we can refer to this paper for understanding that aspect.

Paper - ''Training-Free Long-Context Scaling of Large Language Models'

This paper shows that existing long-context LLMs, which have already supported a 32k context window, can further extrapolate to a 192k context length while maintaining high passkey retrieval accuracy and low perplexity.arxiv.org/abs/2402.17463

Activation Beacon Paper -

Activation Beacon Github Code - arxiv.org/abs/2401.03462
github.com/FlagOpen/FlagE…

Read 7 tweets

Rohan Paul

@rohanpaul_ai

Jun 23

Sliding Window Attention is such a brilliant idea 💡

And it was one of the secret sauces behind the legendary Mistral-7B, which enabled it to handle 100k+ token sequences with linear (ish) complexity.

A long thread 🧵1/n
---

📌 Most Transformers use Vanilla Attention, where each token in the sequence can attend to itself and all the tokens in the past.

📌 So the memory increases linearly with the number of tokens. Hence the problem of higher latency during inference time and smaller throughput due to reduced cache availability.

📌 Sliding Window Attention (SWA) can alleviate those problems and can handle longer sequences of tokens more effectively at a reduced computational cost.

So standard, decoder-only, causal LMs (like the whole GPT series), each token can "attend to" (i.e. "look at") every token that has come before it.

In Sliding Window Attention, earlier layers have a narrower view of history, and this progressively builds up the deeper you go into the model.

----

📌 Performance implications of sliding window attention:

Computational complexity: O(n * w) where n is sequence length, w is window size
Memory usage: O(w) instead of O(n) for full attention
Information retention: Local context preserved, global context approximated

----

📌 Because SWA exploits the stacked attention layers to attend information beyond the window size W.

📌 Each hidden state h in position i of layer k can attend to all hidden states from the previous layer with position between i-W and i. Where `W` is the "Window Size"

📌 This holds for all hidden states. Thus, recursively, a hidden state can access tokens from the input layer at a distance of W x k tokens. With 32 layers and a window size of 4096, this model has an attention span of 131k tokens.

----

📌 Limitations of Sliding Window Attention

Lack of Global Context − Because Sliding Window Attention operates on fixed windows, it may not be able to capture long-range dependencies that span across multiple windows. This can limit the model's ability to understand the global context of the input sequence.

📌 Example, if prompt/instruction text is 16K but Sliding Window Attention's sliding window is only 4K, it may cause my instructions to get ignored, as the window moved to the last 4K of those 16K and will "un-attend" my instructions at the beginning of those 16K.

----------

📌 Global Attention, in contrast, considers the entire input sequence at once, applying attention to all positions simultaneously. It focuses on specific, strategically chosen locations to capture the most relevant information, ensuring that each token with global attention is connected to every other token in the sequence. While Global Attention provides a comprehensive view of the sequence context, it can significantly increase computational demands.

📌 Combining SWA with Global Attention, as seen in architectures like Longformer, offers a balanced approach. This hybrid method maintains efficiency while ensuring the model captures both local and global sequence context, crucial for accurate performance on tasks with long input sequences.

🧵2/n

📌 Common misconceptions about sliding window attention:

* It completely discards all information from earlier tokens

* Linear complexity means no performance trade-offs

* It's always better than full attention for all tasks

🧵3/n

📌 Information loss mitigation strategies:

1. Overlapping windows: Retain some information from previous windows
2. Sparse global attention: Allow select tokens to attend globally
3. Hierarchical structures: Capture long-range dependencies at higher levels

Read 7 tweets

Share this page!

Enter URL or ID to Unroll

Rohan Paul

Try unrolling a thread yourself!

More from @rohanpaul_ai

Rohan Paul

Rohan Paul

Rohan Paul

Rohan Paul

Rohan Paul

Rohan Paul

Did Thread Reader help you today?

Don't want to be a Premium member but still want to support us?

Send Email!