LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models

I'm exploring a simple idea for making agentic LLMs more compute-efficient: not every step in an agent workflow needs the full depth of the model.

In this paper, LayerRoute, I add lightweight per-layer routers and LoRA adapters to a frozen Qwen2.5-0.5B model. The routers learn to selectively skip transformer layers based on the input, using hard binary gates trained with a straight-through estimator (STE). The interesting observation is that structured tool-calling steps naturally learn to skip more layers, while open-ended planning and reasoning steps retain most of the model depth.

On a mixed agentic dataset, the approach achieved a 12.9% skip differential between tool calls and planning steps, while maintaining or improving perplexity through LoRA adaptation. Training required only ~1.1M trainable parameters (0.22% of the backbone) and completed in a few minutes on a single A100.

I'm curious about the community's thoughts on a few questions:

• Would sequence-level routing scale to larger models (7B–70B)?
• Is the learned middle-layer skip pattern a genuine architectural property or an artifact of the training setup?
• Could this be combined with speculative decoding, MoD-style routing, or KV-cache optimizations for larger gains?
• What benchmarks would best evaluate adaptive depth methods in real-world agent systems?

Paper and feedback welcome!