We May Be Teaching AI Filmmaking the Wrong Way

Reposted from my LinkedIn articles, first published March 28, 2026.

1. From prompts to structured production pipelines

Over the past few months, as I’ve been working with and around AI video tools and discussing them with colleagues, I’ve found myself returning to the same question: are we actually framing this in the right way for teaching and practice?

Much of the current conversation is organised around prompt engineering. The emphasis is on how to write better prompts, how to achieve more cinematic results, and how to refine language so that the system produces something closer to what we had in mind. That framing follows quite naturally from the interface, where everything appears to hinge on a text input, but it begins to feel insufficient once you try to work with any degree of control.

The difficulty tends to emerge not at the point of writing the prompt, but in what follows. A clip is generated that is recognisably close to the intention, but unstable in ways that are difficult to diagnose. The timing of actions may be inconsistent, the camera may behave unpredictably, or elements that should remain fixed begin to drift.

At the point of working with a system, this becomes visible quite quickly. A single prompt is often carrying multiple, unresolved decisions: the system is inferring not only what should appear, but how the action unfolds, how long it lasts, and how the camera behaves. In tools such as Runway or Sora, this tends to surface as variation that is difficult to control—timing shifts, camera drift, or inconsistent subject behaviour across generations—because those aspects have not been specified independently of the text input.

From a filmmaking perspective, this is a familiar problem, but it is usually resolved in a different way. When a shot does not work, the issue is rarely attributed to how it was described. Instead, attention shifts to how it has been constructed.

What becomes apparent when working with AI systems is that this structure is no longer implicit. It has to be articulated. Seen in this way, the prompt is not the unit of production. It is a translation layer between a structured description of a shot and the system that generates it.

The discussion that follows explores that shift—moving from prompts to production thinking—by looking more directly at what that structure consists of, how it can be formalised, and how making it explicit begins to stabilise outputs.

2. What’s missing is structure

The issue with AI filmmaking is not primarily one of prompting, but of control. The question that follows is where that control actually comes from, and why it appears so difficult to achieve when working with these systems. What I have come to recognise, approaching this as a research problem, is that the difficulty is one of structure.

In filmmaking, a shot is never simply a description. It is an organised event. Something is present in the frame, something happens, it happens in a particular order, it is viewed from a particular position, and it unfolds over a defined duration. These elements are interdependent, but they are also distinguishable, and in practice they are routinely adjusted independently of one another.

When we move into AI-based workflows, that organisation tends to collapse into a single paragraph. The prompt becomes a container for multiple dimensions of the shot—subject, action, camera, timing, tone, and continuity—without any explicit separation between them. The system is then required not only to interpret what is being asked for, but to infer how these elements relate to each other.

This is where instability begins to emerge. Small changes in wording can produce disproportionately different results, not because the system is inherently unpredictable, but because the structure of the request has not been fixed. The model is not simply generating variation; it is resolving ambiguity across multiple dimensions simultaneously.

One way to address this is to make the structure of the shot explicit before it is translated into a prompt. At a minimum, this involves separating out four elements: what is present in the frame, what happens and in what sequence, how it is viewed, and how long it lasts. These are not new categories, but making them explicit changes how they function within the workflow.

In practice, this separation can be implemented in relatively lightweight ways. A shot can be written out as a short structured description before it is expressed as a prompt, or mapped into fields in systems that support more explicit inputs. In both cases, the aim is the same: to reduce the number of decisions the system is required to infer.

What this changes, in practical terms, is how variation behaves. If the shot is not structured, each generation behaves like a new attempt, because the system is resolving multiple uncertainties at once. A change in wording can alter the action, the timing, and the camera simultaneously, making it difficult to understand why a result has improved or deteriorated.

Once the structure is fixed, those variables become more independent. The action can remain stable while the framing is adjusted, or the duration can be extended without altering the sequence of events. Variation is not eliminated, but it is constrained. The system is no longer inventing the organisation of the shot; it is operating within one that has already been defined.

From a research perspective, this is the point at which prompting begins to resemble a system rather than a purely linguistic interaction. The prompt is no longer the primary object of design, but one component within a larger process that includes specification, translation, generation, and evaluation.

The next step is to look more closely at how this structure can be formalised, and how it begins to function as an intermediate layer between creative intent and model execution.

3. What does that structure actually look like?

Having established the need to make the structure of a shot explicit, rather than relying on a prompt to carry everything implicitly, the next step is to consider what form that structure takes when it is formalised, and how it operates within a working process.

One way of approaching this is to think in terms of representation. Film production already relies on multiple representational layers. A script is not the same as a storyboard, and a storyboard is not the same as a shot list, but each is a translation of the same underlying intention into a form that can be acted upon. What changes across those layers is not the idea itself, but the degree to which it is specified.

AI workflows introduce the need for an additional layer of representation, one that removes ambiguity in a way that a system can interpret. The shot remains the central unit, but it can no longer rely on shared conventions or tacit knowledge. The relationships between its components need to be made explicit.

In practical terms, this means expressing the shot as a structured object. At a minimum, this involves defining the subject and environment, decomposing the action into a sequence of temporal steps, specifying the camera in relation to that action, and fixing the duration over which the sequence unfolds. Additional elements such as lighting, tone, or stylistic constraints can be layered onto this, but the essential move is the separation of these components into distinct fields.

At the level of implementation, this structure can take a number of forms. In some cases, it is expressed as a consistent written template that is reused across shots. In others, it may be formalised as a small block of structured data—often resembling JSON—or mapped into fields within a node-based workflow. The specific format is less important than the fact that the structure is stable and reusable.

What this enables is a separation between the representation of the shot and the systems used to execute it. The same structured description can be translated into a prompt for one tool, into parameter fields for another, or into a more explicit pipeline where the relationships between elements are exposed. The interface changes, but the underlying representation remains consistent.

From a practical perspective, this introduces a different kind of control. Because the shot is decomposed into components, changes can be applied selectively. The timing of an action can be adjusted without affecting the camera, or the framing can be altered without rewriting the sequence of events. This makes it possible to reason about changes in a way that is difficult when everything is embedded in a single block of text.

It also changes how outputs are understood. Instead of each generation being treated as a separate attempt, outputs can be seen as variations of a defined structure. This makes comparison more meaningful, because differences can be related back to specific aspects of the representation rather than to the prompt as a whole.

From a computational perspective, this begins to resemble what is often described as an intermediate representation. It sits between intention and execution, allowing the same description of a shot to be reused, modified, and translated across different systems without being rewritten each time.

At this point, the prompt becomes a derived artefact rather than the primary object of design. It expresses the structure, but it does not define it.

This brings a further constraint into view: the role of time and the difficulty of maintaining coherence across longer sequences.

4. Time is the constraint: why video behaves differently

So far, I’ve been describing AI filmmaking in terms of structure and representation—how a shot is defined, and how that definition can be stabilised before it is passed to a generative system. That framing works reasonably well for still images, but it becomes more complex when applied to video.

The difference is not simply that video contains more data. It is that it introduces time as a constraint on coherence.

In a still image, the system resolves relationships within a single frame. Composition, lighting, and subject are fixed at once. In video, those relationships have to be maintained while change occurs. The system is not only generating frames, but attempting to sustain consistency across them while allowing action to unfold.

This is where the characteristic instabilities appear. Subjects drift in appearance, actions compress or expand unpredictably, and camera behaviour becomes inconsistent across the duration of the shot. These are not isolated artefacts; they reflect the difficulty of maintaining coherence in a probabilistic process over time.

What becomes apparent is that temporal complexity scales disproportionately. A short sequence may remain relatively stable, while a longer sequence generated from the same description begins to diverge. In practice, this is often visible when duration is extended without increasing the specificity of the underlying structure. The system is required to sustain decisions that were never fully defined, and so it resolves them dynamically at each moment in the sequence.

From a workflow perspective, this has a direct implication. Coherence cannot be assumed over extended durations; it has to be managed.

One way to understand this is to treat duration as a variable that interacts with structure. The longer the duration, the more precisely the internal organisation of the shot needs to be defined. Where that organisation is underspecified, the system will generate variation not only in appearance, but in the sequence of events itself.

The practical response is to shift the locus of control. Rather than attempting to generate extended sequences in a single pass, it becomes more effective to work with shorter, well-defined segments. Each segment can be structured, generated, and evaluated independently, and continuity can then be constructed at the level of the edit.

This approach mirrors established filmmaking practice, but the motivation is slightly different. In traditional production, scenes are broken into shots for reasons of coverage and performance. In AI workflows, segmentation also functions as a way of constraining temporal instability.

What this introduces is a different relationship between generation and editing. Generated clips are not treated as self-contained units that must sustain coherence over their full duration. Instead, they become components within a sequence, where continuity is achieved through composition.

Time, in this context, is not simply a property of the output. It is a parameter that shapes how the workflow itself needs to be organised.

Bringing these elements together reveals the pipeline that emerges when structure, representation, and temporal constraints are treated as parts of the same system.

5. From prompt to pipeline: how the workflow reorganises

Across this discussion, a pattern begins to emerge. Structure, representation, and temporal constraint do not operate independently; they reorganise the workflow itself.

The most significant change is a shift in where control is located.

In a prompt-driven approach, control appears to sit at the point of input. A prompt is written, an output is generated, and any refinement takes place by modifying that prompt. The process is iterative, but the iteration is reactive, because the underlying structure of the request may still be changing.

Once a structured representation of the shot is introduced, that logic begins to change. Control moves earlier in the process, into the specification of the shot itself. Decisions about subject, action, camera, and duration are made before generation takes place, and the prompt becomes a way of expressing those decisions rather than discovering them.

This produces a workflow that can be understood in a series of stages: specification, translation, generation, and evaluation.

The specification stage is where the shot is defined in structured terms. This may take the form of a written template, a simple schema, or a more formal representation, but the key point is that the relationships between elements are fixed before the system is engaged.

The translation stage converts that specification into forms that a particular system can process. This typically includes a prompt, but may also involve parameters or control inputs that sit outside the prompt itself. The separation between these layers is important, because not all aspects of the output can be reliably controlled through text alone.

At the point of execution, this separation becomes quite concrete. The same shot specification can be expressed differently depending on the system: as a prompt in a text-based interface, as parameter fields in systems that expose them, or as a node graph in more open environments. What remains consistent is the underlying description of the shot.

Generation then produces outputs that can be understood as variations of a defined structure. Rather than treating each result as a discrete attempt, it becomes possible to evaluate them in relation to specific aspects of the shot.

Evaluation, in this context, becomes more precise. Instead of asking whether a clip “works” in general terms, the question becomes whether it aligns with the defined structure. Does the sequence hold? Is the camera consistent? Is the duration appropriate?

This enables a more controlled form of iteration. Because the shot has been decomposed into components, adjustments can be made selectively, and the effects of those adjustments can be understood more clearly.

What emerges from this is not a prompt-based interaction, but a production pipeline. The prompt is one component within that pipeline, but it no longer carries the full burden of control.

This brings us to what the emerging pipeline might mean for teaching, and how this way of thinking can be introduced without reducing it to tool-specific instruction.

6. What should we actually be teaching?

If the argument developed here holds—that AI filmmaking operates through structured representations and staged workflows rather than isolated prompts—then the question for teaching becomes clearer.

The issue is not which tools to teach, but how the process is organised.

There is a tendency, when new technologies emerge, to focus on platforms: how to access them, how to configure them, and how to use their features. While this can provide an entry point, it is not a stable foundation. Tools evolve quickly, and knowledge tied to a specific interface often becomes obsolete.

What appears to be more durable is an understanding of how the process itself is structured.

If AI filmmaking is understood as a pipeline, then students need to learn how to define, translate, and evaluate a shot within that pipeline. This involves making explicit a set of distinctions that are often implicit in film practice.

At a minimum, this includes the ability to define a shot across its core dimensions—subject, action, camera, and duration—and to represent that definition in a form that can be translated into system inputs. It also involves understanding the separation between semantic description and control parameters, and how those layers interact within different tools.

In practice, this often means asking students to produce not just a prompt, but a structured description that can be reused across multiple generations. That description may begin as a simple template, but it needs to be stable enough to support comparison, modification, and evaluation.

This also introduces a different approach to iteration. Rather than treating each output as a separate result, students need to understand how to generate and compare variations of a defined structure. This brings the process closer to experimental practice, but with a clear relationship to production intent.

At the level of systems, this way of working becomes visible in how different tools are used together. A generation system produces material, an editing environment shapes it, and more open or inspectable systems can be used to understand how generation operates. The emphasis shifts from mastering a single tool to understanding how processes connect across a pipeline.

One concern that often arises is whether this implies a need for programming. In practice, it does not. What is required is not the ability to write code, but the ability to think in terms of structured inputs, controlled variation, and staged processes. Students do not need to become developers, but they do need to understand how their decisions are interpreted within a system.

What emerges from this is not a replacement for filmmaking, but a different articulation of it. The core concerns remain the same, but they are expressed in a way that is legible to a system rather than a crew.

If approached carefully, this creates an opportunity to make aspects of filmmaking that are often tacit more visible and more transferable. The challenge is not whether to adopt these tools, but how to integrate this way of thinking into existing practice in a way that maintains disciplinary coherence while engaging with a landscape that is already shifting.

The ideas developed here are explored in more practical detail in my report, Prompt Architecture for AI Filmmaking: Designing Structured Production Pipelines for Generative Video. The report sets out a workflow in which creative intentions are translated into structured shot specifications, prompts and control parameters before being executed and evaluated across generative video systems. It includes a worked example showing how this approach can support more controlled, repeatable and transferable AI filmmaking practice.

Read the open-access report:

Hanney, R. (2026). Prompt architecture for AI filmmaking: Designing structured production pipelines for generative video (Version 4) [Report]. Zenodo. https://doi.org/10.5281/zenodo.19304134