Allow using Twig templates #2612

m-vo · 2020-12-30T13:32:25Z

Q	A
Fixed issues	Closes #2303
Docs PR or issue	todo

This PR adds the ability to replace PHP templates with Twig templates in your application. If a template <template>.html.twig is registered in the main Twig template path, rendering is forwarded to Twig instead of rendering natively.

There are a few tricks involved to make the full context available without the need of the raw filter:

All data stored in $arrData as well as some context sensitive functions from the FrontendTemplate class will end up in the context.
However, before passing them to Twig, proxy classes wrapping the values are created (where needed). This allows accessing callables like in Contao templates as well as *drumroll* - bypassing Twigs escaping mechanism.

Callables

If a callable is just a lazy replacement for a string (i.e no arguments are needed), they can just be accessed as if they were strings:

$template->my_callable = function(): string { return 'foo'; };

Will output "foo":
{{ my_callable }}

If arguments are needed, the function can be called via the proxy's invoke method:

$template->my_callable = function($a): string { return $a . 'bar'; };

Will output "foobar":
{{ my_callable.invoke('foo') }}

Input encoding magic

The proxy objects are tagged with a special SafeHTMLValueHolderInterface which gets registered as a safe class for the html strategy in Twigs EscaperExtension.

In order to support arrays or objects that can generate values themselves the proxy will intercept calls and wrap the results. This way the effect is transitive on child elements. That's the reason why there are different value holders implemented. The ObjectValueHolder is the most sophisticated one because it needs to reimplement parts of Twigs access strategy, the other ones are quite straight-forward.

It's possible to tag arbitrary classes with the SafeHTMLValueHolderInterface (e.g. for performance reasons) - they will get skipped when creating value holders.

core-bundle/src/Twig/Extension/ContaoTemplateExtension.php

m-vo · 2020-12-31T14:59:45Z

Twig for Contao templates: no |raw needed anymore. 🥳 🎉

I updated the description accordingly.

m-vo · 2021-01-02T11:41:32Z

Just letting this here for further discussions: There is also the option to register a NodeVisitor that's active in the compilation phase. This feature isn't documented but gets mentioned in the section about extensions. Probably allows altering nodes or the node tree itself.

leofeyer · 2021-01-11T14:35:36Z

core-bundle/src/Twig/Compat/ArrayValueHolder.php

+    private $valueName;
+
+    /**
+     * @internal


Shouldn't this be at class level?

Imho it can't be class level because you need to consume this class (but not instantiate it).

But how is someone supposed to use the class if the constructor is internal?

Your IDE at least marks it when you create an object 🙃

I think it's the same with every class that has a factory. The constructor isn't part of the BC promise, the factory method constructing it, however, is. In this case you would use ProxyFactory::createValueHolder(…) instead of new ArrayValueHolder(…).

We've got the same in place for the studio classes for instance. 🙂

I see. Maybe we should add a comment: @internal Do not instantiate this class in your code; use ProxyFactory::createValueHolder() instead

leofeyer · 2021-01-11T14:39:36Z

core-bundle/src/Twig/Compat/ProxyFactory.php

+        // A more sophisticated check could go here in the future that further
+        // limits the amount of proxy objects that need to be created.


This should be a @todo phpDoc comment.

leofeyer · 2021-01-11T14:41:44Z

core-bundle/src/Twig/Extension/ContaoCompatExtension.php

+class ContaoCompatExtension extends AbstractExtension
+{
+    /**
+     * @internal


Why not at class level?

ausi · 2021-01-11T21:41:04Z

Twig for Contao templates: no |raw needed anymore. 🥳 🎉

I’m not sure if that’s what we want. IMO we should integrate twig templates in a way that they would not have to be changed if we switch from input to output encoding.

How about adding an escaper strategy like html_contao that we enable by default for twig templates that are used in place of a Contao template? In this escaper we would use htmlspecialchars() with double_encode set to false. We could then write the templates the same way as we would with output encoding:

<h1>{{ headline }}</h1> 
<div>{{ content|raw }}</div>

In this example the headline would already be (input)-encoded by Contao but because of double_encode: false escaping doesn’t actually change something but still guarantees the safety of the output.

The content would e.g. be from a tinyMCE field which requires the template author to use |raw, the same way as it would be with regular twig templates using output encoding.

m-vo · 2021-01-12T10:41:57Z

I’m not sure if that’s what we want. IMO we should integrate twig templates in a way that they would not have to be changed if we switch from input to output encoding.

I generally agree with your point (and I already started implementing some thing like this but didn't get too far, yet). 😺

IMO, one of our current problems is having prerendered content at all. The output of the RTE should probably be the only exception. And maybe it would be valid solution to have a simple proxy object for this case later on as well. In Symfony land I'll rarely ever have to use |raw as a consumer. The building blocks (like forms for instance) are really built inside Twig with the help of filters and functions. This is a good paradigm, because it shifts the responsibility of building things to the template: HTML should almost exclusively originate from templates.

So, ideally, we would not only drop input encoding in the future but also migrate away from prerendered content. Together with a special handling for the RTE output a regular user wouldn't have to use encoding filters then. I think it all boils down to two things:

What are the things we want users to learn about Twig templates and encoding? This is especially important if they start using their own Twig templates (e.g. within a fragment) that then behave differently.
Would it be helpful in terms of 'user awareness' about encoding if a change is needed at certain places when we're switching. Note, that it makes a huge difference to add |raw to sanitized tinyMCE output versus to any HTML that was sanitized now but not anymore in the future.

aschempp · 2021-01-13T07:54:51Z

I absolutely agree with @ausi that we should try to be forward-compatible with this Twig integration. I think we had several proposals on that already, but none seems to have worked was well as this (thanks @m-vo!).

I missing here how this is supposed to be the default in the future. If I understand correctly, this would only allow one to override a PHP template with Twig. But would it also allow extensions to use Twig (while fixing the same input/output encoding issues)? And on that spot, Twig does not allow to override a template from another extension, right?

m-vo · 2021-01-13T09:59:02Z

And on that spot, Twig does not allow to override a template from another extension, right?

It's possible. You need to preprend configuration that includes a new namespace path in your extension, though.

Edit: here is an example: https://stackoverflow.com/a/52693472/3146819

m-vo · 2021-01-13T10:06:40Z

But would it also allow extensions to use Twig (while fixing the same input/output encoding issues)?

You mean an extension overwriting a core template? Imo this shouldn't make a difference.

The question to me is: Should there also be any support for things like tinyMCE output, if you're using Twig directly (i.e. from your fragment), not via \Contao\Template. 🤔 And "support" would probably mean "default in the future".

leofeyer · 2021-01-14T15:35:24Z

As discussed in Mumble on January 14th:

HTML is a separate data type (like text or integer) and should be sanitized before it is stored in the DB.
If we want to output a HTML string in Twig, we have to add |raw.
A custom Twig encoder that does not double encode could be used to handle strings that are already encoded in the DB (see Allow using Twig templates #2612 (comment)).

leofeyer · 2021-01-14T16:22:40Z

As discussed in Mumble on January 14th, @Toflar is going to create a ticket in which we define the Twig implementation requirements. Then we can discuss again and eventually implement Twig according to these requirements.

m-vo · 2021-05-04T19:00:35Z

superseded by #2988

@ausi

Description ----------- | Q | A | -----------------| --- | Fixed issues | Fixes #2303 and potentially #2638 | Docs PR or issue | todo supersedes #2612 This PR aims at implementing first steps of native Twig support for Contao. # What's inside ### Encoding You now can write Twig templates like so… ```twig <h1>{{ headline }}</h1> <p class="rte">{{ content|raw }}</p> ``` … and provide input encoded data for the variables. They won't be encoded twice thanks to setting `double_encode: false` (idea © @ausi). This means you do not have to use `|raw` for normal values or, put differently, only where you indeed intend to output the raw data - for example when dealing with rte data. This way templates can be written like they would with output encoding and we get a nice upgrade path. This logic is implemented in a `NodeVisitor` that rewrites the escaping strategy to our own one called `contao_html`. It is only applied to template names that follow a certain rule set (`'%^@contao(_[a-zA-Z0-9_-]*)?/%'`) but you can add your own rules as well. ### Loader Twig is configured with a `ChainLoader` that supports multiple loaders by default. We're using this fact and are now providing two additional loaders with a higher priority (default has priority 0): 1. `FailTolerantFilesystemLoader` (priority: 1) - this is just a simple wrapper around the original implementation that doesn't care about adding missing paths. Paths that Symfony adds at compile (i.e. all the Twig bundle paths) do get rewired to this loader in a compiler pass. We kind of used this mechanism already (only with decoration instead of a new loader). 2. `ContaoFilesystemLoader` (priority: 2) - this is where all the magic happens. This loader will provide all Contao templates from bundles, the app, global templates, theme folders… It is also designed to be altered at runtime and uses a cache pool to store its state. Here are some features that differ from Twig's default filesystem loader: - If you request a template but the current page context is 'a theme' and there is a theme variant of the template, you'll get this instead. In fact that's the case for every operation (`getCacheKey`, `getSourceContext`, `exists`, `isFresh`). - When adding template paths (namespace + directory) there is a switch to enable 'tracking templates'. If enabled, the content of the path will be scanned and found templates will be put in a 'hierarchy' that represents a possible inheritance chain (see below). Tracking templates also allows invalidation (see `isFresh`) for associated variants. Our filesystem loader gets primed by a mandatory cache warmer (`ContaoFilesystemLoaderWarmer`) that'll add all the paths and namespaces. Speaking of namespaces, these are the ones getting registered: 1. `Contao_Theme_<themeSlug>` for each theme folder 2. `Contao_Global` for the global templates folder 3. `Contao_<bundleName>` for each bundle path (could be multiple per bundle); this includes `Contao_App` for the app resources (i.e. in a `AppBundle` or `/contao/templates`). 4. additionally `Contao` for all resources Also see the `TemplateLocator` helper service for reference. Btw.: The existing `AddResourcesPass` also searches these paths but behaves differently. Maybe we could merge those two things in the future. But I'm not planning to do this in this PR. ### Inheritance Twig allows inheriting multiple times (`A extends B extends C extends …`) which is a very helpful concept when building complex components. However this doesn't play nicely with an extension system like ours where the code does not know how it will be orchestrated. Imagine you've got an original resource and you are adjusting it in the app: ```twig {# the original thing.html.twig #} {% block content %}{{ foo }}{% endblock %} <footer>{% block footer %}Footer{% endblock %}</footer> ``` ```twig {# App #} {% extends "@Contao/thing.html.twig" %} {% block content %}<section>{{ parent() }}</section>{% endblock %} {% block footer %}My own footer{% footer %} ``` Now you install an extension (think of an extension to a bigger extension) that brings its own modifications: ```twig {# BarExtension #} {% extends "@Contao/thing.html.twig" %} {% block content %}{{ parent() }}? Bar!{% endblock %} ``` You would probably expect this output: ```twig <section>{{ foo }}? Bar!</section> <footer>My own footer</footer> ``` And in fact, this is what you would get with this PR. 😁 You may have noticed the namespace every template is extending from is `@Contao`. In regular Twig land there can only ever win a single template (with the same name) in each namespace, in fact this is why Symfony registers each bundle namespace twice (`@<bundle>` and `@!<bundle>`), so that you can overwrite a template and still extend from it (using the `!` one). We're working around this limitation using two things: we know about templates not only template paths (see tracking in previous section) and we're providing our own version of the `ExtendsTokenParser` that - whenever a `@Contao` template is targeted - will rewrite it to a specific namespace according to the inheritance hierarchy. So `@Contao/foo.html.twig` would for instance become `@Contao_App/foo.html.twig` in the theme template, `@Contao_FooBundle/foo.html.twig` in the app template and `@Contao_CoreBundle/foo.html.twig` in the FooBundle template. ### PHP Template Interop Twig and our current templates are similar in a way that at the end PHP code is evaluated and both support template inheritance. But that's kind of it. 😄 To be able to make them extend from each other seamlessly we IMHO have to transpile PHP to Twig and process them like regular Twig templates. But that's something for the future. What **is** possible with this PR is extending a `.html5` template in Twig (like `{% extends "@contao\nav_default.html5" %}`) but bear in mind that this won't allow certain dynamic things like introducing new blocks and extending again. This part is probably the thing that involves most "magic". Here is the gist how it works: We're also registering `.html5` templates as if they were Twig templates in our loader. The loader, however, strips the PHP code when returning a source context to not confuse Twig's lexer/parser chain but simply keeps the identified block names in there. Later on - in a node visitor - we're then dressing the `ModuleNode` of this template to include a proxy call and inject `BlockNode` s that return `[[TL_PARENT]]` and that are rendered when running the proxy code. If a block was overwritten by a child template it will win, otherwise the placeholder will be output. Contao's template engine then gets run with the blocks prepopulated and will output the result. Commits ------- 0f3c623 add ContaoEscaper that does not double encode input 9cbb7bb add NodeVisitor that can alter escape filter strategies e28bb7c add InteropExtension that allows registering templates that should get processed by the ContaoEscaperNodeVisitor d594574 add our own FilesystemLoader instead of decorating 5eda502 add ContaoTwigTemplateLocator + rewire/register locations in TwigPathsPass compiler pass 5e9b81d delegate rendering to twig if a template surrogate exists 15ebc62 register template for input encoding when rendering via TemplateInheritance c7406f9 fix tests 1cd3c1d add todo note for theme mapping f0f90e6 check for non-existing directories in ContaoTwigTemplateLocator 784af57 cleanup 3d10b53 add and configure TemplateHierarchy + DynamicExtendsTokenParser 0b3e6b3 handle uppercase entities 3188917 rework and simplify the concept f71b882 reimplement contao resources location 193cf15 fix service declaration 80ee873 make signature compatible to legacy interface 93016d7 fix tests 2fdd8e4 raise symfony/dependency-injection dependency see symfony/symfony#40167 (Definition::removeMethodCall won't remove all calls) 018f7be throw if templates are ambiguous (multiple variants in subfolders) b93d098 CS 8db05ee try to find out what's wrong on windows 9ca8e27 fix comments 8f0bd98 canonicalize paths fc2af8a fix locator paths b2f0c62 add shortcut to refresh the loader 87d1374 add automatic cache busting 4c2c0fc add command to list + refresh the current template hierarchy 12780cc fix tests c8474d4 split loader responsibilities c2f1f76 refactor ContaoFilesystemLoader + make it only apply to Contao namespaces cedf138 make warmer auto refresh in dev mode ec092b6 add and improve tests 91476a7 improve regular expressions 181c099 add more tests 3fed107 add more tests fd33065 add more tests 1ee756b hash each individual hierarchy chain 36a9cfb also load html5 templates + allow extending them e7f0720 refactor legacy block rendering 9e72dcd rework cache busting f65800e add + adjust tests 30d68e6 CS + path normalizing babf65f fix call 3647786 fix version constraints d3cf90e improve block regex Co-authored-by: Martin Auswöger <martin@auswoeger.com> b9bbcf2 improve + harden block parsing ee73cf2 revert changes to unaffected code e4243c3 improve naming + comments c970703 remove static keyword from callback function d32aa44 always use the FrontendTemplate class to render legacy templates 6e55565 WIP 7612606 fix filemtime check 8f854fa token parsers: improve support for extends/embed + support include 2d26645 add context transformer to support closures c690031 improve debug command 1e57321 add todo 3b5253f add tests for DebugContaoTwigCommand 1711029 add + fix tests for contao twig extension 7cef52f fix tests 90c4be1 mark classes as final/internal/experimental 23de8dd fix tests 3262d9b add todo placeholders 39d8b4f add tests for ContextHelper 77430e9 fix phpstan errors 9ae391a fix psalm + php7.3 tests b2ea883 simplify + harden twig_include function call 0609427 use named arguments 3625509 ask Twig's CoreExtension for the twig_include callable 39c4761 add and improve tests bf862f5 simplify call de4f44c improve coverage b84cae3 simplify expression 25a3f18 improve coverage 12a7fc1 fix tests 27ca43d ignore coverage for PHP7.3 code 47012da improve coverage 79c0534 improve coverage cdc553a fix test dependency c414250 fix tests on Windows ec65108 add contao_sections + contao_section methods 544a95b small improvement for generated PhpTemplateProxyNode code 9188b03 refactor/unify Twig extensions + runtimes bef9bab fix tests 1c9f019 check for closures instead of callables when transforming the context 9edace8 remove redundant variable bcad333 remove redundant return statement Co-authored-by: Martin Auswöger <martin@auswoeger.com> 80dcb4f handle filemtime only generates warnings, no errors 3b4c877 CS 2c6346f CS f65506b remove redundant return value

m-vo added 5 commits December 30, 2020 13:34

handle rendering twig templates

f569080

add and adjust tests

88e90f2

add c Twig filter

4dcd031

make template functions accessible

04f88c9

add test for CallableProxy

9d32c39

leofeyer assigned m-vo Dec 30, 2020

leofeyer added the feature label Dec 30, 2020

m-vo mentioned this pull request Dec 30, 2020

Allow using Twig templates to implement/replace Contao templates #2303

Closed

ausi reviewed Dec 30, 2020

View reviewed changes

core-bundle/src/Twig/Extension/ContaoTemplateExtension.php Outdated Show resolved Hide resolved

implement automatic value wrapping

0f42e5b

leofeyer added this to the 4.11 milestone Jan 2, 2021

leofeyer reviewed Jan 11, 2021

View reviewed changes

Merge branch 'master' into feature/twig-party-and-everyones-invited

cc6a030

leofeyer force-pushed the feature/twig-party-and-everyones-invited branch 2 times, most recently from 7ca85ea to 5cc76b8 Compare January 11, 2021 14:59

CS

db34a7c

leofeyer force-pushed the feature/twig-party-and-everyones-invited branch from 5cc76b8 to db34a7c Compare January 11, 2021 15:13

ausi added the up for discussion Issues and PRs which will be discussed in our monthly calls. label Jan 11, 2021

leofeyer removed the up for discussion Issues and PRs which will be discussed in our monthly calls. label Jan 14, 2021

leofeyer removed this from the 4.11 milestone Jan 14, 2021

leofeyer added this to the 4.12 milestone Jan 14, 2021

Base automatically changed from master to 4.x January 18, 2021 15:42

m-vo mentioned this pull request Apr 14, 2021

🤔 Allow passing parameters to fn() m-vo/contao-twig#8

Closed

m-vo marked this pull request as draft April 21, 2021 16:07

This was referenced Apr 28, 2021

Add RenderTemplateEvent m-vo/contao-twig#9

Open

Add Twig support #2988

Merged

m-vo closed this May 4, 2021

m-vo deleted the feature/twig-party-and-everyones-invited branch July 5, 2024 12:32

		// A more sophisticated check could go here in the future that further
		// limits the amount of proxy objects that need to be created.

Uh oh!

Allow using Twig templates #2612

Allow using Twig templates #2612

Uh oh!

Conversation

m-vo commented Dec 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Callables

Input encoding magic

Uh oh!

Uh oh!

m-vo commented Dec 31, 2020

Uh oh!

m-vo commented Jan 2, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ausi commented Jan 11, 2021

Uh oh!

m-vo commented Jan 12, 2021

Uh oh!

aschempp commented Jan 13, 2021

Uh oh!

m-vo commented Jan 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m-vo commented Jan 13, 2021

Uh oh!

leofeyer commented Jan 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leofeyer commented Jan 14, 2021

Uh oh!

m-vo commented May 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

m-vo commented Dec 30, 2020 •

edited

Loading

m-vo commented Jan 13, 2021 •

edited

Loading

leofeyer commented Jan 14, 2021 •

edited

Loading